This repository has been archived by the owner on Jul 30, 2022. It is now read-only.
Butter, bread and green cheese
Pre-release
Pre-release
This release adds Frisian as a language. It's being treated as a secend class citizen for now whilst English, Dutch and German remain to have the primary focus.
Also new in this release are the separation of token filtering from the tokenizers, and thus all tokenizers must implement the new ITokenFilter. One such example is the new StopwordTokenFilter, which uses the updated stopwords lists in languages.
You can now use the new BasicStringBuilder to convert a token list back to a string