Examples with ASCIIFoldingFilter org.apache.lucene.analysis.miscellaneous.ASCIIFoldingFilter used on opensource projects

Example 6 with ASCIIFoldingFilter

use of org.apache.lucene.analysis.miscellaneous.ASCIIFoldingFilter in project cogcomp-nlp by CogComp.

the class WikiURLAnalyzer method createComponents.

@Override
protected TokenStreamComponents createComponents(final String fieldName) {
    final Tokenizer source = new KeywordTokenizer();
    TokenStream result = new StandardFilter(source);
    result = new CharacterFilter(result);
    result = new ASCIIFoldingFilter(result);
    result = new LowerCaseFilter(result);
    return new TokenStreamComponents(source, result);
}