Examples with TweetLowerCaseEntityPreservingFilter io.anserini.analysis.TweetLowerCaseEntityPreservingFilter used on opensource projects

Example 1 with TweetLowerCaseEntityPreservingFilter

use of io.anserini.analysis.TweetLowerCaseEntityPreservingFilter in project Anserini by castorini.

the class TRECAnalyzer method createComponents.

@Override
protected TokenStreamComponents createComponents(String fieldName) {
    Tokenizer source = new WhitespaceTokenizer();
    TokenStream filter = new TweetLowerCaseEntityPreservingFilter(source);
    return new TokenStreamComponents(source, filter);
}