Examples with TokenStream org.apache.lucene.analysis.TokenStream used on opensource projects

Example 6 with TokenStream

use of org.apache.lucene.analysis.TokenStream in project elasticsearch by elastic.

the class CustomAnalyzer method createComponents.

@Override
protected TokenStreamComponents createComponents(String fieldName) {
    Tokenizer tokenizer = tokenizerFactory.create();
    TokenStream tokenStream = tokenizer;
    for (TokenFilterFactory tokenFilter : tokenFilters) {
        tokenStream = tokenFilter.create(tokenStream);
    }
    return new TokenStreamComponents(tokenizer, tokenStream);
}

Example 7 with TokenStream

use of org.apache.lucene.analysis.TokenStream in project elasticsearch by elastic.

the class EdgeNGramTokenFilterFactory method create.

@Override
public TokenStream create(TokenStream tokenStream) {
    TokenStream result = tokenStream;
    // side=BACK is not supported anymore but applying ReverseStringFilter up-front and after the token filter has the same effect
    if (side == SIDE_BACK) {
        result = new ReverseStringFilter(result);
    }
    result = new EdgeNGramTokenFilter(result, minGram, maxGram);
    // side=BACK is not supported anymore but applying ReverseStringFilter up-front and after the token filter has the same effect
    if (side == SIDE_BACK) {
        result = new ReverseStringFilter(result);
    }
    return result;
}

Example 8 with TokenStream

use of org.apache.lucene.analysis.TokenStream in project elasticsearch by elastic.

the class FingerprintTokenFilterFactory method create.

@Override
public TokenStream create(TokenStream tokenStream) {
    TokenStream result = tokenStream;
    result = new FingerprintFilter(result, maxOutputSize, separator);
    return result;
}

Example 9 with TokenStream

use of org.apache.lucene.analysis.TokenStream in project elasticsearch by elastic.

the class PatternAnalyzer method createComponents.

@Override
protected TokenStreamComponents createComponents(String s) {
    final Tokenizer tokenizer = new PatternTokenizer(pattern, -1);
    TokenStream stream = tokenizer;
    if (lowercase) {
        stream = new LowerCaseFilter(stream);
    }
    if (stopWords != null) {
        stream = new StopFilter(stream, stopWords);
    }
    return new TokenStreamComponents(tokenizer, stream);
}

Example 10 with TokenStream

use of org.apache.lucene.analysis.TokenStream in project elasticsearch by elastic.

the class SimpleIcuCollationTokenFilterTests method assertCollation.

private void assertCollation(TokenFilterFactory factory, String string1, String string2, int comparison) throws IOException {
    Tokenizer tokenizer = new KeywordTokenizer();
    tokenizer.setReader(new StringReader(string1));
    TokenStream stream1 = factory.create(tokenizer);
    tokenizer = new KeywordTokenizer();
    tokenizer.setReader(new StringReader(string2));
    TokenStream stream2 = factory.create(tokenizer);
    assertCollation(stream1, stream2, comparison);
}