Search in sources :

Example 26 with StandardFilter

use of org.apache.lucene.analysis.standard.StandardFilter in project lucene-solr by apache.

the class CzechAnalyzer method normalize.

@Override
protected TokenStream normalize(String fieldName, TokenStream in) {
    TokenStream result = new StandardFilter(in);
    result = new LowerCaseFilter(result);
    return result;
}
Also used : TokenStream(org.apache.lucene.analysis.TokenStream) StandardFilter(org.apache.lucene.analysis.standard.StandardFilter) LowerCaseFilter(org.apache.lucene.analysis.LowerCaseFilter)

Example 27 with StandardFilter

use of org.apache.lucene.analysis.standard.StandardFilter in project lucene-solr by apache.

the class DanishAnalyzer method normalize.

@Override
protected TokenStream normalize(String fieldName, TokenStream in) {
    TokenStream result = new StandardFilter(in);
    result = new LowerCaseFilter(result);
    return result;
}
Also used : TokenStream(org.apache.lucene.analysis.TokenStream) StandardFilter(org.apache.lucene.analysis.standard.StandardFilter) LowerCaseFilter(org.apache.lucene.analysis.LowerCaseFilter)

Example 28 with StandardFilter

use of org.apache.lucene.analysis.standard.StandardFilter in project lucene-solr by apache.

the class GreekAnalyzer method normalize.

@Override
protected TokenStream normalize(String fieldName, TokenStream in) {
    TokenStream result = new StandardFilter(in);
    result = new GreekLowerCaseFilter(result);
    return result;
}
Also used : TokenStream(org.apache.lucene.analysis.TokenStream) StandardFilter(org.apache.lucene.analysis.standard.StandardFilter)

Example 29 with StandardFilter

use of org.apache.lucene.analysis.standard.StandardFilter in project lucene-solr by apache.

the class GreekAnalyzer method createComponents.

/**
   * Creates
   * {@link org.apache.lucene.analysis.Analyzer.TokenStreamComponents}
   * used to tokenize all the text in the provided {@link Reader}.
   * 
   * @return {@link org.apache.lucene.analysis.Analyzer.TokenStreamComponents}
   *         built from a {@link StandardTokenizer} filtered with
   *         {@link GreekLowerCaseFilter}, {@link StandardFilter},
   *         {@link StopFilter}, and {@link GreekStemFilter}
   */
@Override
protected TokenStreamComponents createComponents(String fieldName) {
    final Tokenizer source = new StandardTokenizer();
    TokenStream result = new GreekLowerCaseFilter(source);
    result = new StandardFilter(result);
    result = new StopFilter(result, stopwords);
    result = new GreekStemFilter(result);
    return new TokenStreamComponents(source, result);
}
Also used : TokenStream(org.apache.lucene.analysis.TokenStream) StandardTokenizer(org.apache.lucene.analysis.standard.StandardTokenizer) StopFilter(org.apache.lucene.analysis.StopFilter) StandardFilter(org.apache.lucene.analysis.standard.StandardFilter) Tokenizer(org.apache.lucene.analysis.Tokenizer) StandardTokenizer(org.apache.lucene.analysis.standard.StandardTokenizer)

Example 30 with StandardFilter

use of org.apache.lucene.analysis.standard.StandardFilter in project lucene-solr by apache.

the class EnglishAnalyzer method normalize.

@Override
protected TokenStream normalize(String fieldName, TokenStream in) {
    TokenStream result = new StandardFilter(in);
    result = new LowerCaseFilter(result);
    return result;
}
Also used : TokenStream(org.apache.lucene.analysis.TokenStream) StandardFilter(org.apache.lucene.analysis.standard.StandardFilter) LowerCaseFilter(org.apache.lucene.analysis.LowerCaseFilter)

Aggregations

StandardFilter (org.apache.lucene.analysis.standard.StandardFilter)73 TokenStream (org.apache.lucene.analysis.TokenStream)71 LowerCaseFilter (org.apache.lucene.analysis.LowerCaseFilter)58 Tokenizer (org.apache.lucene.analysis.Tokenizer)36 StopFilter (org.apache.lucene.analysis.StopFilter)35 StandardTokenizer (org.apache.lucene.analysis.standard.StandardTokenizer)35 SetKeywordMarkerFilter (org.apache.lucene.analysis.miscellaneous.SetKeywordMarkerFilter)30 SnowballFilter (org.apache.lucene.analysis.snowball.SnowballFilter)14 ElisionFilter (org.apache.lucene.analysis.util.ElisionFilter)8 LowerCaseFilter (org.apache.lucene.analysis.core.LowerCaseFilter)5 DecimalDigitFilter (org.apache.lucene.analysis.core.DecimalDigitFilter)4 ASCIIFoldingFilter (org.apache.lucene.analysis.miscellaneous.ASCIIFoldingFilter)4 StopFilter (org.apache.lucene.analysis.core.StopFilter)3 KeywordTokenizer (org.apache.lucene.analysis.core.KeywordTokenizer)2 EnglishPossessiveFilter (org.apache.lucene.analysis.en.EnglishPossessiveFilter)2 PorterStemFilter (org.apache.lucene.analysis.en.PorterStemFilter)2 WordDelimiterFilter (org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter)2 Reader (java.io.Reader)1 Analyzer (org.apache.lucene.analysis.Analyzer)1 CharArraySet (org.apache.lucene.analysis.CharArraySet)1