Search in sources :

Example 1 with NamedRegistry

use of org.opensearch.common.NamedRegistry in project OpenSearch by opensearch-project.

the class AnalysisModule method setupPreConfiguredTokenFilters.

static Map<String, PreConfiguredTokenFilter> setupPreConfiguredTokenFilters(List<AnalysisPlugin> plugins) {
    NamedRegistry<PreConfiguredTokenFilter> preConfiguredTokenFilters = new NamedRegistry<>("pre-configured token_filter");
    // Add filters available in lucene-core
    preConfiguredTokenFilters.register("lowercase", PreConfiguredTokenFilter.singleton("lowercase", true, LowerCaseFilter::new));
    // Add "standard" for old indices (bwc)
    preConfiguredTokenFilters.register("standard", PreConfiguredTokenFilter.openSearchVersion("standard", true, (reader, version) -> {
        // until version 7_5_2
        if (version.before(LegacyESVersion.V_7_6_0)) {
            deprecationLogger.deprecate("standard_deprecation", "The [standard] token filter is deprecated and will be removed in a future version.");
        } else {
            throw new IllegalArgumentException("The [standard] token filter has been removed.");
        }
        return reader;
    }));
    for (AnalysisPlugin plugin : plugins) {
        for (PreConfiguredTokenFilter filter : plugin.getPreConfiguredTokenFilters()) {
            preConfiguredTokenFilters.register(filter.getName(), filter);
        }
    }
    return unmodifiableMap(preConfiguredTokenFilters.getRegistry());
}
Also used : TokenizerFactory(org.opensearch.index.analysis.TokenizerFactory) IndexMetadata(org.opensearch.cluster.metadata.IndexMetadata) PreBuiltAnalyzerProviderFactory(org.opensearch.index.analysis.PreBuiltAnalyzerProviderFactory) StopAnalyzerProvider(org.opensearch.index.analysis.StopAnalyzerProvider) TokenFilterFactory(org.opensearch.index.analysis.TokenFilterFactory) Version(org.opensearch.Version) StopTokenFilterFactory(org.opensearch.index.analysis.StopTokenFilterFactory) DeprecationLogger(org.opensearch.common.logging.DeprecationLogger) SimpleAnalyzerProvider(org.opensearch.index.analysis.SimpleAnalyzerProvider) AnalysisRegistry(org.opensearch.index.analysis.AnalysisRegistry) LegacyESVersion(org.opensearch.LegacyESVersion) KeywordAnalyzerProvider(org.opensearch.index.analysis.KeywordAnalyzerProvider) AnalysisPlugin.requiresAnalysisSettings(org.opensearch.plugins.AnalysisPlugin.requiresAnalysisSettings) CharFilterFactory(org.opensearch.index.analysis.CharFilterFactory) Locale(java.util.Locale) Map(java.util.Map) StandardTokenizerFactory(org.opensearch.index.analysis.StandardTokenizerFactory) PreConfiguredTokenizer(org.opensearch.index.analysis.PreConfiguredTokenizer) Environment(org.opensearch.env.Environment) LowerCaseFilter(org.apache.lucene.analysis.LowerCaseFilter) TokenStream(org.apache.lucene.analysis.TokenStream) PreConfiguredTokenFilter(org.opensearch.index.analysis.PreConfiguredTokenFilter) AbstractTokenFilterFactory(org.opensearch.index.analysis.AbstractTokenFilterFactory) Settings(org.opensearch.common.settings.Settings) IOException(java.io.IOException) PreConfiguredCharFilter(org.opensearch.index.analysis.PreConfiguredCharFilter) ShingleTokenFilterFactory(org.opensearch.index.analysis.ShingleTokenFilterFactory) LowercaseNormalizerProvider(org.opensearch.index.analysis.LowercaseNormalizerProvider) AnalysisPlugin(org.opensearch.plugins.AnalysisPlugin) List(java.util.List) AnalyzerProvider(org.opensearch.index.analysis.AnalyzerProvider) NamedRegistry(org.opensearch.common.NamedRegistry) IndexSettings(org.opensearch.index.IndexSettings) WhitespaceAnalyzerProvider(org.opensearch.index.analysis.WhitespaceAnalyzerProvider) HunspellTokenFilterFactory(org.opensearch.index.analysis.HunspellTokenFilterFactory) Collections.unmodifiableMap(java.util.Collections.unmodifiableMap) StandardAnalyzerProvider(org.opensearch.index.analysis.StandardAnalyzerProvider) NamedRegistry(org.opensearch.common.NamedRegistry) PreConfiguredTokenFilter(org.opensearch.index.analysis.PreConfiguredTokenFilter) AnalysisPlugin(org.opensearch.plugins.AnalysisPlugin)

Example 2 with NamedRegistry

use of org.opensearch.common.NamedRegistry in project OpenSearch by opensearch-project.

the class AnalysisModule method setupPreConfiguredTokenizers.

static Map<String, PreConfiguredTokenizer> setupPreConfiguredTokenizers(List<AnalysisPlugin> plugins) {
    NamedRegistry<PreConfiguredTokenizer> preConfiguredTokenizers = new NamedRegistry<>("pre-configured tokenizer");
    // Temporary shim to register old style pre-configured tokenizers
    for (PreBuiltTokenizers tokenizer : PreBuiltTokenizers.values()) {
        String name = tokenizer.name().toLowerCase(Locale.ROOT);
        PreConfiguredTokenizer preConfigured;
        switch(tokenizer.getCachingStrategy()) {
            case ONE:
                preConfigured = PreConfiguredTokenizer.singleton(name, () -> tokenizer.create(Version.CURRENT));
                break;
            default:
                throw new UnsupportedOperationException("Caching strategy unsupported by temporary shim [" + tokenizer + "]");
        }
        preConfiguredTokenizers.register(name, preConfigured);
    }
    for (AnalysisPlugin plugin : plugins) {
        for (PreConfiguredTokenizer tokenizer : plugin.getPreConfiguredTokenizers()) {
            preConfiguredTokenizers.register(tokenizer.getName(), tokenizer);
        }
    }
    return unmodifiableMap(preConfiguredTokenizers.getRegistry());
}
Also used : NamedRegistry(org.opensearch.common.NamedRegistry) PreConfiguredTokenizer(org.opensearch.index.analysis.PreConfiguredTokenizer) AnalysisPlugin(org.opensearch.plugins.AnalysisPlugin)

Example 3 with NamedRegistry

use of org.opensearch.common.NamedRegistry in project OpenSearch by opensearch-project.

the class AnalysisModule method setupTokenFilters.

private NamedRegistry<AnalysisProvider<TokenFilterFactory>> setupTokenFilters(List<AnalysisPlugin> plugins, HunspellService hunspellService) {
    NamedRegistry<AnalysisProvider<TokenFilterFactory>> tokenFilters = new NamedRegistry<>("token_filter");
    tokenFilters.register("stop", StopTokenFilterFactory::new);
    // Add "standard" for old indices (bwc)
    tokenFilters.register("standard", new AnalysisProvider<TokenFilterFactory>() {

        @Override
        public TokenFilterFactory get(IndexSettings indexSettings, Environment environment, String name, Settings settings) {
            if (indexSettings.getIndexVersionCreated().before(LegacyESVersion.V_7_0_0)) {
                deprecationLogger.deprecate("standard_deprecation", "The [standard] token filter name is deprecated and will be removed in a future version.");
            } else {
                throw new IllegalArgumentException("The [standard] token filter has been removed.");
            }
            return new AbstractTokenFilterFactory(indexSettings, name, settings) {

                @Override
                public TokenStream create(TokenStream tokenStream) {
                    return tokenStream;
                }
            };
        }

        @Override
        public boolean requiresAnalysisSettings() {
            return false;
        }
    });
    tokenFilters.register("shingle", ShingleTokenFilterFactory::new);
    tokenFilters.register("hunspell", requiresAnalysisSettings((indexSettings, env, name, settings) -> new HunspellTokenFilterFactory(indexSettings, name, settings, hunspellService)));
    tokenFilters.extractAndRegister(plugins, AnalysisPlugin::getTokenFilters);
    return tokenFilters;
}
Also used : TokenizerFactory(org.opensearch.index.analysis.TokenizerFactory) IndexMetadata(org.opensearch.cluster.metadata.IndexMetadata) PreBuiltAnalyzerProviderFactory(org.opensearch.index.analysis.PreBuiltAnalyzerProviderFactory) StopAnalyzerProvider(org.opensearch.index.analysis.StopAnalyzerProvider) TokenFilterFactory(org.opensearch.index.analysis.TokenFilterFactory) Version(org.opensearch.Version) StopTokenFilterFactory(org.opensearch.index.analysis.StopTokenFilterFactory) DeprecationLogger(org.opensearch.common.logging.DeprecationLogger) SimpleAnalyzerProvider(org.opensearch.index.analysis.SimpleAnalyzerProvider) AnalysisRegistry(org.opensearch.index.analysis.AnalysisRegistry) LegacyESVersion(org.opensearch.LegacyESVersion) KeywordAnalyzerProvider(org.opensearch.index.analysis.KeywordAnalyzerProvider) AnalysisPlugin.requiresAnalysisSettings(org.opensearch.plugins.AnalysisPlugin.requiresAnalysisSettings) CharFilterFactory(org.opensearch.index.analysis.CharFilterFactory) Locale(java.util.Locale) Map(java.util.Map) StandardTokenizerFactory(org.opensearch.index.analysis.StandardTokenizerFactory) PreConfiguredTokenizer(org.opensearch.index.analysis.PreConfiguredTokenizer) Environment(org.opensearch.env.Environment) LowerCaseFilter(org.apache.lucene.analysis.LowerCaseFilter) TokenStream(org.apache.lucene.analysis.TokenStream) PreConfiguredTokenFilter(org.opensearch.index.analysis.PreConfiguredTokenFilter) AbstractTokenFilterFactory(org.opensearch.index.analysis.AbstractTokenFilterFactory) Settings(org.opensearch.common.settings.Settings) IOException(java.io.IOException) PreConfiguredCharFilter(org.opensearch.index.analysis.PreConfiguredCharFilter) ShingleTokenFilterFactory(org.opensearch.index.analysis.ShingleTokenFilterFactory) LowercaseNormalizerProvider(org.opensearch.index.analysis.LowercaseNormalizerProvider) AnalysisPlugin(org.opensearch.plugins.AnalysisPlugin) List(java.util.List) AnalyzerProvider(org.opensearch.index.analysis.AnalyzerProvider) NamedRegistry(org.opensearch.common.NamedRegistry) IndexSettings(org.opensearch.index.IndexSettings) WhitespaceAnalyzerProvider(org.opensearch.index.analysis.WhitespaceAnalyzerProvider) HunspellTokenFilterFactory(org.opensearch.index.analysis.HunspellTokenFilterFactory) Collections.unmodifiableMap(java.util.Collections.unmodifiableMap) StandardAnalyzerProvider(org.opensearch.index.analysis.StandardAnalyzerProvider) StopTokenFilterFactory(org.opensearch.index.analysis.StopTokenFilterFactory) TokenStream(org.apache.lucene.analysis.TokenStream) ShingleTokenFilterFactory(org.opensearch.index.analysis.ShingleTokenFilterFactory) IndexSettings(org.opensearch.index.IndexSettings) AbstractTokenFilterFactory(org.opensearch.index.analysis.AbstractTokenFilterFactory) TokenFilterFactory(org.opensearch.index.analysis.TokenFilterFactory) StopTokenFilterFactory(org.opensearch.index.analysis.StopTokenFilterFactory) AbstractTokenFilterFactory(org.opensearch.index.analysis.AbstractTokenFilterFactory) ShingleTokenFilterFactory(org.opensearch.index.analysis.ShingleTokenFilterFactory) HunspellTokenFilterFactory(org.opensearch.index.analysis.HunspellTokenFilterFactory) NamedRegistry(org.opensearch.common.NamedRegistry) Environment(org.opensearch.env.Environment) AnalysisPlugin.requiresAnalysisSettings(org.opensearch.plugins.AnalysisPlugin.requiresAnalysisSettings) Settings(org.opensearch.common.settings.Settings) IndexSettings(org.opensearch.index.IndexSettings) HunspellTokenFilterFactory(org.opensearch.index.analysis.HunspellTokenFilterFactory) AnalysisPlugin(org.opensearch.plugins.AnalysisPlugin)

Example 4 with NamedRegistry

use of org.opensearch.common.NamedRegistry in project OpenSearch by opensearch-project.

the class SearchModule method setupHighlighters.

private Map<String, Highlighter> setupHighlighters(Settings settings, List<SearchPlugin> plugins) {
    NamedRegistry<Highlighter> highlighters = new NamedRegistry<>("highlighter");
    highlighters.register("fvh", new FastVectorHighlighter(settings));
    highlighters.register("plain", new PlainHighlighter());
    highlighters.register("unified", new UnifiedHighlighter());
    highlighters.extractAndRegister(plugins, SearchPlugin::getHighlighters);
    return unmodifiableMap(highlighters.getRegistry());
}
Also used : NamedRegistry(org.opensearch.common.NamedRegistry) SearchPlugin(org.opensearch.plugins.SearchPlugin) UnifiedHighlighter(org.opensearch.search.fetch.subphase.highlight.UnifiedHighlighter) PlainHighlighter(org.opensearch.search.fetch.subphase.highlight.PlainHighlighter) FastVectorHighlighter(org.opensearch.search.fetch.subphase.highlight.FastVectorHighlighter) Highlighter(org.opensearch.search.fetch.subphase.highlight.Highlighter) FastVectorHighlighter(org.opensearch.search.fetch.subphase.highlight.FastVectorHighlighter) PlainHighlighter(org.opensearch.search.fetch.subphase.highlight.PlainHighlighter) UnifiedHighlighter(org.opensearch.search.fetch.subphase.highlight.UnifiedHighlighter)

Aggregations

NamedRegistry (org.opensearch.common.NamedRegistry)4 PreConfiguredTokenizer (org.opensearch.index.analysis.PreConfiguredTokenizer)3 AnalysisPlugin (org.opensearch.plugins.AnalysisPlugin)3 IOException (java.io.IOException)2 Collections.unmodifiableMap (java.util.Collections.unmodifiableMap)2 List (java.util.List)2 Locale (java.util.Locale)2 Map (java.util.Map)2 LowerCaseFilter (org.apache.lucene.analysis.LowerCaseFilter)2 TokenStream (org.apache.lucene.analysis.TokenStream)2 LegacyESVersion (org.opensearch.LegacyESVersion)2 Version (org.opensearch.Version)2 IndexMetadata (org.opensearch.cluster.metadata.IndexMetadata)2 DeprecationLogger (org.opensearch.common.logging.DeprecationLogger)2 Settings (org.opensearch.common.settings.Settings)2 Environment (org.opensearch.env.Environment)2 IndexSettings (org.opensearch.index.IndexSettings)2 AbstractTokenFilterFactory (org.opensearch.index.analysis.AbstractTokenFilterFactory)2 AnalysisRegistry (org.opensearch.index.analysis.AnalysisRegistry)2 AnalyzerProvider (org.opensearch.index.analysis.AnalyzerProvider)2