Search in sources :

Example 1 with AnalysisModule

use of org.opensearch.indices.analysis.AnalysisModule in project OpenSearch by opensearch-project.

the class CompoundAnalysisTests method analyze.

private List<String> analyze(Settings settings, String analyzerName, String text) throws IOException {
    IndexSettings idxSettings = IndexSettingsModule.newIndexSettings("test", settings);
    AnalysisModule analysisModule = createAnalysisModule(settings);
    IndexAnalyzers indexAnalyzers = analysisModule.getAnalysisRegistry().build(idxSettings);
    Analyzer analyzer = indexAnalyzers.get(analyzerName).analyzer();
    TokenStream stream = analyzer.tokenStream("", text);
    stream.reset();
    CharTermAttribute termAtt = stream.addAttribute(CharTermAttribute.class);
    List<String> terms = new ArrayList<>();
    while (stream.incrementToken()) {
        String tokText = termAtt.toString();
        terms.add(tokText);
    }
    return terms;
}
Also used : TokenStream(org.apache.lucene.analysis.TokenStream) CharTermAttribute(org.apache.lucene.analysis.tokenattributes.CharTermAttribute) IndexSettings(org.opensearch.index.IndexSettings) ArrayList(java.util.ArrayList) IndexAnalyzers(org.opensearch.index.analysis.IndexAnalyzers) AnalysisModule(org.opensearch.indices.analysis.AnalysisModule) Analyzer(org.apache.lucene.analysis.Analyzer)

Example 2 with AnalysisModule

use of org.opensearch.indices.analysis.AnalysisModule in project OpenSearch by opensearch-project.

the class EdgeNGramTokenizerTests method buildAnalyzers.

private IndexAnalyzers buildAnalyzers(Version version, String tokenizer) throws IOException {
    Settings settings = Settings.builder().put(Environment.PATH_HOME_SETTING.getKey(), createTempDir().toString()).build();
    Settings indexSettings = Settings.builder().put(IndexMetadata.SETTING_VERSION_CREATED, version).put("index.analysis.analyzer.my_analyzer.tokenizer", tokenizer).build();
    IndexSettings idxSettings = IndexSettingsModule.newIndexSettings("index", indexSettings);
    return new AnalysisModule(TestEnvironment.newEnvironment(settings), Collections.singletonList(new CommonAnalysisPlugin())).getAnalysisRegistry().build(idxSettings);
}
Also used : IndexSettings(org.opensearch.index.IndexSettings) AnalysisModule(org.opensearch.indices.analysis.AnalysisModule) Settings(org.opensearch.common.settings.Settings) IndexSettings(org.opensearch.index.IndexSettings)

Example 3 with AnalysisModule

use of org.opensearch.indices.analysis.AnalysisModule in project OpenSearch by opensearch-project.

the class AnalysisRegistryTests method setUp.

@Override
public void setUp() throws Exception {
    super.setUp();
    Settings settings = Settings.builder().put(Environment.PATH_HOME_SETTING.getKey(), createTempDir().toString()).build();
    emptyRegistry = emptyAnalysisRegistry(settings);
    // Module loaded to register in-built normalizers for testing
    AnalysisModule module = new AnalysisModule(TestEnvironment.newEnvironment(settings), singletonList(new MockAnalysisPlugin()));
    nonEmptyRegistry = module.getAnalysisRegistry();
}
Also used : AnalysisModule(org.opensearch.indices.analysis.AnalysisModule) Settings(org.opensearch.common.settings.Settings) IndexSettings(org.opensearch.index.IndexSettings)

Example 4 with AnalysisModule

use of org.opensearch.indices.analysis.AnalysisModule in project OpenSearch by opensearch-project.

the class AnalysisTestsHelper method createTestAnalysisFromSettings.

public static OpenSearchTestCase.TestAnalysis createTestAnalysisFromSettings(final Settings settings, final Path configPath, final AnalysisPlugin... plugins) throws IOException {
    final Settings actualSettings;
    if (settings.get(IndexMetadata.SETTING_VERSION_CREATED) == null) {
        actualSettings = Settings.builder().put(settings).put(IndexMetadata.SETTING_VERSION_CREATED, Version.CURRENT).build();
    } else {
        actualSettings = settings;
    }
    final IndexSettings indexSettings = IndexSettingsModule.newIndexSettings("test", actualSettings);
    final AnalysisRegistry analysisRegistry = new AnalysisModule(new Environment(actualSettings, configPath), Arrays.asList(plugins)).getAnalysisRegistry();
    return new OpenSearchTestCase.TestAnalysis(analysisRegistry.build(indexSettings), analysisRegistry.buildTokenFilterFactories(indexSettings), analysisRegistry.buildTokenizerFactories(indexSettings), analysisRegistry.buildCharFilterFactories(indexSettings));
}
Also used : IndexSettings(org.opensearch.index.IndexSettings) Environment(org.opensearch.env.Environment) AnalysisModule(org.opensearch.indices.analysis.AnalysisModule) IndexSettings(org.opensearch.index.IndexSettings) Settings(org.opensearch.common.settings.Settings)

Example 5 with AnalysisModule

use of org.opensearch.indices.analysis.AnalysisModule in project OpenSearch by opensearch-project.

the class OpenSearchTestCase method createTestAnalysis.

/**
 * Creates an TestAnalysis with all the default analyzers configured.
 */
public static TestAnalysis createTestAnalysis(IndexSettings indexSettings, Settings nodeSettings, AnalysisPlugin... analysisPlugins) throws IOException {
    Environment env = TestEnvironment.newEnvironment(nodeSettings);
    AnalysisModule analysisModule = new AnalysisModule(env, Arrays.asList(analysisPlugins));
    AnalysisRegistry analysisRegistry = analysisModule.getAnalysisRegistry();
    return new TestAnalysis(analysisRegistry.build(indexSettings), analysisRegistry.buildTokenFilterFactories(indexSettings), analysisRegistry.buildTokenizerFactories(indexSettings), analysisRegistry.buildCharFilterFactories(indexSettings));
}
Also used : AnalysisRegistry(org.opensearch.index.analysis.AnalysisRegistry) NodeEnvironment(org.opensearch.env.NodeEnvironment) Environment(org.opensearch.env.Environment) TestEnvironment(org.opensearch.env.TestEnvironment) AnalysisModule(org.opensearch.indices.analysis.AnalysisModule)

Aggregations

AnalysisModule (org.opensearch.indices.analysis.AnalysisModule)13 IndexSettings (org.opensearch.index.IndexSettings)12 Settings (org.opensearch.common.settings.Settings)11 IndexAnalyzers (org.opensearch.index.analysis.IndexAnalyzers)6 Environment (org.opensearch.env.Environment)5 TokenStream (org.apache.lucene.analysis.TokenStream)4 TestEnvironment (org.opensearch.env.TestEnvironment)4 NamedAnalyzer (org.opensearch.index.analysis.NamedAnalyzer)4 AnalysisProvider (org.opensearch.indices.analysis.AnalysisModule.AnalysisProvider)3 AnalysisPlugin (org.opensearch.plugins.AnalysisPlugin)3 HashMap (java.util.HashMap)2 MockTokenFilter (org.apache.lucene.analysis.MockTokenFilter)2 CharTermAttribute (org.apache.lucene.analysis.tokenattributes.CharTermAttribute)2 AnalysisRegistry (org.opensearch.index.analysis.AnalysisRegistry)2 TokenFilterFactory (org.opensearch.index.analysis.TokenFilterFactory)2 Script (org.opensearch.script.Script)2 ScriptContext (org.opensearch.script.ScriptContext)2 ScriptService (org.opensearch.script.ScriptService)2 IOException (java.io.IOException)1 Reader (java.io.Reader)1