Examples with NgramAnalyzer com.apple.foundationdb.record.lucene.ngram.NgramAnalyzer used on opensource projects

Example 1 with NgramAnalyzer

use of com.apple.foundationdb.record.lucene.ngram.NgramAnalyzer in project fdb-record-layer by FoundationDB.

the class LuceneAnalyzerTest method testNgramAnalyzerWithStopWords.

@Test
void testNgramAnalyzerWithStopWords() throws Exception {
    final CharArraySet stopSet = new CharArraySet(List.of("hello"), false);
    CharArraySet stopWords = CharArraySet.unmodifiableSet(stopSet);
    String input = "hello RL";
    Collection<String> result = new HashSet<>();
    tokenizeWithAnalyzer(result, input, new NgramAnalyzer(stopWords, 3, 10, false));
    Assertions.assertEquals(ImmutableSet.of("rl"), result);
}

Example 2 with NgramAnalyzer

use of com.apple.foundationdb.record.lucene.ngram.NgramAnalyzer in project fdb-record-layer by FoundationDB.

the class LuceneAnalyzerTest method testEdgesOnlyNgramAnalyzer.

@Test
void testEdgesOnlyNgramAnalyzer() throws Exception {
    String input = "hello RL";
    Collection<String> result = new HashSet<>();
    tokenizeWithAnalyzer(result, input, new NgramAnalyzer(null, 3, 10, true));
    Assertions.assertEquals(ImmutableSet.of("hel", "hell", "hello", "rl"), result);
}

Example 3 with NgramAnalyzer

use of com.apple.foundationdb.record.lucene.ngram.NgramAnalyzer in project fdb-record-layer by FoundationDB.

the class LuceneAnalyzerTest method testNgramAnalyzer.

@Test
void testNgramAnalyzer() throws Exception {
    String input = "hello RL";
    Collection<String> result = new HashSet<>();
    tokenizeWithAnalyzer(result, input, new NgramAnalyzer(null, 3, 10, false));
    Assertions.assertEquals(ImmutableSet.of("hel", "ell", "llo", "hell", "ello", "hello", "rl"), result);
}