Examples with SimpleTokenizer com.yahoo.language.simple.SimpleTokenizer used on opensource projects

Example 1 with SimpleTokenizer

use of com.yahoo.language.simple.SimpleTokenizer in project vespa by vespa-engine.

the class StemmerImplTestCase method assertStem.

private static void assertStem(String input, List<String> expectedStems) {
    Stemmer stemmer = new StemmerImpl(new SimpleTokenizer(new SimpleNormalizer()));
    List<String> got = new ArrayList<>();
    for (StemList word : stemmer.stem(input, StemMode.ALL, Language.ENGLISH)) {
        got.add(word.get(0));
    }
    assertEquals(expectedStems, got);
}