Search in sources :

Example 1 with SimpleNormalizer

use of com.yahoo.language.simple.SimpleNormalizer in project vespa by vespa-engine.

the class StemmerImplTestCase method assertStem.

private static void assertStem(String input, List<String> expectedStems) {
    Stemmer stemmer = new StemmerImpl(new SimpleTokenizer(new SimpleNormalizer()));
    List<String> got = new ArrayList<>();
    for (StemList word : stemmer.stem(input, StemMode.ALL, Language.ENGLISH)) {
        got.add(word.get(0));
    }
    assertEquals(expectedStems, got);
}
Also used : SimpleTokenizer(com.yahoo.language.simple.SimpleTokenizer) ArrayList(java.util.ArrayList) SimpleNormalizer(com.yahoo.language.simple.SimpleNormalizer)

Aggregations

SimpleNormalizer (com.yahoo.language.simple.SimpleNormalizer)1 SimpleTokenizer (com.yahoo.language.simple.SimpleTokenizer)1 ArrayList (java.util.ArrayList)1