Examples with PunctuationFilter com.twitter.common.text.filter.PunctuationFilter used on opensource projects

Example 1 with PunctuationFilter

use of com.twitter.common.text.filter.PunctuationFilter in project commons by twitter.

the class PunctuationExceptionCombinerTest method testPunctuationFilterDoesNotRemoveExceptionChars.

@Test
public void testPunctuationFilterDoesNotRemoveExceptionChars() {
    TwitterTokenStream stream = new PunctuationFilter(new PunctuationExceptionCombiner.Builder(tokenized).build());
    stream.reset("I .. exceptions!! ");
    assertEquals(ImmutableList.of("I", "exceptions"), stream.toStringList());
    stream.reset("I ♥♥ exceptions!!");
    assertEquals(ImmutableList.of("I", "♥♥", "exceptions"), stream.toStringList());
}