Examples with WhitespaceTokenizer org.apache.flink.util.StringValueUtils.WhitespaceTokenizer used on opensource projects

Example 1 with WhitespaceTokenizer

use of org.apache.flink.util.StringValueUtils.WhitespaceTokenizer in project flink by apache.

the class StringValueUtilsTest method testTokenizerOnStringWithNexToken.

@Test
public void testTokenizerOnStringWithNexToken() {
    StringValue testString = new StringValue("test test");
    StringValueUtils.WhitespaceTokenizer tokenizer = new WhitespaceTokenizer();
    tokenizer.setStringToTokenize(testString);
    assertTrue(tokenizer.next(testString));
}

Example 2 with WhitespaceTokenizer

use of org.apache.flink.util.StringValueUtils.WhitespaceTokenizer in project flink by apache.

the class StringValueUtilsTest method testTokenizerOnStringOnlyWithDelimiter.

@Test
public void testTokenizerOnStringOnlyWithDelimiter() {
    StringValue testString = new StringValue("    ");
    StringValueUtils.WhitespaceTokenizer tokenizer = new WhitespaceTokenizer();
    tokenizer.setStringToTokenize(testString);
    assertFalse(tokenizer.next(testString));
}

Example 3 with WhitespaceTokenizer

use of org.apache.flink.util.StringValueUtils.WhitespaceTokenizer in project flink by apache.

the class StringValueUtilsTest method testTokenizerOnStringWithoutNexToken.

@Test
public void testTokenizerOnStringWithoutNexToken() {
    StringValue testString = new StringValue("test");
    StringValueUtils.WhitespaceTokenizer tokenizer = new WhitespaceTokenizer();
    tokenizer.setStringToTokenize(testString);
    // first token
    tokenizer.next(testString);
    // next token is not exist
    assertFalse(tokenizer.next(testString));
}