Examples with WhitespaceTokenizer org.apache.flink.util.SimpleStringUtils.WhitespaceTokenizer used on opensource projects

Example 1 with WhitespaceTokenizer

use of org.apache.flink.util.SimpleStringUtils.WhitespaceTokenizer in project flink by apache.

the class SimpleStringUtilsTest method testTokenizerOnStringWithNexToken.

@Test
public void testTokenizerOnStringWithNexToken() {
    StringValue testString = new StringValue("test test");
    SimpleStringUtils.WhitespaceTokenizer tokenizer = new WhitespaceTokenizer();
    tokenizer.setStringToTokenize(testString);
    assertTrue(tokenizer.next(testString));
}

Example 2 with WhitespaceTokenizer

use of org.apache.flink.util.SimpleStringUtils.WhitespaceTokenizer in project flink by apache.

the class SimpleStringUtilsTest method testTokenizerOnStringWithoutNexToken.

@Test
public void testTokenizerOnStringWithoutNexToken() {
    StringValue testString = new StringValue("test");
    SimpleStringUtils.WhitespaceTokenizer tokenizer = new WhitespaceTokenizer();
    tokenizer.setStringToTokenize(testString);
    //first token
    tokenizer.next(testString);
    //next token is not exist
    assertFalse(tokenizer.next(testString));
}

Example 3 with WhitespaceTokenizer

use of org.apache.flink.util.SimpleStringUtils.WhitespaceTokenizer in project flink by apache.

the class SimpleStringUtilsTest method testTokenizerOnStringOnlyWithDelimiter.

@Test
public void testTokenizerOnStringOnlyWithDelimiter() {
    StringValue testString = new StringValue("    ");
    SimpleStringUtils.WhitespaceTokenizer tokenizer = new WhitespaceTokenizer();
    tokenizer.setStringToTokenize(testString);
    assertFalse(tokenizer.next(testString));
}