Examples with UuencodeFullTokenizer org.opensolaris.opengrok.analysis.uue.UuencodeFullTokenizer used on opensource projects

Example 1 with UuencodeFullTokenizer

use of org.opensolaris.opengrok.analysis.uue.UuencodeFullTokenizer in project OpenGrok by OpenGrok.

the class JFlexTokenizerTest method truncatedUuencodedFile.

/**
     * Truncated uuencoded files used to cause infinite loops. Verify that they
     * work now.
     *
     * @throws java.io.IOException
     */
@Test
public void truncatedUuencodedFile() throws IOException {
    UuencodeFullTokenizer tokenizer = new UuencodeFullTokenizer(new StringReader("begin 644 test\n"));
    CharTermAttribute term = tokenizer.addAttribute(CharTermAttribute.class);
    assertTrue(tokenizer.incrementToken());
    assertEquals("begin", term.toString());
    assertTrue(tokenizer.incrementToken());
    assertEquals("644", term.toString());
    assertTrue(tokenizer.incrementToken());
    assertEquals("test", term.toString());
    // This call used to hang forever.
    assertFalse(tokenizer.incrementToken());
}