Search in sources :

Example 41 with LexerInterpreter

use of org.antlr.v4.runtime.LexerInterpreter in project antlr4 by antlr.

the class TestTokenStreamRewriter method testCombineInsertOnLeftWithReplace.

@Test
public void testCombineInsertOnLeftWithReplace() throws Exception {
    LexerGrammar g = new LexerGrammar("lexer grammar T;\n" + "A : 'a';\n" + "B : 'b';\n" + "C : 'c';\n");
    String input = "abc";
    LexerInterpreter lexEngine = g.createLexerInterpreter(new ANTLRInputStream(input));
    CommonTokenStream stream = new CommonTokenStream(lexEngine);
    stream.fill();
    TokenStreamRewriter tokens = new TokenStreamRewriter(stream);
    tokens.replace(0, 2, "foo");
    tokens.insertBefore(0, "z");
    stream.fill();
    // combine with left edge of rewrite
    String result = tokens.getText();
    String expecting = "zfoo";
    assertEquals(expecting, result);
}
Also used : CommonTokenStream(org.antlr.v4.runtime.CommonTokenStream) LexerInterpreter(org.antlr.v4.runtime.LexerInterpreter) LexerGrammar(org.antlr.v4.tool.LexerGrammar) ANTLRInputStream(org.antlr.v4.runtime.ANTLRInputStream) TokenStreamRewriter(org.antlr.v4.runtime.TokenStreamRewriter) BaseJavaTest(org.antlr.v4.test.runtime.java.BaseJavaTest) Test(org.junit.Test)

Example 42 with LexerInterpreter

use of org.antlr.v4.runtime.LexerInterpreter in project antlr4 by antlr.

the class TestTokenStreamRewriter method testReplaceRangeThenInsertAtLeftEdge.

@Test
public void testReplaceRangeThenInsertAtLeftEdge() throws Exception {
    LexerGrammar g = new LexerGrammar("lexer grammar T;\n" + "A : 'a';\n" + "B : 'b';\n" + "C : 'c';\n");
    String input = "abcccba";
    LexerInterpreter lexEngine = g.createLexerInterpreter(new ANTLRInputStream(input));
    CommonTokenStream stream = new CommonTokenStream(lexEngine);
    stream.fill();
    TokenStreamRewriter tokens = new TokenStreamRewriter(stream);
    tokens.replace(2, 4, "x");
    tokens.insertBefore(2, "y");
    String result = tokens.getText();
    String expecting = "abyxba";
    assertEquals(expecting, result);
}
Also used : CommonTokenStream(org.antlr.v4.runtime.CommonTokenStream) LexerInterpreter(org.antlr.v4.runtime.LexerInterpreter) LexerGrammar(org.antlr.v4.tool.LexerGrammar) ANTLRInputStream(org.antlr.v4.runtime.ANTLRInputStream) TokenStreamRewriter(org.antlr.v4.runtime.TokenStreamRewriter) BaseJavaTest(org.antlr.v4.test.runtime.java.BaseJavaTest) Test(org.junit.Test)

Example 43 with LexerInterpreter

use of org.antlr.v4.runtime.LexerInterpreter in project antlr4 by antlr.

the class TestTokenStreamRewriter method testDropIdenticalReplace.

@Test
public void testDropIdenticalReplace() throws Exception {
    LexerGrammar g = new LexerGrammar("lexer grammar T;\n" + "A : 'a';\n" + "B : 'b';\n" + "C : 'c';\n");
    String input = "abcc";
    LexerInterpreter lexEngine = g.createLexerInterpreter(new ANTLRInputStream(input));
    CommonTokenStream stream = new CommonTokenStream(lexEngine);
    stream.fill();
    TokenStreamRewriter tokens = new TokenStreamRewriter(stream);
    tokens.replace(1, 2, "foo");
    tokens.replace(1, 2, "foo");
    stream.fill();
    // drop previous, identical
    String result = tokens.getText();
    String expecting = "afooc";
    assertEquals(expecting, result);
}
Also used : CommonTokenStream(org.antlr.v4.runtime.CommonTokenStream) LexerInterpreter(org.antlr.v4.runtime.LexerInterpreter) LexerGrammar(org.antlr.v4.tool.LexerGrammar) ANTLRInputStream(org.antlr.v4.runtime.ANTLRInputStream) TokenStreamRewriter(org.antlr.v4.runtime.TokenStreamRewriter) BaseJavaTest(org.antlr.v4.test.runtime.java.BaseJavaTest) Test(org.junit.Test)

Example 44 with LexerInterpreter

use of org.antlr.v4.runtime.LexerInterpreter in project antlr4 by antlr.

the class TestTokenStreamRewriter method test2InsertBeforeAfterMiddleIndex.

@Test
public void test2InsertBeforeAfterMiddleIndex() throws Exception {
    LexerGrammar g = new LexerGrammar("lexer grammar T;\n" + "A : 'a';\n" + "B : 'b';\n" + "C : 'c';\n");
    String input = "abc";
    LexerInterpreter lexEngine = g.createLexerInterpreter(new ANTLRInputStream(input));
    CommonTokenStream stream = new CommonTokenStream(lexEngine);
    stream.fill();
    TokenStreamRewriter tokens = new TokenStreamRewriter(stream);
    tokens.insertBefore(1, "x");
    tokens.insertAfter(1, "x");
    String result = tokens.getText();
    String expecting = "axbxc";
    assertEquals(expecting, result);
}
Also used : CommonTokenStream(org.antlr.v4.runtime.CommonTokenStream) LexerInterpreter(org.antlr.v4.runtime.LexerInterpreter) LexerGrammar(org.antlr.v4.tool.LexerGrammar) ANTLRInputStream(org.antlr.v4.runtime.ANTLRInputStream) TokenStreamRewriter(org.antlr.v4.runtime.TokenStreamRewriter) BaseJavaTest(org.antlr.v4.test.runtime.java.BaseJavaTest) Test(org.junit.Test)

Example 45 with LexerInterpreter

use of org.antlr.v4.runtime.LexerInterpreter in project antlr4 by antlr.

the class TestTokenStreamRewriter method testInsertAfterLastIndex.

@Test
public void testInsertAfterLastIndex() throws Exception {
    LexerGrammar g = new LexerGrammar("lexer grammar T;\n" + "A : 'a';\n" + "B : 'b';\n" + "C : 'c';\n");
    String input = "abc";
    LexerInterpreter lexEngine = g.createLexerInterpreter(new ANTLRInputStream(input));
    CommonTokenStream stream = new CommonTokenStream(lexEngine);
    stream.fill();
    TokenStreamRewriter tokens = new TokenStreamRewriter(stream);
    tokens.insertAfter(2, "x");
    String result = tokens.getText();
    String expecting = "abcx";
    assertEquals(expecting, result);
}
Also used : CommonTokenStream(org.antlr.v4.runtime.CommonTokenStream) LexerInterpreter(org.antlr.v4.runtime.LexerInterpreter) LexerGrammar(org.antlr.v4.tool.LexerGrammar) ANTLRInputStream(org.antlr.v4.runtime.ANTLRInputStream) TokenStreamRewriter(org.antlr.v4.runtime.TokenStreamRewriter) BaseJavaTest(org.antlr.v4.test.runtime.java.BaseJavaTest) Test(org.junit.Test)

Aggregations

LexerInterpreter (org.antlr.v4.runtime.LexerInterpreter)62 ANTLRInputStream (org.antlr.v4.runtime.ANTLRInputStream)59 Test (org.junit.Test)54 LexerGrammar (org.antlr.v4.tool.LexerGrammar)53 CommonTokenStream (org.antlr.v4.runtime.CommonTokenStream)52 TokenStreamRewriter (org.antlr.v4.runtime.TokenStreamRewriter)43 BaseJavaTest (org.antlr.v4.test.runtime.java.BaseJavaTest)43 CharStream (org.antlr.v4.runtime.CharStream)11 Token (org.antlr.v4.runtime.Token)6 TokenStream (org.antlr.v4.runtime.TokenStream)6 ParseTree (org.antlr.v4.runtime.tree.ParseTree)6 GrammarParserInterpreter (org.antlr.v4.tool.GrammarParserInterpreter)6 BufferedTokenStream (org.antlr.v4.runtime.BufferedTokenStream)5 StringReader (java.io.StringReader)4 ParserInterpreter (org.antlr.v4.runtime.ParserInterpreter)3 ParserRuleContext (org.antlr.v4.runtime.ParserRuleContext)3 ArrayList (java.util.ArrayList)2 DecisionInfo (org.antlr.v4.runtime.atn.DecisionInfo)2 Grammar (org.antlr.v4.tool.Grammar)2 ByteArrayInputStream (java.io.ByteArrayInputStream)1