use of org.antlr.v4.runtime.Recognizer.EOF in project antlr4 by tunnelvisionlabs.
the class DefaultErrorStrategy method getErrorRecoverySet.
/* Compute the error recovery set for the current rule. During
* rule invocation, the parser pushes the set of tokens that can
* follow that rule reference on the stack; this amounts to
* computing FIRST of what follows the rule reference in the
* enclosing rule. See LinearApproximator.FIRST().
* This local follow set only includes tokens
* from within the rule; i.e., the FIRST computation done by
* ANTLR stops at the end of a rule.
*
* EXAMPLE
*
* When you find a "no viable alt exception", the input is not
* consistent with any of the alternatives for rule r. The best
* thing to do is to consume tokens until you see something that
* can legally follow a call to r *or* any rule that called r.
* You don't want the exact set of viable next tokens because the
* input might just be missing a token--you might consume the
* rest of the input looking for one of the missing tokens.
*
* Consider grammar:
*
* a : '[' b ']'
* | '(' b ')'
* ;
* b : c '^' INT ;
* c : ID
* | INT
* ;
*
* At each rule invocation, the set of tokens that could follow
* that rule is pushed on a stack. Here are the various
* context-sensitive follow sets:
*
* FOLLOW(b1_in_a) = FIRST(']') = ']'
* FOLLOW(b2_in_a) = FIRST(')') = ')'
* FOLLOW(c_in_b) = FIRST('^') = '^'
*
* Upon erroneous input "[]", the call chain is
*
* a -> b -> c
*
* and, hence, the follow context stack is:
*
* depth follow set start of rule execution
* 0 <EOF> a (from main())
* 1 ']' b
* 2 '^' c
*
* Notice that ')' is not included, because b would have to have
* been called from a different context in rule a for ')' to be
* included.
*
* For error recovery, we cannot consider FOLLOW(c)
* (context-sensitive or otherwise). We need the combined set of
* all context-sensitive FOLLOW sets--the set of all tokens that
* could follow any reference in the call chain. We need to
* resync to one of those tokens. Note that FOLLOW(c)='^' and if
* we resync'd to that token, we'd consume until EOF. We need to
* sync to context-sensitive FOLLOWs for a, b, and c: {']','^'}.
* In this case, for input "[]", LA(1) is ']' and in the set, so we would
* not consume anything. After printing an error, rule c would
* return normally. Rule b would not find the required '^' though.
* At this point, it gets a mismatched token error and throws an
* exception (since LA(1) is not in the viable following token
* set). The rule exception handler tries to recover, but finds
* the same recovery set and doesn't consume anything. Rule b
* exits normally returning to rule a. Now it finds the ']' (and
* with the successful match exits errorRecovery mode).
*
* So, you can see that the parser walks up the call chain looking
* for the token that was a member of the recovery set.
*
* Errors are not generated in errorRecovery mode.
*
* ANTLR's error recovery mechanism is based upon original ideas:
*
* "Algorithms + Data Structures = Programs" by Niklaus Wirth
*
* and
*
* "A note on error recovery in recursive descent parsers":
* http://portal.acm.org/citation.cfm?id=947902.947905
*
* Later, Josef Grosch had some good ideas:
*
* "Efficient and Comfortable Error Recovery in Recursive Descent
* Parsers":
* ftp://www.cocolab.com/products/cocktail/doca4.ps/ell.ps.zip
*
* Like Grosch I implement context-sensitive FOLLOW sets that are combined
* at run-time upon error to avoid overhead during parsing.
*/
@NotNull
protected IntervalSet getErrorRecoverySet(@NotNull Parser recognizer) {
ATN atn = recognizer.getInterpreter().atn;
RuleContext ctx = recognizer._ctx;
IntervalSet recoverSet = new IntervalSet();
while (ctx != null && ctx.invokingState >= 0) {
// compute what follows who invoked us
ATNState invokingState = atn.states.get(ctx.invokingState);
RuleTransition rt = (RuleTransition) invokingState.transition(0);
IntervalSet follow = atn.nextTokens(rt.followState);
recoverSet.addAll(follow);
ctx = ctx.parent;
}
recoverSet.remove(Token.EPSILON);
// System.out.println("recover set "+recoverSet.toString(recognizer.getTokenNames()));
return recoverSet;
}
use of org.antlr.v4.runtime.Recognizer.EOF in project antlr4 by tunnelvisionlabs.
the class TestSymbolIssues method testStringLiteralRedefs.
@Test
public void testStringLiteralRedefs() throws Exception {
String grammar = "lexer grammar L;\n" + "A : 'a' ;\n" + "mode X;\n" + "B : 'a' ;\n" + "mode Y;\n" + "C : 'a' ;\n";
LexerGrammar g = new LexerGrammar(grammar);
String expectedTokenIDToTypeMap = "{EOF=-1, A=1, B=2, C=3}";
String expectedStringLiteralToTypeMap = "{}";
String expectedTypeToTokenList = "[A, B, C]";
assertEquals(expectedTokenIDToTypeMap, g.tokenNameToTypeMap.toString());
assertEquals(expectedStringLiteralToTypeMap, g.stringLiteralToTypeMap.toString());
assertEquals(expectedTypeToTokenList, realElements(g.typeToTokenList).toString());
}
use of org.antlr.v4.runtime.Recognizer.EOF in project groovy by apache.
the class DescriptiveErrorStrategy method createNoViableAlternativeErrorMessage.
protected String createNoViableAlternativeErrorMessage(Parser recognizer, NoViableAltException e) {
TokenStream tokens = recognizer.getInputStream();
String input;
if (tokens != null) {
if (e.getStartToken().getType() == Token.EOF) {
input = "<EOF>";
} else {
input = charStream.getText(Interval.of(e.getStartToken().getStartIndex(), e.getOffendingToken().getStopIndex()));
}
} else {
input = "<unknown input>";
}
return "Unexpected input: " + escapeWSAndQuote(input);
}
use of org.antlr.v4.runtime.Recognizer.EOF in project antlr4 by antlr.
the class ATN method getExpectedTokens.
/**
* Computes the set of input symbols which could follow ATN state number
* {@code stateNumber} in the specified full {@code context}. This method
* considers the complete parser context, but does not evaluate semantic
* predicates (i.e. all predicates encountered during the calculation are
* assumed true). If a path in the ATN exists from the starting state to the
* {@link RuleStopState} of the outermost context without matching any
* symbols, {@link Token#EOF} is added to the returned set.
*
* <p>If {@code context} is {@code null}, it is treated as {@link ParserRuleContext#EMPTY}.</p>
*
* Note that this does NOT give you the set of all tokens that could
* appear at a given token position in the input phrase. In other words,
* it does not answer:
*
* "Given a specific partial input phrase, return the set of all tokens
* that can follow the last token in the input phrase."
*
* The big difference is that with just the input, the parser could
* land right in the middle of a lookahead decision. Getting
* all *possible* tokens given a partial input stream is a separate
* computation. See https://github.com/antlr/antlr4/issues/1428
*
* For this function, we are specifying an ATN state and call stack to compute
* what token(s) can come next and specifically: outside of a lookahead decision.
* That is what you want for error reporting and recovery upon parse error.
*
* @param stateNumber the ATN state number
* @param context the full parse context
* @return The set of potentially valid input symbols which could follow the
* specified state in the specified context.
* @throws IllegalArgumentException if the ATN does not contain a state with
* number {@code stateNumber}
*/
public IntervalSet getExpectedTokens(int stateNumber, RuleContext context) {
if (stateNumber < 0 || stateNumber >= states.size()) {
throw new IllegalArgumentException("Invalid state number.");
}
RuleContext ctx = context;
ATNState s = states.get(stateNumber);
IntervalSet following = nextTokens(s);
if (!following.contains(Token.EPSILON)) {
return following;
}
IntervalSet expected = new IntervalSet();
expected.addAll(following);
expected.remove(Token.EPSILON);
while (ctx != null && ctx.invokingState >= 0 && following.contains(Token.EPSILON)) {
ATNState invokingState = states.get(ctx.invokingState);
RuleTransition rt = (RuleTransition) invokingState.transition(0);
following = nextTokens(rt.followState);
expected.addAll(following);
expected.remove(Token.EPSILON);
ctx = ctx.parent;
}
if (following.contains(Token.EPSILON)) {
expected.add(Token.EOF);
}
return expected;
}
use of org.antlr.v4.runtime.Recognizer.EOF in project antlr4 by antlr.
the class LL1Analyzer method LOOK.
/**
* Compute set of tokens that can follow {@code s} in the ATN in the
* specified {@code ctx}.
*
* <p>If {@code ctx} is {@code null} and the end of the rule containing
* {@code s} is reached, {@link Token#EPSILON} is added to the result set.
* If {@code ctx} is not {@code null} and the end of the outermost rule is
* reached, {@link Token#EOF} is added to the result set.</p>
*
* @param s the ATN state
* @param stopState the ATN state to stop at. This can be a
* {@link BlockEndState} to detect epsilon paths through a closure.
* @param ctx the complete parser context, or {@code null} if the context
* should be ignored
*
* @return The set of tokens that can follow {@code s} in the ATN in the
* specified {@code ctx}.
*/
public IntervalSet LOOK(ATNState s, ATNState stopState, RuleContext ctx) {
IntervalSet r = new IntervalSet();
// ignore preds; get all lookahead
boolean seeThruPreds = true;
PredictionContext lookContext = ctx != null ? PredictionContext.fromRuleContext(s.atn, ctx) : null;
_LOOK(s, stopState, lookContext, r, new HashSet<ATNConfig>(), new BitSet(), seeThruPreds, true);
return r;
}
Aggregations