Examples with DFA org.antlr.v4.runtime.dfa.DFA used on opensource projects

Example 36 with DFA

use of org.antlr.v4.runtime.dfa.DFA in project checkstyle by checkstyle.

the class JavaAstVisitorTest method testNoStackOverflowOnDeepStringConcat.

/**
 * This test exists to kill surviving mutation from pitest removing expression AST building
 * optimization in {@link JavaAstVisitor#visitBinOp(JavaLanguageParser.BinOpContext)}.
 * We do not use {@link JavaParser#parse(FileContents)} here due to DFA clearing hack.
 *
 * <p>
 * Reason: we have iterative expression AST building to avoid stackoverflow
 * in {@link JavaAstVisitor#visitBinOp(JavaLanguageParser.BinOpContext)}. In actual
 * generated parser, we avoid stackoverflow thanks to the left recursive expression
 * rule (eliminating unnecessary recursive calls to hierarchical expression production rules).
 * However, ANTLR's ParserATNSimulator has no such optimization. So, the number of recursive
 * calls to ParserATNSimulator#closure when calling ParserATNSimulator#clearDFA causes a
 * StackOverflow error. We avoid this by using the single argument constructor (thus not
 * forcing DFA clearing) in this test.
 * </p>
 *
 * @throws Exception if input file does not exist
 */
@Test
public void testNoStackOverflowOnDeepStringConcat() throws Exception {
    final File file = new File(getPath("InputJavaAstVisitorNoStackOverflowOnDeepStringConcat.java"));
    final FileText fileText = new FileText(file, StandardCharsets.UTF_8.name());
    final FileContents contents = new FileContents(fileText);
    final String fullText = contents.getText().getFullText().toString();
    final CharStream codePointCharStream = CharStreams.fromString(fullText);
    final JavaLanguageLexer lexer = new JavaLanguageLexer(codePointCharStream, true);
    lexer.setCommentListener(contents);
    final CommonTokenStream tokenStream = new CommonTokenStream(lexer);
    final JavaLanguageParser parser = new JavaLanguageParser(tokenStream);
    final JavaLanguageParser.CompilationUnitContext compilationUnit = parser.compilationUnit();
    // We restrict execution to use limited resources here, so that we can
    // kill surviving pitest mutation from removal of nested binary operation
    // optimization in JavaAstVisitor#visitBinOp. Limited resources (small stack size)
    // ensure that we throw a StackOverflowError if optimization is removed.
    final DetailAST root = TestUtil.getResultWithLimitedResources(() -> new JavaAstVisitor(tokenStream).visit(compilationUnit));
    assertWithMessage("File parsing and AST building should complete successfully.").that(root).isNotNull();
}

Example 37 with DFA

use of org.antlr.v4.runtime.dfa.DFA in project antlr4 by antlr.

the class ParserATNSimulator method execATN.

/**
 * Performs ATN simulation to compute a predicted alternative based
 *  upon the remaining input, but also updates the DFA cache to avoid
 *  having to traverse the ATN again for the same input sequence.
 *
 *	 There are some key conditions we're looking for after computing a new
 *	 set of ATN configs (proposed DFA state):
 * if the set is empty, there is no viable alternative for current symbol
 * does the state uniquely predict an alternative?
 * does the state have a conflict that would prevent us from
 *	         putting it on the work list?
 *
 *	 We also have some key operations to do:
 * add an edge from previous DFA state to potentially new DFA state, D,
 *	         upon current symbol but only if adding to work list, which means in all
 *	         cases except no viable alternative (and possibly non-greedy decisions?)
 * collecting predicates and adding semantic context to DFA accept states
 * adding rule context to context-sensitive DFA accept states
 * consuming an input symbol
 * reporting a conflict
 * reporting an ambiguity
 * reporting a context sensitivity
 * reporting insufficient predicates
 *
 *	 cover these cases:
 *	    dead end
 *	    single alt
 *	    single alt + preds
 *	    conflict
 *	    conflict + preds
 */
protected int execATN(DFA dfa, DFAState s0, TokenStream input, int startIndex, ParserRuleContext outerContext) {
    if (debug || debug_list_atn_decisions) {
        System.out.println("execATN decision " + dfa.decision + " exec LA(1)==" + getLookaheadName(input) + " line " + input.LT(1).getLine() + ":" + input.LT(1).getCharPositionInLine());
    }
    DFAState previousD = s0;
    if (debug)
        System.out.println("s0 = " + s0);
    int t = input.LA(1);
    while (true) {
        // while more work
        DFAState D = getExistingTargetState(previousD, t);
        if (D == null) {
            D = computeTargetState(dfa, previousD, t);
        }
        if (D == ERROR) {
            // if any configs in previous dipped into outer context, that
            // means that input up to t actually finished entry rule
            // at least for SLL decision. Full LL doesn't dip into outer
            // so don't need special case.
            // We will get an error no matter what so delay until after
            // decision; better error message. Also, no reachable target
            // ATN states in SLL implies LL will also get nowhere.
            // If conflict in states that dip out, choose min since we
            // will get error no matter what.
            NoViableAltException e = noViableAlt(input, outerContext, previousD.configs, startIndex);
            input.seek(startIndex);
            int alt = getSynValidOrSemInvalidAltThatFinishedDecisionEntryRule(previousD.configs, outerContext);
            if (alt != ATN.INVALID_ALT_NUMBER) {
                return alt;
            }
            throw e;
        }
        if (D.requiresFullContext && mode != PredictionMode.SLL) {
            // IF PREDS, MIGHT RESOLVE TO SINGLE ALT => SLL (or syntax error)
            BitSet conflictingAlts = D.configs.conflictingAlts;
            if (D.predicates != null) {
                if (debug)
                    System.out.println("DFA state has preds in DFA sim LL failover");
                int conflictIndex = input.index();
                if (conflictIndex != startIndex) {
                    input.seek(startIndex);
                }
                conflictingAlts = evalSemanticContext(D.predicates, outerContext, true);
                if (conflictingAlts.cardinality() == 1) {
                    if (debug)
                        System.out.println("Full LL avoided");
                    return conflictingAlts.nextSetBit(0);
                }
                if (conflictIndex != startIndex) {
                    // restore the index so reporting the fallback to full
                    // context occurs with the index at the correct spot
                    input.seek(conflictIndex);
                }
            }
            if (dfa_debug)
                System.out.println("ctx sensitive state " + outerContext + " in " + D);
            boolean fullCtx = true;
            ATNConfigSet s0_closure = computeStartState(dfa.atnStartState, outerContext, fullCtx);
            reportAttemptingFullContext(dfa, conflictingAlts, D.configs, startIndex, input.index());
            int alt = execATNWithFullContext(dfa, D, s0_closure, input, startIndex, outerContext);
            return alt;
        }
        if (D.isAcceptState) {
            if (D.predicates == null) {
                return D.prediction;
            }
            int stopIndex = input.index();
            input.seek(startIndex);
            BitSet alts = evalSemanticContext(D.predicates, outerContext, true);
            switch(alts.cardinality()) {
                case 0:
                    throw noViableAlt(input, outerContext, D.configs, startIndex);
                case 1:
                    return alts.nextSetBit(0);
                default:
                    // report ambiguity after predicate evaluation to make sure the correct
                    // set of ambig alts is reported.
                    reportAmbiguity(dfa, D, startIndex, stopIndex, false, alts, D.configs);
                    return alts.nextSetBit(0);
            }
        }
        previousD = D;
        if (t != IntStream.EOF) {
            input.consume();
            t = input.LA(1);
        }
    }
}

Example 38 with DFA

use of org.antlr.v4.runtime.dfa.DFA in project antlr4 by antlr.

the class ParserATNSimulator method reportAmbiguity.

/**
 * If context sensitive parsing, we know it's ambiguity not conflict
 */
protected void reportAmbiguity(DFA dfa, // the DFA state from execATN() that had SLL conflicts
DFAState D, int startIndex, int stopIndex, boolean exact, BitSet ambigAlts, // configs that LL not SLL considered conflicting
ATNConfigSet configs) {
    if (debug || retry_debug) {
        Interval interval = Interval.of(startIndex, stopIndex);
        System.out.println("reportAmbiguity " + ambigAlts + ":" + configs + ", input=" + parser.getTokenStream().getText(interval));
    }
    if (parser != null)
        parser.getErrorListenerDispatch().reportAmbiguity(parser, dfa, startIndex, stopIndex, exact, ambigAlts, configs);
}

Example 39 with DFA

use of org.antlr.v4.runtime.dfa.DFA in project antlr4 by antlr.

the class ParserATNSimulator method adaptivePredict.

public int adaptivePredict(TokenStream input, int decision, ParserRuleContext outerContext) {
    if (debug || debug_list_atn_decisions) {
        System.out.println("adaptivePredict decision " + decision + " exec LA(1)==" + getLookaheadName(input) + " line " + input.LT(1).getLine() + ":" + input.LT(1).getCharPositionInLine());
    }
    _input = input;
    _startIndex = input.index();
    _outerContext = outerContext;
    DFA dfa = decisionToDFA[decision];
    _dfa = dfa;
    int m = input.mark();
    int index = _startIndex;
    // But, do we still need an initial state?
    try {
        DFAState s0;
        if (dfa.isPrecedenceDfa()) {
            // the start state for a precedence DFA depends on the current
            // parser precedence, and is provided by a DFA method.
            s0 = dfa.getPrecedenceStartState(parser.getPrecedence());
        } else {
            // the start state for a "regular" DFA is just s0
            s0 = dfa.s0;
        }
        if (s0 == null) {
            if (outerContext == null)
                outerContext = ParserRuleContext.EMPTY;
            if (debug || debug_list_atn_decisions) {
                System.out.println("predictATN decision " + dfa.decision + " exec LA(1)==" + getLookaheadName(input) + ", outerContext=" + outerContext.toString(parser));
            }
            boolean fullCtx = false;
            ATNConfigSet s0_closure = computeStartState(dfa.atnStartState, ParserRuleContext.EMPTY, fullCtx);
            if (dfa.isPrecedenceDfa()) {
                /* If this is a precedence DFA, we use applyPrecedenceFilter
					 * to convert the computed start state to a precedence start
					 * state. We then use DFA.setPrecedenceStartState to set the
					 * appropriate start state for the precedence level rather
					 * than simply setting DFA.s0.
					 */
                // not used for prediction but useful to know start configs anyway
                dfa.s0.configs = s0_closure;
                s0_closure = applyPrecedenceFilter(s0_closure);
                s0 = addDFAState(dfa, new DFAState(s0_closure));
                dfa.setPrecedenceStartState(parser.getPrecedence(), s0);
            } else {
                s0 = addDFAState(dfa, new DFAState(s0_closure));
                dfa.s0 = s0;
            }
        }
        int alt = execATN(dfa, s0, input, index, outerContext);
        if (debug)
            System.out.println("DFA after predictATN: " + dfa.toString(parser.getVocabulary()));
        return alt;
    } finally {
        // wack cache after each prediction
        mergeCache = null;
        _dfa = null;
        input.seek(index);
        input.release(m);
    }
}

Example 40 with DFA

use of org.antlr.v4.runtime.dfa.DFA in project antlr4 by antlr.

the class ParserATNSimulator method reportAttemptingFullContext.

protected void reportAttemptingFullContext(DFA dfa, BitSet conflictingAlts, ATNConfigSet configs, int startIndex, int stopIndex) {
    if (debug || retry_debug) {
        Interval interval = Interval.of(startIndex, stopIndex);
        System.out.println("reportAttemptingFullContext decision=" + dfa.decision + ":" + configs + ", input=" + parser.getTokenStream().getText(interval));
    }
    if (parser != null)
        parser.getErrorListenerDispatch().reportAttemptingFullContext(parser, dfa, startIndex, stopIndex, conflictingAlts, configs);
}