Examples with EofNode com.google.auto.value.processor.escapevelocity.TokenNode.EofNode used on opensource projects

Example 1 with EofNode

use of com.google.auto.value.processor.escapevelocity.TokenNode.EofNode in project auto by google.

the class Parser method parse.

/**
   * Parse the input completely to produce a {@link Template}.
   *
   * <p>Parsing happens in two phases. First, we parse a sequence of "tokens", where tokens include
   * entire references such as <pre>
   *    ${x.foo()[23]}
   * </pre>or entire directives such as<pre>
   *    #set ($x = $y + $z)
   * </pre>But tokens do not span complex constructs. For example,<pre>
   *    #if ($x == $y) something #end
   * </pre>is three tokens:<pre>
   *    #if ($x == $y)
   *    (literal text " something ")
   *   #end
   * </pre>
   *
   * <p>The second phase then takes the sequence of tokens and constructs a parse tree out of it.
   * Some nodes in the parse tree will be unchanged from the token sequence, such as the <pre>
   *    ${x.foo()[23]}
   *    #set ($x = $y + $z)
   * </pre> examples above. But a construct such as the {@code #if ... #end} mentioned above will
   * become a single IfNode in the parse tree in the second phase.
   *
   * <p>The main reason for this approach is that Velocity has two kinds of lexical contexts. At the
   * top level, there can be arbitrary literal text; references like <code>${x.foo()}</code>; and
   * directives like {@code #if} or {@code #set}. Inside the parentheses of a directive, however,
   * neither arbitrary text nor directives can appear, but expressions can, so we need to tokenize
   * the inside of <pre>
   *    #if ($x == $a + $b)
   * </pre> as the five tokens "$x", "==", "$a", "+", "$b". Rather than having a classical
   * parser/lexer combination, where the lexer would need to switch between these two modes, we
   * replace the lexer with an ad-hoc parser that is the first phase described above, and we
   * define a simple parser over the resultant tokens that is the second phase.
   */
Template parse() throws IOException {
    ImmutableList.Builder<Node> tokens = ImmutableList.builder();
    Node token;
    do {
        token = parseNode();
        tokens.add(token);
    } while (!(token instanceof EofNode));
    return new Reparser(tokens.build()).reparse();
}

Example 2 with EofNode

use of com.google.auto.value.processor.escapevelocity.TokenNode.EofNode in project auto by google.

the class Reparser method reparse.

Template reparse() {
    Node root = parseTo(EOF_SET, new EofNode(1));
    linkMacroCalls();
    return new Template(root);
}

Example 3 with EofNode

use of com.google.auto.value.processor.escapevelocity.TokenNode.EofNode in project auto by google.

the class Reparser method removeSpaceBeforeSet.

/**
   * Returns a copy of the given list where spaces have been moved where appropriate after {@code
   * #set}. This hack is needed to match what appears to be special treatment in Apache Velocity of
   * spaces before {@code #set} directives. If you have <i>thing</i> <i>whitespace</i> {@code #set},
   * then the whitespace is deleted if the <i>thing</i> is a comment ({@code ##...\n}); a reference
   * ({@code $x} or {@code $x.foo} etc); a macro definition; or another {@code #set}.
   */
private static ImmutableList<Node> removeSpaceBeforeSet(ImmutableList<Node> nodes) {
    assert Iterables.getLast(nodes) instanceof EofNode;
    // Since the last node is EofNode, the i + 1 and i + 2 accesses below are safe.
    ImmutableList.Builder<Node> newNodes = ImmutableList.builder();
    for (int i = 0; i < nodes.size(); i++) {
        Node nodeI = nodes.get(i);
        newNodes.add(nodeI);
        if (shouldDeleteSpaceBetweenThisAndSet(nodeI) && isWhitespaceLiteral(nodes.get(i + 1)) && nodes.get(i + 2) instanceof SetNode) {
            // Skip the space.
            i++;
        }
    }
    return newNodes.build();
}

Example 4 with EofNode

use of com.google.auto.value.processor.escapevelocity.TokenNode.EofNode in project auto by google.

the class Reparser method parseTo.

/**
   * Parse subtrees until one of the token types in {@code stopSet} is encountered.
   * If this is the top level, {@code stopSet} will include {@link EofNode} so parsing will stop
   * when it reaches the end of the input. Otherwise, if an {@code EofNode} is encountered it is an
   * error because we have something like {@code #if} without {@code #end}.
   *
   * @param stopSet the kinds of tokens that will stop the parse. For example, if we are parsing
   *     after an {@code #if}, we will stop at any of {@code #else}, {@code #elseif},
   *     or {@code #end}.
   * @param forWhat the token that triggered this call, for example the {@code #if} whose
   *     {@code #end} etc we are looking for.
   *
   * @return a Node that is the concatenation of the parsed subtrees
   */
private Node parseTo(Set<Class<? extends TokenNode>> stopSet, TokenNode forWhat) {
    ImmutableList.Builder<Node> nodeList = ImmutableList.builder();
    while (true) {
        Node currentNode = currentNode();
        if (stopSet.contains(currentNode.getClass())) {
            break;
        }
        if (currentNode instanceof EofNode) {
            throw new ParseException("Reached end of file while parsing " + forWhat.name(), forWhat.lineNumber);
        }
        Node parsed;
        if (currentNode instanceof TokenNode) {
            parsed = parseTokenNode();
        } else {
            parsed = currentNode;
            nextNode();
        }
        nodeList.add(parsed);
    }
    return Node.cons(forWhat.lineNumber, nodeList.build());
}