Examples with GraqlLexer ai.grakn.graql.internal.antlr.GraqlLexer used on opensource projects

Example 1 with GraqlLexer

use of ai.grakn.graql.internal.antlr.GraqlLexer in project grakn by graknlabs.

the class QueryParserImpl method createLexer.

private static GraqlLexer createLexer(CharStream input, GraqlErrorListener errorListener) {
    GraqlLexer lexer = new GraqlLexer(input);
    lexer.removeErrorListeners();
    lexer.addErrorListener(errorListener);
    return lexer;
}

Example 2 with GraqlLexer

use of ai.grakn.graql.internal.antlr.GraqlLexer in project grakn by graknlabs.

the class QueryParserImpl method parseList.

/**
 * @param reader a reader representing several queries
 * @return a list of queries
 */
@Override
public <T extends Query<?>> Stream<T> parseList(Reader reader) {
    UnbufferedCharStream charStream = new UnbufferedCharStream(reader);
    GraqlErrorListener errorListener = GraqlErrorListener.withoutQueryString();
    GraqlLexer lexer = createLexer(charStream, errorListener);
    /*
            We tell the lexer to copy the text into each generated token.
            Normally when calling `Token#getText`, it will look into the underlying `TokenStream` and call
            `TokenStream#size` to check it is in-bounds. However, `UnbufferedTokenStream#size` is not supported
            (because then it would have to read the entire input). To avoid this issue, we set this flag which will
            copy over the text into each `Token`, s.t. that `Token#getText` will just look up the copied text field.
        */
    lexer.setTokenFactory(new CommonTokenFactory(true));
    // Use an unbuffered token stream so we can handle extremely large input strings
    UnbufferedTokenStream tokenStream = new UnbufferedTokenStream(ChannelTokenSource.of(lexer));
    GraqlParser parser = createParser(tokenStream, errorListener);
    /*
            The "bail" error strategy prevents us reading all the way to the end of the input, e.g.

            ```
            match $x isa person; insert $x has name "Bob"; match $x isa movie; get;
                                                           ^
            ```

            In this example, when ANTLR reaches the indicated `match`, it considers two possibilities:

            1. this is the end of the query
            2. the user has made a mistake. Maybe they accidentally pasted the `match` here.

            Because of case 2, ANTLR will parse beyond the `match` in order to produce a more helpful error message.
            This causes memory issues for very large queries, so we use the simpler "bail" strategy that will
            immediately stop when it hits `match`.
        */
    parser.setErrorHandler(new BailErrorStrategy());
    // This is a lazy iterator that will only consume a single query at a time, without parsing any further.
    // This means it can pass arbitrarily long streams of queries in constant memory!
    Iterable<T> queryIterator = () -> new AbstractIterator<T>() {

        @Nullable
        @Override
        protected T computeNext() {
            int latestToken = tokenStream.LA(1);
            if (latestToken == Token.EOF) {
                endOfData();
                return null;
            } else {
                // When we next run it, it will start where it left off in the stream
                return (T) QUERY.parse(parser, errorListener);
            }
        }
    };
    return StreamSupport.stream(queryIterator.spliterator(), false);
}

Example 3 with GraqlLexer

use of ai.grakn.graql.internal.antlr.GraqlLexer in project grakn by graknlabs.

the class Autocomplete method getTokens.

/**
 * @param query a graql query
 * @return a list of tokens from running the lexer on the query
 */
private static List<? extends Token> getTokens(String query) {
    ANTLRInputStream input = new ANTLRInputStream(query);
    GraqlLexer lexer = new GraqlLexer(input);
    // Ignore syntax errors
    lexer.removeErrorListeners();
    lexer.addErrorListener(new BaseErrorListener());
    return lexer.getAllTokens();
}