Examples with Scanner com.sun.tools.javac.parser.Scanner used on opensource projects

Example 1 with Scanner

use of com.sun.tools.javac.parser.Scanner in project error-prone by google.

the class ErrorProneTokens method getTokens.

/** Returns the tokens for the given source text, including comments. */
public static ImmutableList<ErrorProneToken> getTokens(String source, Context context) {
    if (source == null) {
        return ImmutableList.of();
    }
    ScannerFactory fac = ScannerFactory.instance(context);
    char[] buffer = source.toCharArray();
    Scanner scanner = new AccessibleScanner(fac, new CommentSavingTokenizer(fac, buffer, buffer.length));
    ImmutableList.Builder<ErrorProneToken> tokens = ImmutableList.builder();
    do {
        scanner.nextToken();
        tokens.add(new ErrorProneToken(scanner.token()));
    } while (scanner.token().kind != TokenKind.EOF);
    return tokens.build();
}