Examples with TokenStreamHiddenTokenFilter antlr.TokenStreamHiddenTokenFilter used on opensource projects

Example 1 with TokenStreamHiddenTokenFilter

use of antlr.TokenStreamHiddenTokenFilter in project checkstyle by checkstyle.

the class TreeWalker method parse.

/**
     * Static helper method to parses a Java source file.
     *
     * @param contents
     *                contains the contents of the file
     * @return the root of the AST
     * @throws TokenStreamException
     *                 if lexing failed
     * @throws RecognitionException
     *                 if parsing failed
     */
public static DetailAST parse(FileContents contents) throws RecognitionException, TokenStreamException {
    final String fullText = contents.getText().getFullText().toString();
    final Reader reader = new StringReader(fullText);
    final GeneratedJavaLexer lexer = new GeneratedJavaLexer(reader);
    lexer.setFilename(contents.getFileName());
    lexer.setCommentListener(contents);
    lexer.setTreatAssertAsKeyword(true);
    lexer.setTreatEnumAsKeyword(true);
    lexer.setTokenObjectClass("antlr.CommonHiddenStreamToken");
    final TokenStreamHiddenTokenFilter filter = new TokenStreamHiddenTokenFilter(lexer);
    filter.hide(TokenTypes.SINGLE_LINE_COMMENT);
    filter.hide(TokenTypes.BLOCK_COMMENT_BEGIN);
    final GeneratedJavaRecognizer parser = new GeneratedJavaRecognizer(filter);
    parser.setFilename(contents.getFileName());
    parser.setASTNodeClass(DetailAST.class.getName());
    parser.compilationUnit();
    return (DetailAST) parser.getAST();
}

Example 2 with TokenStreamHiddenTokenFilter

use of antlr.TokenStreamHiddenTokenFilter in project cxf by apache.

the class IDLProcessor method process.

public void process() throws ToolException {
    String location = env.get(ToolCorbaConstants.CFG_IDLFILE).toString();
    File file = new File(location).getAbsoluteFile();
    if (!file.exists()) {
        throw new ToolException("IDL file " + file.getName() + " doesn't exist");
    }
    try {
        URL orig = file.toURI().toURL();
        DefaultIncludeResolver includeResolver = getDefaultIncludeResolver(file.getParentFile());
        DefineState defineState = new DefineState(new HashMap<String, String>());
        preprocessor = new IdlPreprocessorReader(orig, location, includeResolver, defineState);
        IDLLexer lexer = new IDLLexer(new java.io.LineNumberReader(preprocessor));
        lexer.setTokenObjectClass("antlr.CommonHiddenStreamToken");
        TokenStreamHiddenTokenFilter filter = new TokenStreamHiddenTokenFilter(lexer);
        filter.discard(IDLTokenTypes.WS);
        filter.hide(IDLTokenTypes.SL_COMMENT);
        filter.hide(IDLTokenTypes.ML_COMMENT);
        parser = new IDLParser(filter);
        parser.setASTNodeClass("antlr.CommonASTWithHiddenTokens");
        parser.specification();
    } catch (Exception ex) {
        throw new ToolException(ex);
    }
}