Search in sources :

Example 1 with Token

use of net.sourceforge.pmd.lang.python.ast.Token in project pmd by pmd.

the class PythonTokenizer method tokenize.

@Override
public void tokenize(SourceCode sourceCode, Tokens tokenEntries) {
    StringBuilder buffer = sourceCode.getCodeBuffer();
    Reader reader = null;
    try {
        LanguageVersionHandler languageVersionHandler = LanguageRegistry.getLanguage(PythonLanguageModule.NAME).getDefaultVersion().getLanguageVersionHandler();
        reader = new StringReader(buffer.toString());
        reader = IOUtil.skipBOM(reader);
        TokenManager tokenManager = languageVersionHandler.getParser(languageVersionHandler.getDefaultParserOptions()).getTokenManager(sourceCode.getFileName(), reader);
        Token currentToken = (Token) tokenManager.getNextToken();
        while (currentToken.image.length() > 0) {
            tokenEntries.add(new TokenEntry(currentToken.image, sourceCode.getFileName(), currentToken.beginLine));
            currentToken = (Token) tokenManager.getNextToken();
        }
        tokenEntries.add(TokenEntry.getEOF());
        System.err.println("Added " + sourceCode);
    } catch (TokenMgrError err) {
        err.printStackTrace();
        System.err.println("Skipping " + sourceCode + " due to parse error");
        tokenEntries.add(TokenEntry.getEOF());
    } finally {
        IOUtils.closeQuietly(reader);
    }
}
Also used : StringReader(java.io.StringReader) StringReader(java.io.StringReader) Reader(java.io.Reader) Token(net.sourceforge.pmd.lang.python.ast.Token) TokenMgrError(net.sourceforge.pmd.lang.ast.TokenMgrError) LanguageVersionHandler(net.sourceforge.pmd.lang.LanguageVersionHandler) TokenManager(net.sourceforge.pmd.lang.TokenManager)

Aggregations

Reader (java.io.Reader)1 StringReader (java.io.StringReader)1 LanguageVersionHandler (net.sourceforge.pmd.lang.LanguageVersionHandler)1 TokenManager (net.sourceforge.pmd.lang.TokenManager)1 TokenMgrError (net.sourceforge.pmd.lang.ast.TokenMgrError)1 Token (net.sourceforge.pmd.lang.python.ast.Token)1