Examples with BSLTokenizer com.github._1c_syntax.bsl.parser.BSLTokenizer used on opensource projects

Example 1 with BSLTokenizer

use of com.github._1c_syntax.bsl.parser.BSLTokenizer in project bsl-language-server by 1c-syntax.

the class DocumentContext method rebuild.

public void rebuild(String content, int version) {
    computeLock.lock();
    boolean versionMatches = version == this.version && version != 0;
    boolean contentWasCleared = this.content == null;
    if (versionMatches && !contentWasCleared) {
        clearDependantData();
        computeLock.unlock();
        return;
    }
    clearSecondaryData();
    symbolTree.clear();
    this.content = content;
    tokenizer = new BSLTokenizer(content);
    this.version = version;
    computeLock.unlock();
}

Example 2 with BSLTokenizer

use of com.github._1c_syntax.bsl.parser.BSLTokenizer in project bsl-language-server by 1c-syntax.

the class CommentedCodeDiagnostic method isTextParsedAsCode.

private boolean isTextParsedAsCode(String text) {
    if (!codeRecognizer.meetsCondition(text)) {
        return false;
    }
    BSLTokenizer tokenizer = new BSLTokenizer(uncomment(text));
    final List<Token> tokens = tokenizer.getTokens();
    // Если меньше двух токенов нет смысла анализировать - это код
    if (tokens.size() >= MINIMAL_TOKEN_COUNT) {
        List<Integer> tokenTypes = tokens.stream().map(Token::getType).filter(t -> t != BSLParser.WHITE_SPACE).collect(Collectors.toList());
        // Если два идентификатора идут подряд - это не код
        for (int i = 0; i < tokenTypes.size() - 1; i++) {
            if (tokenTypes.get(i) == BSLParser.IDENTIFIER && tokenTypes.get(i + 1) == BSLParser.IDENTIFIER) {
                return false;
            }
        }
    }
    return true;
}