Examples with PendingToken org.opengrok.indexer.analysis.PendingToken used on opensource projects

Example 1 with PendingToken

use of org.opengrok.indexer.analysis.PendingToken in project OpenGrok by OpenGrok.

the class DefinitionsTokenStream method incrementToken.

/**
 * Publishes the next, pending token from
 * {@link #initialize(org.opengrok.indexer.analysis.Definitions, org.opengrok.indexer.analysis.StreamSource,
 * org.opengrok.indexer.util.ReaderWrapper)},
 * if one is available.
 * @return false if no more tokens; otherwise true
 * @throws IOException in case of I/O error
 */
@Override
public final boolean incrementToken() throws IOException {
    if (offset < events.size()) {
        PendingToken tok = events.get(offset++);
        setAttribs(tok);
        return true;
    }
    clearAttributes();
    return false;
}

Example 2 with PendingToken

use of org.opengrok.indexer.analysis.PendingToken in project OpenGrok by OpenGrok.

the class DefinitionsTokenStream method createTokens.

private void createTokens(Definitions defs, LineBreaker brk) {
    for (Definitions.Tag tag : defs.getTags()) {
        // Shift from ctags's convention.
        int lineno = tag.line - 1;
        if (lineno >= 0 && lineno < brk.count() && tag.symbol != null && tag.text != null) {
            int lineoff = brk.getOffset(lineno);
            if (tag.lineStart >= 0) {
                PendingToken tok = new PendingToken(tag.symbol, lineoff + tag.lineStart, lineoff + tag.lineEnd);
                events.add(tok);
            }
        }
    }
    events.sort(PendingTokenOffsetsComparator.INSTANCE);
}