Examples with AnalyzedToken org.languagetool.AnalyzedToken used on opensource projects

Example 76 with AnalyzedToken

use of org.languagetool.AnalyzedToken in project languagetool by languagetool-org.

the class BaseTagger method tag.

@Override
public List<AnalyzedTokenReadings> tag(List<String> sentenceTokens) throws IOException {
    List<AnalyzedTokenReadings> tokenReadings = new ArrayList<>();
    int pos = 0;
    for (String word : sentenceTokens) {
        List<AnalyzedToken> l = getAnalyzedTokens(word);
        tokenReadings.add(new AnalyzedTokenReadings(l, pos));
        pos += word.length();
    }
    return tokenReadings;
}

Example 77 with AnalyzedToken

use of org.languagetool.AnalyzedToken in project languagetool by languagetool-org.

the class MultiWordChunker method prepareNewReading.

private AnalyzedTokenReadings prepareNewReading(String tokens, String tok, AnalyzedTokenReadings token, boolean isLast) {
    StringBuilder sb = new StringBuilder();
    sb.append('<');
    if (isLast) {
        sb.append('/');
    }
    sb.append(mFull.get(tokens));
    sb.append('>');
    AnalyzedToken tokenStart = new AnalyzedToken(tok, sb.toString(), tokens);
    return setAndAnnotate(token, tokenStart);
}

Example 78 with AnalyzedToken

use of org.languagetool.AnalyzedToken in project languagetool by languagetool-org.

the class DisambiguationRuleHandler method addNewWord.

private void addNewWord(String word, String lemma, String pos) {
    AnalyzedToken newWd = new AnalyzedToken(word, pos, lemma);
    if (newWdList == null) {
        newWdList = new ArrayList<>();
    }
    newWdList.add(newWd);
}

Example 79 with AnalyzedToken

use of org.languagetool.AnalyzedToken in project languagetool by languagetool-org.

the class DemoTagger method tag.

@Override
public List<AnalyzedTokenReadings> tag(List<String> sentenceTokens) {
    List<AnalyzedTokenReadings> tokenReadings = new ArrayList<>();
    for (String word : sentenceTokens) {
        List<AnalyzedToken> l = new ArrayList<>();
        // a real tagger would need to assign a POS tag
        // in the next line instead of null:
        l.add(new AnalyzedToken(word, null, null));
        tokenReadings.add(new AnalyzedTokenReadings(l, 0));
    }
    return tokenReadings;
}

Example 80 with AnalyzedToken

use of org.languagetool.AnalyzedToken in project languagetool by languagetool-org.

the class DemoPartialPosTagFilter method tag.

@Override
protected List<AnalyzedTokenReadings> tag(String token) {
    if ("accurate".equals(token)) {
        AnalyzedToken resultToken = new AnalyzedToken(token, "JJ", "fake");
        List<AnalyzedToken> resultTokens = Collections.singletonList(resultToken);
        List<AnalyzedTokenReadings> result = new ArrayList<>();
        result.add(new AnalyzedTokenReadings(resultTokens, 0));
        return result;
    }
    return null;
}