Examples with TokenIntPair org.apache.tika.eval.tokens.TokenIntPair used on opensource projects

Example 1 with TokenIntPair

use of org.apache.tika.eval.tokens.TokenIntPair in project tika by apache.

the class ExtractComparer method writeContrastString.

private void writeContrastString(Map<Cols, String> data, Cols col, TokenIntPair[] tokenIntPairs) {
    int i = 0;
    StringBuilder sb = new StringBuilder();
    for (TokenIntPair p : tokenIntPairs) {
        if (i++ > 0) {
            sb.append(" | ");
        }
        sb.append(p.getToken()).append(": ").append(p.getValue());
    }
    data.put(col, sb.toString());
}

Example 2 with TokenIntPair

use of org.apache.tika.eval.tokens.TokenIntPair in project tika by apache.

the class AbstractProfiler method writeTokenCounts.

void writeTokenCounts(Map<Cols, String> data, String field, TokenCounter tokenCounter) {
    int stops = 0;
    int i = 0;
    StringBuilder sb = new StringBuilder();
    TokenStatistics tokenStatistics = tokenCounter.getTokenStatistics(field);
    for (TokenIntPair t : tokenStatistics.getTopN()) {
        if (i++ > 0) {
            sb.append(" | ");
        }
        sb.append(t.getToken() + ": " + t.getValue());
    }
    data.put(Cols.TOP_N_TOKENS, sb.toString());
}