Examples with FeatureExtractors io.anserini.ltr.feature.FeatureExtractors used on opensource projects

Example 11 with FeatureExtractors

use of io.anserini.ltr.feature.FeatureExtractors in project Anserini by castorini.

the class FeatureExtractorChainFromJsonTest method testChainSingleExtractorNoParam.

@Test
public void testChainSingleExtractorNoParam() throws Exception {
    String jsonString = "{extractors: [ {name: \"AvgSCQ\"} ]}";
    JsonObject json = parser.parse(jsonString).getAsJsonObject();
    String testText = "test document";
    String testQuery = "document";
    //idf = 0.28768
    //tf =1
    float[] expected = { -0.24590f };
    FeatureExtractors chain = FeatureExtractors.fromJson(json);
    assertFeatureValues(expected, testQuery, testText, chain);
}

Example 12 with FeatureExtractors

use of io.anserini.ltr.feature.FeatureExtractors in project Anserini by castorini.

the class FeatureExtractorChainFromJsonTest method testChainSingleExtractorParam.

@Test
public void testChainSingleExtractorParam() throws Exception {
    String jsonString = "{extractors: [ {name: \"BM25Feature\", params: {k1:0.9, b:0.4}} ]}";
    JsonObject json = parser.parse(jsonString).getAsJsonObject();
    String docText = "single document test case";
    String queryText = "test";
    //df, tf =1, avgFL = 4, numDocs = 1
    //idf = log(1 + (0.5 / 1 + 0.5)) = 0.287682
    // 0.287682* 1.9 / (1 + 0.9 * (0.6 + 0.4 * (4/4))) = 1 * 0.287682
    // 0.287682 * 2.25 / (1 + 1.25 *(0.25 + 0.75)) = 0.287682
    float[] expected = { 0.287682f };
    FeatureExtractors chain = FeatureExtractors.fromJson(json);
    assertFeatureValues(expected, queryText, docText, chain);
}

Example 13 with FeatureExtractors

use of io.anserini.ltr.feature.FeatureExtractors in project Anserini by castorini.

the class FeatureExtractorChainFromJsonTest method testMultipleExtractorNoParam.

@Test
public void testMultipleExtractorNoParam() throws Exception {
    String jsonString = "{extractors: [ {name: \"AvgIDF\"}, {name: \"SumTermFrequency\"} ]}";
    JsonObject json = parser.parse(jsonString).getAsJsonObject();
    String docText = "document missing token";
    String queryText = "document test";
    float[] expected = { 0.836985f, 1f };
    FeatureExtractors chain = FeatureExtractors.fromJson(json);
    assertFeatureValues(expected, queryText, docText, chain);
}

Example 14 with FeatureExtractors

use of io.anserini.ltr.feature.FeatureExtractors in project Anserini by castorini.

the class TermFrequencyFeatureExtractorTest method getChain.

private FeatureExtractors getChain() {
    FeatureExtractors chain = new FeatureExtractors();
    chain.add(new TermFrequencyFeatureExtractor());
    return chain;
}

Example 15 with FeatureExtractors

use of io.anserini.ltr.feature.FeatureExtractors in project Anserini by castorini.

the class BigramFeaturesTest method getAllPairsUnOrdered.

private FeatureExtractors getAllPairsUnOrdered() {
    FeatureExtractors chain = new FeatureExtractors();
    chain.add(new UnorderedQueryPairsFeatureExtractor(2));
    chain.add(new UnorderedQueryPairsFeatureExtractor(4));
    chain.add(new UnorderedQueryPairsFeatureExtractor(6));
    return chain;
}