Examples with SimConfigurator edu.illinois.cs.cogcomp.config.SimConfigurator used on opensource projects

Example 1 with SimConfigurator

use of edu.illinois.cs.cogcomp.config.SimConfigurator in project cogcomp-nlp by CogComp.

the class LlmComparatorTest method testRemoveStopwords.

@Test
public void testRemoveStopwords() {
    WordListFilter filter = null;
    try {
        filter = new WordListFilter(new SimConfigurator().getDefaultConfig());
    } catch (IOException e) {
        e.printStackTrace();
        fail(e.getMessage());
    }
    String sent = "This sentence is filled with unnecessary filler like their pronouns , punctuation and function " + "words such as for , by , from , him , her , and to .";
    String[] tokens = sent.split("\\s+");
    String[] filteredTokens = filter.filter(tokens);
    int numSkipped = 0;
    List<String> filteredToks = new LinkedList<>();
    for (int i = 0; i < tokens.length; ++i) {
        String tok = filteredTokens[i];
        if (null == tok) {
            numSkipped++;
            filteredToks.add(tokens[i]);
        }
    }
    assert (numSkipped > 0);
    assert (filteredToks.contains("is"));
    System.out.println("Original text: " + sent);
    System.out.println("Filtered tokens: ");
    System.out.println(StringUtils.join(filteredToks, "; "));
}

Example 2 with SimConfigurator

use of edu.illinois.cs.cogcomp.config.SimConfigurator in project cogcomp-nlp by CogComp.

the class LLMStringSim method compare.

@Override
public MetricResponse compare(String arg1, String arg2) throws IllegalArgumentException {
    String reason = "";
    try {
        ResourceManager fullRm = new SimConfigurator().getConfig(rm_);
        double score;
        if (fullRm.getBoolean(SimConfigurator.USE_PHRASE_COMPARISON.key)) {
            System.out.println("using phrase representations");
            arg1 = preprocess.getPhrase(arg1, list);
            arg2 = preprocess.getPhrase(arg2, list);
        // System.out.println(arg1+"||"+arg2);
        }
        if (fullRm.getBoolean(SimConfigurator.USE_NE_COMPARISON.key)) {
            System.out.println("using NER annotator");
            TextAnnotation ta1 = preprocess.runNER(arg1);
            TextAnnotation ta2 = preprocess.runNER(arg2);
            score = llm.compareAnnotation(ta1, ta2);
        } else
            score = llm.compareStrings_(arg1, arg2);
        return new MetricResponse(score, reason);
    } catch (Exception e) {
    }
    return null;
}

Example 3 with SimConfigurator

use of edu.illinois.cs.cogcomp.config.SimConfigurator in project cogcomp-nlp by CogComp.

the class LlmStringComparator method initialize.

private void initialize(ResourceManager rm_, Comparator<String, EntailmentResult> comparator) throws IOException {
    ResourceManager fullRm = new SimConfigurator().getConfig(rm_);
    double threshold = fullRm.getDouble(SimConfigurator.LLM_ENTAILMENT_THRESHOLD.key);
    tokenizer = new IllinoisTokenizer();
    this.comparator = comparator;
    filter = new WordListFilter(fullRm);
    neAligner = new Aligner<String, EntailmentResult>(new NEComparator(), filter);
    aligner = new Aligner<String, EntailmentResult>(comparator, filter);
    scorer = new GreedyAlignmentScorer<String>(threshold);
}