Examples with CodingAgreementResult de.tudarmstadt.ukp.clarin.webanno.agreement.results.coding.CodingAgreementResult used on opensource projects

Example 1 with CodingAgreementResult

use of de.tudarmstadt.ukp.clarin.webanno.agreement.results.coding.CodingAgreementResult in project webanno by webanno.

the class FleissKappaAgreementMeasureTest method twoWithoutLabelTest.

@Test
public void twoWithoutLabelTest() throws Exception {
    PairwiseAnnotationResult<CodingAgreementResult> agreement = twoWithoutLabelTest(sut, traits);
    CodingAgreementResult result = agreement.getStudy("user1", "user2");
    ICodingAnnotationItem item1 = result.getStudy().getItem(0);
    ICodingAnnotationItem item2 = result.getStudy().getItem(1);
    assertEquals("", item1.getUnit(0).getCategory());
    assertEquals("", item1.getUnit(1).getCategory());
    assertEquals("A", item2.getUnit(0).getCategory());
    assertEquals(4, result.getTotalSetCount());
    assertEquals(0, result.getIrrelevantSets().size());
    assertEquals(2, result.getIncompleteSetsByPosition().size());
    assertEquals(0, result.getIncompleteSetsByLabel().size());
    assertEquals(1, result.getSetsWithDifferences().size());
    assertEquals(4, result.getRelevantSetCount());
    assertEquals(0.2, result.getAgreement(), 0.01);
}

Example 2 with CodingAgreementResult

use of de.tudarmstadt.ukp.clarin.webanno.agreement.results.coding.CodingAgreementResult in project webanno by webanno.

the class TwoPairedKappaTest method testTwoUserDiffArcAndSpanAnnotation.

@Test
public void testTwoUserDiffArcAndSpanAnnotation() throws Exception {
    Map<User, List<SourceDocument>> userDocs = new HashMap<>();
    userDocs.put(user1, asList(document));
    userDocs.put(user2, asList(document));
    Map<User, CAS> userCases = new HashMap<>();
    userCases.put(user1, kappatestCas);
    userCases.put(user2, kappaspanarcdiff);
    Map<SourceDocument, Map<User, CAS>> documentJCases = new HashMap<>();
    documentJCases.put(document, userCases);
    // Check against new impl
    CasDiff diff = doDiff(asList(DEPENDENCY_DIFF_ADAPTER), LINK_TARGET_AS_LABEL, convert(userCases));
    DiffResult result = diff.toResult();
    CodingAgreementResult agreement = getCohenKappaAgreement(diff, Dependency.class.getName(), "DependencyType", convert(userCases));
    // Asserts
    System.out.printf("Agreement: %s%n", agreement.toString());
    result.print(System.out);
    AgreementUtils.dumpAgreementStudy(System.out, agreement);
    assertEquals(0.86153d, agreement.getAgreement(), 0.00001d);
    assertEquals(9, result.size());
    assertEquals(1, result.getDifferingConfigurationSets().size());
    assertEquals(0, result.getIncompleteConfigurationSets().size());
}

Example 3 with CodingAgreementResult

use of de.tudarmstadt.ukp.clarin.webanno.agreement.results.coding.CodingAgreementResult in project webanno by webanno.

the class CohenKappaAgreementMeasureTest method multiLinkWithRoleLabelDifference.

@Test
public void multiLinkWithRoleLabelDifference() throws Exception {
    PairwiseAnnotationResult<CodingAgreementResult> agreement = multiLinkWithRoleLabelDifferenceTest(sut);
    CodingAgreementResult result = agreement.getStudy("user1", "user2");
    DiffResult diff = result.getDiff();
    diff.print(System.out);
    assertEquals(3, diff.size());
    assertEquals(0, diff.getDifferingConfigurationSets().size());
    assertEquals(2, diff.getIncompleteConfigurationSets().size());
    assertEquals(NaN, result.getAgreement(), 0.00001d);
}

Example 4 with CodingAgreementResult

use of de.tudarmstadt.ukp.clarin.webanno.agreement.results.coding.CodingAgreementResult in project webanno by webanno.

the class KrippendorffAlphaNominalAgreementMeasureTest method multiLinkWithRoleLabelDifference.

@Test
public void multiLinkWithRoleLabelDifference() throws Exception {
    PairwiseAnnotationResult<CodingAgreementResult> agreement = multiLinkWithRoleLabelDifferenceTest(sut);
    CodingAgreementResult result = agreement.getStudy("user1", "user2");
    DiffResult diff = result.getDiff();
    diff.print(System.out);
    assertEquals(3, diff.size());
    assertEquals(0, diff.getDifferingConfigurationSets().size());
    assertEquals(2, diff.getIncompleteConfigurationSets().size());
    assertEquals(NaN, result.getAgreement(), 0.00001d);
}

Example 5 with CodingAgreementResult

use of de.tudarmstadt.ukp.clarin.webanno.agreement.results.coding.CodingAgreementResult in project webanno by webanno.

the class KrippendorffAlphaNominalAgreementMeasureTest method singleNoDifferencesWithAdditionalCasTest.

@Test
public void singleNoDifferencesWithAdditionalCasTest() throws Exception {
    PairwiseAnnotationResult<CodingAgreementResult> agreement = singleNoDifferencesWithAdditionalCasTest(sut);
    CodingAgreementResult result1 = agreement.getStudy("user1", "user2");
    assertEquals(0, result1.getTotalSetCount());
    assertEquals(0, result1.getIrrelevantSets().size());
    assertEquals(0, result1.getRelevantSetCount());
    CodingAgreementResult result2 = agreement.getStudy("user1", "user3");
    assertEquals(1, result2.getTotalSetCount());
    assertEquals(0, result2.getIrrelevantSets().size());
    assertEquals(1, result2.getRelevantSetCount());
}