Examples with CodingAgreementResult de.tudarmstadt.ukp.clarin.webanno.agreement.results.coding.CodingAgreementResult used on opensource projects

Example 6 with CodingAgreementResult

use of de.tudarmstadt.ukp.clarin.webanno.agreement.results.coding.CodingAgreementResult in project webanno by webanno.

the class KrippendorffAlphaNominalAgreementMeasureTest method testTwoWithoutLabel_noExcludeIncomplete.

@Test
public void testTwoWithoutLabel_noExcludeIncomplete() throws Exception {
    traits.setExcludeIncomplete(false);
    PairwiseAnnotationResult<CodingAgreementResult> agreement = twoWithoutLabelTest(sut, traits);
    CodingAgreementResult result = agreement.getStudy("user1", "user2");
    ICodingAnnotationItem item1 = result.getStudy().getItem(0);
    ICodingAnnotationItem item2 = result.getStudy().getItem(1);
    ICodingAnnotationItem item3 = result.getStudy().getItem(2);
    assertEquals("", item1.getUnit(0).getCategory());
    assertEquals("", item1.getUnit(1).getCategory());
    assertEquals("", item2.getUnit(0).getCategory());
    assertEquals(null, item2.getUnit(1).getCategory());
    assertEquals(null, item3.getUnit(0).getCategory());
    assertEquals("", item3.getUnit(1).getCategory());
    assertEquals(4, result.getTotalSetCount());
    assertEquals(0, result.getIrrelevantSets().size());
    // the following two counts are zero because the incomplete sets are not excluded!
    assertEquals(2, result.getIncompleteSetsByPosition().size());
    assertEquals(0, result.getIncompleteSetsByLabel().size());
    assertEquals(3, result.getSetsWithDifferences().size());
    assertEquals(4, result.getRelevantSetCount());
    assertEquals(0.4, result.getAgreement(), 0.01);
}

Example 7 with CodingAgreementResult

use of de.tudarmstadt.ukp.clarin.webanno.agreement.results.coding.CodingAgreementResult in project webanno by webanno.

the class CohenKappaAgreementMeasureTest method twoWithoutLabelTest.

@Test
public void twoWithoutLabelTest() throws Exception {
    PairwiseAnnotationResult<CodingAgreementResult> agreement = twoWithoutLabelTest(sut, traits);
    CodingAgreementResult result = agreement.getStudy("user1", "user2");
    result.getDiff().print(System.out);
    ICodingAnnotationItem item1 = result.getStudy().getItem(0);
    ICodingAnnotationItem item2 = result.getStudy().getItem(1);
    assertEquals("", item1.getUnit(0).getCategory());
    assertEquals("", item1.getUnit(1).getCategory());
    assertEquals("A", item2.getUnit(0).getCategory());
    assertEquals(4, result.getTotalSetCount());
    assertEquals(0, result.getIrrelevantSets().size());
    assertEquals(2, result.getIncompleteSetsByPosition().size());
    assertEquals(0, result.getIncompleteSetsByLabel().size());
    assertEquals(1, result.getSetsWithDifferences().size());
    assertEquals(4, result.getRelevantSetCount());
    assertEquals(0.333, result.getAgreement(), 0.01);
}

Example 8 with CodingAgreementResult

use of de.tudarmstadt.ukp.clarin.webanno.agreement.results.coding.CodingAgreementResult in project webanno by webanno.

the class CohenKappaAgreementMeasureTest method twoEmptyCasTest.

@Test
public void twoEmptyCasTest() throws Exception {
    PairwiseAnnotationResult<CodingAgreementResult> agreement = twoEmptyCasTest(sut);
    CodingAgreementResult result = agreement.getStudy("user1", "user2");
    DiffResult diff = result.getDiff();
    assertEquals(0, diff.size());
    assertEquals(0, diff.getDifferingConfigurationSets().size());
    assertEquals(0, diff.getIncompleteConfigurationSets().size());
    assertEquals(NaN, result.getAgreement(), 0.000001d);
    assertEquals(0, result.getIncompleteSetsByPosition().size());
}

Example 9 with CodingAgreementResult

use of de.tudarmstadt.ukp.clarin.webanno.agreement.results.coding.CodingAgreementResult in project webanno by webanno.

the class CohenKappaAgreementMeasureTest method singleNoDifferencesWithAdditionalCasTest.

@Test
public void singleNoDifferencesWithAdditionalCasTest() throws Exception {
    PairwiseAnnotationResult<CodingAgreementResult> agreement = singleNoDifferencesWithAdditionalCasTest(sut);
    CodingAgreementResult result1 = agreement.getStudy("user1", "user2");
    assertEquals(0, result1.getTotalSetCount());
    assertEquals(0, result1.getIrrelevantSets().size());
    assertEquals(0, result1.getRelevantSetCount());
    CodingAgreementResult result2 = agreement.getStudy("user1", "user3");
    assertEquals(1, result2.getTotalSetCount());
    assertEquals(0, result2.getIrrelevantSets().size());
    assertEquals(1, result2.getRelevantSetCount());
}

Example 10 with CodingAgreementResult

use of de.tudarmstadt.ukp.clarin.webanno.agreement.results.coding.CodingAgreementResult in project webanno by webanno.

the class KrippendorffAlphaNominalAgreementMeasureTest method twoEmptyCasTest.

@Test
public void twoEmptyCasTest() throws Exception {
    PairwiseAnnotationResult<CodingAgreementResult> agreement = twoEmptyCasTest(sut);
    CodingAgreementResult result = agreement.getStudy("user1", "user2");
    DiffResult diff = result.getDiff();
    assertEquals(0, diff.size());
    assertEquals(0, diff.getDifferingConfigurationSets().size());
    assertEquals(0, diff.getIncompleteConfigurationSets().size());
    assertEquals(NaN, result.getAgreement(), 0.000001d);
    assertEquals(0, result.getIncompleteSetsByPosition().size());
}