Examples with SimpleLinguistics com.yahoo.language.simple.SimpleLinguistics used on opensource projects

Example 26 with SimpleLinguistics

use of com.yahoo.language.simple.SimpleLinguistics in project vespa by vespa-engine.

the class TokenizeTestCase method requireThatExpressionCanBeVerified.

@Test
public void requireThatExpressionCanBeVerified() {
    Expression exp = new TokenizeExpression(new SimpleLinguistics(), new AnnotatorConfig());
    assertVerify(DataType.STRING, exp, DataType.STRING);
    assertVerifyThrows(null, exp, "Expected string input, got null.");
    assertVerifyThrows(DataType.INT, exp, "Expected string input, got int.");
}

Example 27 with SimpleLinguistics

use of com.yahoo.language.simple.SimpleLinguistics in project vespa by vespa-engine.

the class TokenizeTestCase method requireThatValueIsAnnotated.

@Test
public void requireThatValueIsAnnotated() {
    ExecutionContext ctx = new ExecutionContext(new SimpleTestAdapter());
    ctx.setValue(new StringFieldValue("foo"));
    new TokenizeExpression(new SimpleLinguistics(), new AnnotatorConfig()).execute(ctx);
    FieldValue val = ctx.getValue();
    assertTrue(val instanceof StringFieldValue);
    assertNotNull(((StringFieldValue) val).getSpanTree(SpanTrees.LINGUISTICS));
}

Example 28 with SimpleLinguistics

use of com.yahoo.language.simple.SimpleLinguistics in project vespa by vespa-engine.

the class TokenizeTestCase method requireThatAccessorsWork.

@Test
public void requireThatAccessorsWork() {
    Linguistics linguistics = new SimpleLinguistics();
    AnnotatorConfig config = new AnnotatorConfig();
    TokenizeExpression exp = new TokenizeExpression(linguistics, config);
    assertSame(linguistics, exp.getLinguistics());
    assertSame(config, exp.getConfig());
}

Example 29 with SimpleLinguistics

use of com.yahoo.language.simple.SimpleLinguistics in project vespa by vespa-engine.

the class LinguisticsAnnotatorTestCase method requireThatTokenizeCappingWorks.

@Test
public void requireThatTokenizeCappingWorks() {
    String shortString = "short string";
    SpanTree spanTree = new SpanTree(SpanTrees.LINGUISTICS);
    spanTree.setStringFieldValue(new StringFieldValue(shortString));
    spanTree.spanList().span(0, 5).annotate(new Annotation(AnnotationTypes.TERM));
    spanTree.spanList().span(6, 6).annotate(new Annotation(AnnotationTypes.TERM));
    StringFieldValue shortValue = new StringFieldValue(shortString);
    Linguistics linguistics = new SimpleLinguistics();
    LinguisticsAnnotator annotator = new LinguisticsAnnotator(linguistics, new AnnotatorConfig().setMaxTokenLength(12));
    assertTrue(annotator.annotate(shortValue));
    assertEquals(spanTree, shortValue.getSpanTree(SpanTrees.LINGUISTICS));
    assertEquals(shortString, shortValue.getSpanTree(SpanTrees.LINGUISTICS).getStringFieldValue().getString());
    StringFieldValue cappedValue = new StringFieldValue(shortString + " a longer string");
    assertTrue(annotator.annotate(cappedValue));
    assertEquals((shortString + " a longer string"), cappedValue.getSpanTree(SpanTrees.LINGUISTICS).getStringFieldValue().getString());
}

Example 30 with SimpleLinguistics

use of com.yahoo.language.simple.SimpleLinguistics in project vespa by vespa-engine.

the class ExpressionTestCase method assertExpression.

private static void assertExpression(Class expectedClass, String str) throws ParseException {
    Linguistics linguistics = new SimpleLinguistics();
    Expression foo = Expression.fromString(str, linguistics);
    assertEquals(expectedClass, foo.getClass());
    Expression bar = Expression.fromString(foo.toString(), linguistics);
    assertEquals(foo.hashCode(), bar.hashCode());
    assertEquals(foo, bar);
}