Search in sources :

Example 1 with Tokens

use of com.yahoo.vespa.configdefinition.SpecialtokensConfig.Tokenlist.Tokens in project vespa by vespa-engine.

the class SpecialTokenRegistry method build.

private void build(SpecialtokensConfig config) {
    List<SpecialTokens> list = new ArrayList<>();
    for (Iterator<Tokenlist> i = config.tokenlist().iterator(); i.hasNext(); ) {
        Tokenlist tokenList = i.next();
        SpecialTokens tokens = new SpecialTokens(tokenList.name());
        for (Iterator<Tokens> j = tokenList.tokens().iterator(); j.hasNext(); ) {
            Tokens token = j.next();
            tokens.addSpecialToken(token.token(), token.replace());
        }
        tokens.freeze();
        list.add(tokens);
    }
    addSpecialTokens(list);
}
Also used : Tokenlist(com.yahoo.vespa.configdefinition.SpecialtokensConfig.Tokenlist) Tokens(com.yahoo.vespa.configdefinition.SpecialtokensConfig.Tokenlist.Tokens)

Example 2 with Tokens

use of com.yahoo.vespa.configdefinition.SpecialtokensConfig.Tokenlist.Tokens in project vespa by vespa-engine.

the class RewriterFeaturesTestCase method testConvertStringToQTree.

@Test
public final void testConvertStringToQTree() {
    Execution placeholder = new Execution(Context.createContextStub());
    SpecialTokenRegistry tokenRegistry = new SpecialTokenRegistry(new SpecialtokensConfig(new SpecialtokensConfig.Builder().tokenlist(new Tokenlist.Builder().name("default").tokens(new Tokens.Builder().token(ASCII_ELLIPSIS)))));
    placeholder.context().setTokenRegistry(tokenRegistry);
    Query query = new Query();
    query.getModel().setExecution(placeholder);
    Item parsed = RewriterFeatures.convertStringToQTree(query, "a b c " + ASCII_ELLIPSIS);
    assertSame(AndItem.class, parsed.getClass());
    assertEquals(4, ((CompositeItem) parsed).getItemCount());
    assertEquals(ASCII_ELLIPSIS, ((CompositeItem) parsed).getItem(3).toString());
}
Also used : CompositeItem(com.yahoo.prelude.query.CompositeItem) Item(com.yahoo.prelude.query.Item) AndItem(com.yahoo.prelude.query.AndItem) CompositeItem(com.yahoo.prelude.query.CompositeItem) Execution(com.yahoo.search.searchchain.Execution) SpecialTokenRegistry(com.yahoo.prelude.query.parser.SpecialTokenRegistry) Query(com.yahoo.search.Query) SpecialtokensConfig(com.yahoo.vespa.configdefinition.SpecialtokensConfig) Tokenlist(com.yahoo.vespa.configdefinition.SpecialtokensConfig.Tokenlist) Tokens(com.yahoo.vespa.configdefinition.SpecialtokensConfig.Tokenlist.Tokens) Test(org.junit.Test)

Aggregations

Tokenlist (com.yahoo.vespa.configdefinition.SpecialtokensConfig.Tokenlist)2 Tokens (com.yahoo.vespa.configdefinition.SpecialtokensConfig.Tokenlist.Tokens)2 AndItem (com.yahoo.prelude.query.AndItem)1 CompositeItem (com.yahoo.prelude.query.CompositeItem)1 Item (com.yahoo.prelude.query.Item)1 SpecialTokenRegistry (com.yahoo.prelude.query.parser.SpecialTokenRegistry)1 Query (com.yahoo.search.Query)1 Execution (com.yahoo.search.searchchain.Execution)1 SpecialtokensConfig (com.yahoo.vespa.configdefinition.SpecialtokensConfig)1 Test (org.junit.Test)1