Search in sources :

Example 41 with WordItem

use of com.yahoo.prelude.query.WordItem in project vespa by vespa-engine.

the class LowercasingTestCase method smoke.

@Test
public void smoke() {
    Query q = new Query();
    AndItem root = new AndItem();
    WordItem tmp;
    tmp = new WordItem("Gnuff", BAMSE, true);
    root.addItem(tmp);
    tmp = new WordItem("Blaff", TEDDY, true);
    root.addItem(tmp);
    tmp = new WordItem("Blyant", "", true);
    root.addItem(tmp);
    q.getModel().getQueryTree().setRoot(root);
    Result r = execution.search(q);
    root = (AndItem) r.getQuery().getModel().getQueryTree().getRoot();
    WordItem w0 = (WordItem) root.getItem(0);
    WordItem w1 = (WordItem) root.getItem(1);
    WordItem w2 = (WordItem) root.getItem(2);
    assertEquals("gnuff", w0.getWord());
    assertEquals("Blaff", w1.getWord());
    assertEquals("blyant", w2.getWord());
}
Also used : Query(com.yahoo.search.Query) AndItem(com.yahoo.prelude.query.AndItem) WordItem(com.yahoo.prelude.query.WordItem) Result(com.yahoo.search.Result) Test(org.junit.Test)

Example 42 with WordItem

use of com.yahoo.prelude.query.WordItem in project vespa by vespa-engine.

the class YqlParser method instantiateWordItem.

@NonNull
private Item instantiateWordItem(String field, String rawWord, OperatorNode<ExpressionOperator> ast, Class<?> parent, SegmentWhen segmentPolicy, boolean exactMatch, Language language) {
    String wordData = rawWord;
    if (getAnnotation(ast, NFKC, Boolean.class, Boolean.FALSE, "setting for whether to NFKC normalize input data")) {
        // NOTE: If this is set to FALSE (default), we will still NFKC normalize text data
        // during tokenization/segmentation, as that is always turned on also on the indexing side.
        wordData = normalizer.normalize(wordData);
    }
    boolean fromQuery = getAnnotation(ast, IMPLICIT_TRANSFORMS, Boolean.class, Boolean.TRUE, IMPLICIT_TRANSFORMS_DESCRIPTION);
    boolean prefixMatch = getAnnotation(ast, PREFIX, Boolean.class, Boolean.FALSE, "setting for whether to use prefix match of input data");
    boolean suffixMatch = getAnnotation(ast, SUFFIX, Boolean.class, Boolean.FALSE, "setting for whether to use suffix match of input data");
    boolean substrMatch = getAnnotation(ast, SUBSTRING, Boolean.class, Boolean.FALSE, "setting for whether to use substring match of input data");
    Preconditions.checkArgument((prefixMatch ? 1 : 0) + (substrMatch ? 1 : 0) + (suffixMatch ? 1 : 0) < 2, "Only one of prefix, substring and suffix can be set.");
    @NonNull final TaggableItem wordItem;
    if (exactMatch) {
        wordItem = new ExactStringItem(wordData, fromQuery);
    } else if (prefixMatch) {
        wordItem = new PrefixItem(wordData, fromQuery);
    } else if (suffixMatch) {
        wordItem = new SuffixItem(wordData, fromQuery);
    } else if (substrMatch) {
        wordItem = new SubstringItem(wordData, fromQuery);
    } else {
        switch(segmentPolicy) {
            case NEVER:
                wordItem = new WordItem(wordData, fromQuery);
                break;
            case POSSIBLY:
                if (shouldResegmentWord(field, fromQuery)) {
                    wordItem = resegment(field, ast, wordData, fromQuery, parent, language);
                } else {
                    wordItem = new WordItem(wordData, fromQuery);
                }
                break;
            case ALWAYS:
                wordItem = resegment(field, ast, wordData, fromQuery, parent, language);
                break;
            default:
                throw new IllegalArgumentException("Unexpected segmenting rule: " + segmentPolicy);
        }
    }
    if (wordItem instanceof WordItem) {
        prepareWord(field, ast, fromQuery, (WordItem) wordItem);
    }
    if (// mark the language used, unless it's the default
    language != Language.ENGLISH)
        ((Item) wordItem).setLanguage(language);
    return (Item) leafStyleSettings(ast, wordItem);
}
Also used : SuffixItem(com.yahoo.prelude.query.SuffixItem) CompositeItem(com.yahoo.prelude.query.CompositeItem) WordAlternativesItem(com.yahoo.prelude.query.WordAlternativesItem) NullItem(com.yahoo.prelude.query.NullItem) PrefixItem(com.yahoo.prelude.query.PrefixItem) OrItem(com.yahoo.prelude.query.OrItem) PhraseItem(com.yahoo.prelude.query.PhraseItem) TaggableItem(com.yahoo.prelude.query.TaggableItem) SubstringItem(com.yahoo.prelude.query.SubstringItem) AndItem(com.yahoo.prelude.query.AndItem) RankItem(com.yahoo.prelude.query.RankItem) EquivItem(com.yahoo.prelude.query.EquivItem) WeightedSetItem(com.yahoo.prelude.query.WeightedSetItem) PhraseSegmentItem(com.yahoo.prelude.query.PhraseSegmentItem) ExactStringItem(com.yahoo.prelude.query.ExactStringItem) PredicateQueryItem(com.yahoo.prelude.query.PredicateQueryItem) WeakAndItem(com.yahoo.prelude.query.WeakAndItem) ONearItem(com.yahoo.prelude.query.ONearItem) DotProductItem(com.yahoo.prelude.query.DotProductItem) Item(com.yahoo.prelude.query.Item) SuffixItem(com.yahoo.prelude.query.SuffixItem) AndSegmentItem(com.yahoo.prelude.query.AndSegmentItem) SegmentItem(com.yahoo.prelude.query.SegmentItem) IntItem(com.yahoo.prelude.query.IntItem) WandItem(com.yahoo.prelude.query.WandItem) RegExpItem(com.yahoo.prelude.query.RegExpItem) RangeItem(com.yahoo.prelude.query.RangeItem) WordItem(com.yahoo.prelude.query.WordItem) NotItem(com.yahoo.prelude.query.NotItem) NearItem(com.yahoo.prelude.query.NearItem) TaggableItem(com.yahoo.prelude.query.TaggableItem) NonNull(edu.umd.cs.findbugs.annotations.NonNull) SubstringItem(com.yahoo.prelude.query.SubstringItem) WordItem(com.yahoo.prelude.query.WordItem) PrefixItem(com.yahoo.prelude.query.PrefixItem) ExactStringItem(com.yahoo.prelude.query.ExactStringItem) NonNull(edu.umd.cs.findbugs.annotations.NonNull)

Example 43 with WordItem

use of com.yahoo.prelude.query.WordItem in project vespa by vespa-engine.

the class ItemEncodingTestCase method testNoRankedNoPositionDataWordItemEncoding.

@Test
public void testNoRankedNoPositionDataWordItemEncoding() {
    WordItem word = new WordItem("test");
    word.setRanked(false);
    word.setPositionData(false);
    ByteBuffer buffer = ByteBuffer.allocate(128);
    int count = word.encode(buffer);
    buffer.flip();
    assertEquals("Serialization count", 1, count);
    assertType(buffer, 4, 4);
    assertEquals(0x05, buffer.get());
    assertEquals("Index length", 0, buffer.get());
    assertEquals("Word length", 4, buffer.get());
    assertEquals("Word length", 4, buffer.remaining());
    assertEquals('t', buffer.get());
    assertEquals('e', buffer.get());
    assertEquals('s', buffer.get());
    assertEquals('t', buffer.get());
}
Also used : MarkerWordItem(com.yahoo.prelude.query.MarkerWordItem) WordItem(com.yahoo.prelude.query.WordItem) ByteBuffer(java.nio.ByteBuffer) Test(org.junit.Test)

Example 44 with WordItem

use of com.yahoo.prelude.query.WordItem in project vespa by vespa-engine.

the class ItemEncodingTestCase method testEndHostMarkerEncoding.

@Test
public void testEndHostMarkerEncoding() {
    WordItem word = MarkerWordItem.createEndOfHost();
    ByteBuffer buffer = ByteBuffer.allocate(128);
    int count = word.encode(buffer);
    buffer.flip();
    assertEquals("Serialization count", 1, count);
    assertType(buffer, 4, 0);
    assertEquals("Index length", 0, buffer.get());
    assertEquals("Word length", 7, buffer.get());
    assertEquals("Word length", 7, buffer.remaining());
    assertEquals('E', buffer.get());
    assertEquals('n', buffer.get());
    assertEquals('D', buffer.get());
    assertEquals('h', buffer.get());
    assertEquals('O', buffer.get());
    assertEquals('s', buffer.get());
    assertEquals('T', buffer.get());
}
Also used : MarkerWordItem(com.yahoo.prelude.query.MarkerWordItem) WordItem(com.yahoo.prelude.query.WordItem) ByteBuffer(java.nio.ByteBuffer) Test(org.junit.Test)

Example 45 with WordItem

use of com.yahoo.prelude.query.WordItem in project vespa by vespa-engine.

the class ItemEncodingTestCase method testStartHostMarkerEncoding.

@Test
public void testStartHostMarkerEncoding() {
    WordItem word = MarkerWordItem.createStartOfHost();
    ByteBuffer buffer = ByteBuffer.allocate(128);
    int count = word.encode(buffer);
    buffer.flip();
    assertEquals("Serialization count", 1, count);
    assertType(buffer, 4, 0);
    assertEquals("Index length", 0, buffer.get());
    assertEquals("Word length", 9, buffer.get());
    assertEquals("Word length", 9, buffer.remaining());
    assertEquals('S', buffer.get());
    assertEquals('t', buffer.get());
    assertEquals('A', buffer.get());
    assertEquals('r', buffer.get());
    assertEquals('T', buffer.get());
    assertEquals('h', buffer.get());
    assertEquals('O', buffer.get());
    assertEquals('s', buffer.get());
    assertEquals('T', buffer.get());
}
Also used : MarkerWordItem(com.yahoo.prelude.query.MarkerWordItem) WordItem(com.yahoo.prelude.query.WordItem) ByteBuffer(java.nio.ByteBuffer) Test(org.junit.Test)

Aggregations

WordItem (com.yahoo.prelude.query.WordItem)93 Test (org.junit.Test)76 AndItem (com.yahoo.prelude.query.AndItem)45 PhraseItem (com.yahoo.prelude.query.PhraseItem)31 Query (com.yahoo.search.Query)25 PhraseSegmentItem (com.yahoo.prelude.query.PhraseSegmentItem)20 CompositeItem (com.yahoo.prelude.query.CompositeItem)18 Item (com.yahoo.prelude.query.Item)17 MarkerWordItem (com.yahoo.prelude.query.MarkerWordItem)16 NotItem (com.yahoo.prelude.query.NotItem)16 OrItem (com.yahoo.prelude.query.OrItem)13 ByteBuffer (java.nio.ByteBuffer)11 PrefixItem (com.yahoo.prelude.query.PrefixItem)10 SubstringItem (com.yahoo.prelude.query.SubstringItem)10 SuffixItem (com.yahoo.prelude.query.SuffixItem)10 Execution (com.yahoo.search.searchchain.Execution)10 RankItem (com.yahoo.prelude.query.RankItem)9 PhraseMatcher (com.yahoo.prelude.querytransform.PhraseMatcher)9 IntItem (com.yahoo.prelude.query.IntItem)8 WeakAndItem (com.yahoo.prelude.query.WeakAndItem)7