Search in sources :

Example 1 with StatementExpression

use of com.yahoo.vespa.indexinglanguage.expressions.StatementExpression in project vespa by vespa-engine.

the class ExpressionVisitorTestCase method requireThatAllExpressionsAreVisited.

@SuppressWarnings("unchecked")
@Test
public void requireThatAllExpressionsAreVisited() {
    assertCount(3, new ArithmeticExpression(new InputExpression("foo"), ArithmeticExpression.Operator.ADD, new InputExpression("bar")));
    assertCount(1, new AttributeExpression("foo"));
    assertCount(1, new Base64DecodeExpression());
    assertCount(1, new Base64EncodeExpression());
    assertCount(3, new CatExpression(new InputExpression("foo"), new IndexExpression("bar")));
    assertCount(1, new ClearStateExpression());
    assertCount(1, new EchoExpression());
    assertCount(2, new ForEachExpression(new IndexExpression("foo")));
    assertCount(1, new GetFieldExpression("foo"));
    assertCount(1, new GetVarExpression("foo"));
    assertCount(2, new GuardExpression(new IndexExpression("foo")));
    assertCount(1, new HexDecodeExpression());
    assertCount(1, new HexEncodeExpression());
    assertCount(1, new HostNameExpression());
    assertCount(5, new IfThenExpression(new InputExpression("foo"), IfThenExpression.Comparator.EQ, new InputExpression("bar"), new IndexExpression("baz"), new IndexExpression("cox")));
    assertCount(1, new IndexExpression("foo"));
    assertCount(1, new InputExpression("foo"));
    assertCount(1, new JoinExpression("foo"));
    assertCount(1, new LowerCaseExpression());
    assertCount(1, new NormalizeExpression(new SimpleLinguistics()));
    assertCount(1, new NowExpression());
    assertCount(1, new OptimizePredicateExpression());
    assertCount(2, new ParenthesisExpression(new InputExpression("foo")));
    assertCount(1, new RandomExpression(69));
    assertCount(3, new ScriptExpression(new StatementExpression(new InputExpression("foo"))));
    assertCount(3, new SelectInputExpression(new Pair<String, Expression>("foo", new IndexExpression("bar")), new Pair<String, Expression>("bar", new IndexExpression("foo"))));
    assertCount(1, new SetLanguageExpression());
    assertCount(1, new SetValueExpression(new IntegerFieldValue(69)));
    assertCount(1, new SetVarExpression("foo"));
    assertCount(1, new SplitExpression("foo"));
    assertCount(2, new StatementExpression(new InputExpression("foo")));
    assertCount(1, new SummaryExpression("foo"));
    assertCount(1, new SubstringExpression(6, 9));
    assertCount(3, new SwitchExpression(Collections.singletonMap("foo", (Expression) new IndexExpression("bar")), new InputExpression("baz")));
    assertCount(1, new ThisExpression());
    assertCount(1, new ToArrayExpression());
    assertCount(1, new ToByteExpression());
    assertCount(1, new ToDoubleExpression());
    assertCount(1, new ToFloatExpression());
    assertCount(1, new ToIntegerExpression());
    assertCount(1, new TokenizeExpression(new SimpleLinguistics(), new AnnotatorConfig()));
    assertCount(1, new ToLongExpression());
    assertCount(1, new ToPositionExpression());
    assertCount(1, new ToStringExpression());
    assertCount(1, new ToWsetExpression(false, false));
    assertCount(1, new TrimExpression());
    assertCount(1, new ZCurveExpression());
}
Also used : SummaryExpression(com.yahoo.vespa.indexinglanguage.expressions.SummaryExpression) SwitchExpression(com.yahoo.vespa.indexinglanguage.expressions.SwitchExpression) AnnotatorConfig(com.yahoo.vespa.indexinglanguage.linguistics.AnnotatorConfig) StatementExpression(com.yahoo.vespa.indexinglanguage.expressions.StatementExpression) IntegerFieldValue(com.yahoo.document.datatypes.IntegerFieldValue) OptimizePredicateExpression(com.yahoo.vespa.indexinglanguage.expressions.OptimizePredicateExpression) SplitExpression(com.yahoo.vespa.indexinglanguage.expressions.SplitExpression) ToPositionExpression(com.yahoo.vespa.indexinglanguage.expressions.ToPositionExpression) ThisExpression(com.yahoo.vespa.indexinglanguage.expressions.ThisExpression) ToByteExpression(com.yahoo.vespa.indexinglanguage.expressions.ToByteExpression) TokenizeExpression(com.yahoo.vespa.indexinglanguage.expressions.TokenizeExpression) IfThenExpression(com.yahoo.vespa.indexinglanguage.expressions.IfThenExpression) ToStringExpression(com.yahoo.vespa.indexinglanguage.expressions.ToStringExpression) SubstringExpression(com.yahoo.vespa.indexinglanguage.expressions.SubstringExpression) EchoExpression(com.yahoo.vespa.indexinglanguage.expressions.EchoExpression) SetVarExpression(com.yahoo.vespa.indexinglanguage.expressions.SetVarExpression) HexDecodeExpression(com.yahoo.vespa.indexinglanguage.expressions.HexDecodeExpression) SelectInputExpression(com.yahoo.vespa.indexinglanguage.expressions.SelectInputExpression) InputExpression(com.yahoo.vespa.indexinglanguage.expressions.InputExpression) AttributeExpression(com.yahoo.vespa.indexinglanguage.expressions.AttributeExpression) ScriptExpression(com.yahoo.vespa.indexinglanguage.expressions.ScriptExpression) SimpleLinguistics(com.yahoo.language.simple.SimpleLinguistics) ToArrayExpression(com.yahoo.vespa.indexinglanguage.expressions.ToArrayExpression) SetLanguageExpression(com.yahoo.vespa.indexinglanguage.expressions.SetLanguageExpression) LowerCaseExpression(com.yahoo.vespa.indexinglanguage.expressions.LowerCaseExpression) ToIntegerExpression(com.yahoo.vespa.indexinglanguage.expressions.ToIntegerExpression) ForEachExpression(com.yahoo.vespa.indexinglanguage.expressions.ForEachExpression) SelectInputExpression(com.yahoo.vespa.indexinglanguage.expressions.SelectInputExpression) ToWsetExpression(com.yahoo.vespa.indexinglanguage.expressions.ToWsetExpression) Base64DecodeExpression(com.yahoo.vespa.indexinglanguage.expressions.Base64DecodeExpression) IndexExpression(com.yahoo.vespa.indexinglanguage.expressions.IndexExpression) GetFieldExpression(com.yahoo.vespa.indexinglanguage.expressions.GetFieldExpression) RandomExpression(com.yahoo.vespa.indexinglanguage.expressions.RandomExpression) TrimExpression(com.yahoo.vespa.indexinglanguage.expressions.TrimExpression) JoinExpression(com.yahoo.vespa.indexinglanguage.expressions.JoinExpression) HexEncodeExpression(com.yahoo.vespa.indexinglanguage.expressions.HexEncodeExpression) Pair(com.yahoo.collections.Pair) SetValueExpression(com.yahoo.vespa.indexinglanguage.expressions.SetValueExpression) NowExpression(com.yahoo.vespa.indexinglanguage.expressions.NowExpression) ToDoubleExpression(com.yahoo.vespa.indexinglanguage.expressions.ToDoubleExpression) ToLongExpression(com.yahoo.vespa.indexinglanguage.expressions.ToLongExpression) ToFloatExpression(com.yahoo.vespa.indexinglanguage.expressions.ToFloatExpression) Base64EncodeExpression(com.yahoo.vespa.indexinglanguage.expressions.Base64EncodeExpression) CatExpression(com.yahoo.vespa.indexinglanguage.expressions.CatExpression) GuardExpression(com.yahoo.vespa.indexinglanguage.expressions.GuardExpression) ParenthesisExpression(com.yahoo.vespa.indexinglanguage.expressions.ParenthesisExpression) NormalizeExpression(com.yahoo.vespa.indexinglanguage.expressions.NormalizeExpression) ArithmeticExpression(com.yahoo.vespa.indexinglanguage.expressions.ArithmeticExpression) HostNameExpression(com.yahoo.vespa.indexinglanguage.expressions.HostNameExpression) GetVarExpression(com.yahoo.vespa.indexinglanguage.expressions.GetVarExpression) ClearStateExpression(com.yahoo.vespa.indexinglanguage.expressions.ClearStateExpression) ZCurveExpression(com.yahoo.vespa.indexinglanguage.expressions.ZCurveExpression) Test(org.junit.Test)

Example 2 with StatementExpression

use of com.yahoo.vespa.indexinglanguage.expressions.StatementExpression in project vespa by vespa-engine.

the class ExpressionConverterTestCase method requireThatScriptElementsCanBeRemoved.

@Test
public void requireThatScriptElementsCanBeRemoved() {
    StatementExpression foo = new StatementExpression(new AttributeExpression("foo"));
    StatementExpression bar = new StatementExpression(new AttributeExpression("bar"));
    ScriptExpression before = new ScriptExpression(foo, bar);
    Expression after = new SearchReplace(foo, null).convert(before);
    assertTrue(after instanceof ScriptExpression);
    assertEquals(1, ((ScriptExpression) after).size());
    assertEquals(bar, ((ScriptExpression) after).get(0));
    after = new SearchReplace(bar, null).convert(before);
    assertTrue(after instanceof ScriptExpression);
    assertEquals(1, ((ScriptExpression) after).size());
    assertEquals(foo, ((ScriptExpression) after).get(0));
}
Also used : StatementExpression(com.yahoo.vespa.indexinglanguage.expressions.StatementExpression) GetFieldExpression(com.yahoo.vespa.indexinglanguage.expressions.GetFieldExpression) SetVarExpression(com.yahoo.vespa.indexinglanguage.expressions.SetVarExpression) Base64DecodeExpression(com.yahoo.vespa.indexinglanguage.expressions.Base64DecodeExpression) SummaryExpression(com.yahoo.vespa.indexinglanguage.expressions.SummaryExpression) SelectInputExpression(com.yahoo.vespa.indexinglanguage.expressions.SelectInputExpression) IndexExpression(com.yahoo.vespa.indexinglanguage.expressions.IndexExpression) SubstringExpression(com.yahoo.vespa.indexinglanguage.expressions.SubstringExpression) ClearStateExpression(com.yahoo.vespa.indexinglanguage.expressions.ClearStateExpression) EchoExpression(com.yahoo.vespa.indexinglanguage.expressions.EchoExpression) ToByteExpression(com.yahoo.vespa.indexinglanguage.expressions.ToByteExpression) TrimExpression(com.yahoo.vespa.indexinglanguage.expressions.TrimExpression) GuardExpression(com.yahoo.vespa.indexinglanguage.expressions.GuardExpression) ScriptExpression(com.yahoo.vespa.indexinglanguage.expressions.ScriptExpression) JoinExpression(com.yahoo.vespa.indexinglanguage.expressions.JoinExpression) LowerCaseExpression(com.yahoo.vespa.indexinglanguage.expressions.LowerCaseExpression) GetVarExpression(com.yahoo.vespa.indexinglanguage.expressions.GetVarExpression) NowExpression(com.yahoo.vespa.indexinglanguage.expressions.NowExpression) HostNameExpression(com.yahoo.vespa.indexinglanguage.expressions.HostNameExpression) SwitchExpression(com.yahoo.vespa.indexinglanguage.expressions.SwitchExpression) CompositeExpression(com.yahoo.vespa.indexinglanguage.expressions.CompositeExpression) AttributeExpression(com.yahoo.vespa.indexinglanguage.expressions.AttributeExpression) CatExpression(com.yahoo.vespa.indexinglanguage.expressions.CatExpression) SetLanguageExpression(com.yahoo.vespa.indexinglanguage.expressions.SetLanguageExpression) Expression(com.yahoo.vespa.indexinglanguage.expressions.Expression) SetValueExpression(com.yahoo.vespa.indexinglanguage.expressions.SetValueExpression) ToLongExpression(com.yahoo.vespa.indexinglanguage.expressions.ToLongExpression) ZCurveExpression(com.yahoo.vespa.indexinglanguage.expressions.ZCurveExpression) ToDoubleExpression(com.yahoo.vespa.indexinglanguage.expressions.ToDoubleExpression) ArithmeticExpression(com.yahoo.vespa.indexinglanguage.expressions.ArithmeticExpression) InputExpression(com.yahoo.vespa.indexinglanguage.expressions.InputExpression) ToFloatExpression(com.yahoo.vespa.indexinglanguage.expressions.ToFloatExpression) OptimizePredicateExpression(com.yahoo.vespa.indexinglanguage.expressions.OptimizePredicateExpression) TokenizeExpression(com.yahoo.vespa.indexinglanguage.expressions.TokenizeExpression) ToArrayExpression(com.yahoo.vespa.indexinglanguage.expressions.ToArrayExpression) ToIntegerExpression(com.yahoo.vespa.indexinglanguage.expressions.ToIntegerExpression) ThisExpression(com.yahoo.vespa.indexinglanguage.expressions.ThisExpression) ForEachExpression(com.yahoo.vespa.indexinglanguage.expressions.ForEachExpression) NormalizeExpression(com.yahoo.vespa.indexinglanguage.expressions.NormalizeExpression) RandomExpression(com.yahoo.vespa.indexinglanguage.expressions.RandomExpression) Base64EncodeExpression(com.yahoo.vespa.indexinglanguage.expressions.Base64EncodeExpression) IfThenExpression(com.yahoo.vespa.indexinglanguage.expressions.IfThenExpression) HexDecodeExpression(com.yahoo.vespa.indexinglanguage.expressions.HexDecodeExpression) StatementExpression(com.yahoo.vespa.indexinglanguage.expressions.StatementExpression) ToWsetExpression(com.yahoo.vespa.indexinglanguage.expressions.ToWsetExpression) ParenthesisExpression(com.yahoo.vespa.indexinglanguage.expressions.ParenthesisExpression) HexEncodeExpression(com.yahoo.vespa.indexinglanguage.expressions.HexEncodeExpression) ToPositionExpression(com.yahoo.vespa.indexinglanguage.expressions.ToPositionExpression) ToStringExpression(com.yahoo.vespa.indexinglanguage.expressions.ToStringExpression) SplitExpression(com.yahoo.vespa.indexinglanguage.expressions.SplitExpression) AttributeExpression(com.yahoo.vespa.indexinglanguage.expressions.AttributeExpression) ScriptExpression(com.yahoo.vespa.indexinglanguage.expressions.ScriptExpression) Test(org.junit.Test)

Example 3 with StatementExpression

use of com.yahoo.vespa.indexinglanguage.expressions.StatementExpression in project vespa by vespa-engine.

the class ScriptParserTestCase method requireThatStatementParserBacksUpStream.

@Test
public void requireThatStatementParserBacksUpStream() throws ParseException {
    ScriptParserContext config = newContext("input foo input bar");
    assertEquals(new StatementExpression(new InputExpression("foo")), ScriptParser.parseStatement(config));
    assertEquals(new StatementExpression(new InputExpression("bar")), ScriptParser.parseStatement(config));
}
Also used : StatementExpression(com.yahoo.vespa.indexinglanguage.expressions.StatementExpression) InputExpression(com.yahoo.vespa.indexinglanguage.expressions.InputExpression) Test(org.junit.Test)

Example 4 with StatementExpression

use of com.yahoo.vespa.indexinglanguage.expressions.StatementExpression in project vespa by vespa-engine.

the class ScriptParserTestCase method requireThatExpressionParserCanBeInvoked.

@Test
public void requireThatExpressionParserCanBeInvoked() throws ParseException {
    try {
        ScriptParser.parseExpression(newContext("foo"));
    } catch (ParseException e) {
        assertException(e, "Encountered \" <IDENTIFIER> \"foo \"\" at line 1, column 1.");
    }
    assertEquals(new InputExpression("foo"), ScriptParser.parseExpression(newContext("input foo")));
    assertEquals(new StatementExpression(new InputExpression("foo"), new EchoExpression()), ScriptParser.parseExpression(newContext("input foo | echo")));
    assertEquals(new ScriptExpression(new StatementExpression(new InputExpression("foo")), new StatementExpression(new EchoExpression())), ScriptParser.parseExpression(newContext("{ input foo; echo }")));
}
Also used : StatementExpression(com.yahoo.vespa.indexinglanguage.expressions.StatementExpression) EchoExpression(com.yahoo.vespa.indexinglanguage.expressions.EchoExpression) ParseException(com.yahoo.vespa.indexinglanguage.parser.ParseException) InputExpression(com.yahoo.vespa.indexinglanguage.expressions.InputExpression) ScriptExpression(com.yahoo.vespa.indexinglanguage.expressions.ScriptExpression) Test(org.junit.Test)

Example 5 with StatementExpression

use of com.yahoo.vespa.indexinglanguage.expressions.StatementExpression in project vespa by vespa-engine.

the class ScriptManager method createScriptsMap.

private static Map<String, Map<String, DocumentScript>> createScriptsMap(DocumentTypeManager docTypeMgr, IlscriptsConfig config, Linguistics linguistics) {
    Map<String, Map<String, DocumentScript>> documentFieldScripts = new HashMap<>(config.ilscript().size());
    ScriptParserContext parserContext = new ScriptParserContext(linguistics);
    parserContext.getAnnotatorConfig().setMaxTermOccurrences(config.maxtermoccurrences());
    parserContext.getAnnotatorConfig().setMaxTokenLength(config.fieldmatchmaxlength());
    for (IlscriptsConfig.Ilscript ilscript : config.ilscript()) {
        InputExpression.FieldPathOptimizer fieldPathOptimizer = new InputExpression.FieldPathOptimizer(docTypeMgr.getDocumentType(ilscript.doctype()));
        List<StatementExpression> expressions = new ArrayList<>(ilscript.content().size());
        Map<String, DocumentScript> fieldScripts = new HashMap<>(ilscript.content().size());
        for (String content : ilscript.content()) {
            expressions.add(parse(ilscript.doctype(), parserContext, content));
            StatementExpression statement = parse(ilscript.doctype(), parserContext, content);
            InputExpression.InputFieldNameExtractor inputFieldNameExtractor = new InputExpression.InputFieldNameExtractor();
            statement.select(inputFieldNameExtractor, inputFieldNameExtractor);
            statement.select(fieldPathOptimizer, fieldPathOptimizer);
            if (inputFieldNameExtractor.getInputFieldNames().size() == 1) {
                String fieldName = inputFieldNameExtractor.getInputFieldNames().get(0);
                ScriptExpression script;
                if (fieldScripts.containsKey(fieldName)) {
                    DocumentScript prev = fieldScripts.get(fieldName);
                    List<StatementExpression> appendedList = new ArrayList<>(((ScriptExpression) prev.getExpression()).asList());
                    appendedList.add(statement);
                    script = new ScriptExpression(appendedList);
                    log.log(Level.FINE, "Appending script for field '" + fieldName + "' = " + statement);
                    log.log(Level.FINE, "Full script for field '" + fieldName + "' = " + appendedList);
                } else {
                    script = new ScriptExpression(statement);
                    log.log(Level.FINE, "Setting script for field '" + fieldName + "' = " + statement);
                }
                DocumentScript documentScript = new DocumentScript(ilscript.doctype(), inputFieldNameExtractor.getInputFieldNames(), script);
                fieldScripts.put(fieldName, documentScript);
            } else {
                log.log(Level.FINE, "Non single(" + inputFieldNameExtractor.getInputFieldNames().size() + ") inputs = " + inputFieldNameExtractor.getInputFieldNames() + ". Script = " + statement);
            }
        }
        ScriptExpression script = new ScriptExpression(expressions);
        script.select(fieldPathOptimizer, fieldPathOptimizer);
        fieldScripts.put(FULL, new DocumentScript(ilscript.doctype(), ilscript.docfield(), script));
        documentFieldScripts.put(ilscript.doctype(), Collections.unmodifiableMap(fieldScripts));
    }
    return Collections.unmodifiableMap(documentFieldScripts);
}
Also used : StatementExpression(com.yahoo.vespa.indexinglanguage.expressions.StatementExpression) InputExpression(com.yahoo.vespa.indexinglanguage.expressions.InputExpression) ScriptParserContext(com.yahoo.vespa.indexinglanguage.ScriptParserContext) ScriptExpression(com.yahoo.vespa.indexinglanguage.expressions.ScriptExpression) IlscriptsConfig(com.yahoo.vespa.configdefinition.IlscriptsConfig)

Aggregations

StatementExpression (com.yahoo.vespa.indexinglanguage.expressions.StatementExpression)12 InputExpression (com.yahoo.vespa.indexinglanguage.expressions.InputExpression)10 ScriptExpression (com.yahoo.vespa.indexinglanguage.expressions.ScriptExpression)10 Test (org.junit.Test)10 AttributeExpression (com.yahoo.vespa.indexinglanguage.expressions.AttributeExpression)6 EchoExpression (com.yahoo.vespa.indexinglanguage.expressions.EchoExpression)6 ArithmeticExpression (com.yahoo.vespa.indexinglanguage.expressions.ArithmeticExpression)5 CatExpression (com.yahoo.vespa.indexinglanguage.expressions.CatExpression)5 ForEachExpression (com.yahoo.vespa.indexinglanguage.expressions.ForEachExpression)5 GuardExpression (com.yahoo.vespa.indexinglanguage.expressions.GuardExpression)5 IfThenExpression (com.yahoo.vespa.indexinglanguage.expressions.IfThenExpression)5 IndexExpression (com.yahoo.vespa.indexinglanguage.expressions.IndexExpression)5 ParenthesisExpression (com.yahoo.vespa.indexinglanguage.expressions.ParenthesisExpression)5 SelectInputExpression (com.yahoo.vespa.indexinglanguage.expressions.SelectInputExpression)5 SwitchExpression (com.yahoo.vespa.indexinglanguage.expressions.SwitchExpression)5 Base64DecodeExpression (com.yahoo.vespa.indexinglanguage.expressions.Base64DecodeExpression)4 Base64EncodeExpression (com.yahoo.vespa.indexinglanguage.expressions.Base64EncodeExpression)4 ClearStateExpression (com.yahoo.vespa.indexinglanguage.expressions.ClearStateExpression)4 GetFieldExpression (com.yahoo.vespa.indexinglanguage.expressions.GetFieldExpression)4 GetVarExpression (com.yahoo.vespa.indexinglanguage.expressions.GetVarExpression)4