Search in sources :

Example 36 with JetSqlRow

use of com.hazelcast.sql.impl.row.JetSqlRow in project hazelcast by hazelcast.

the class CreateDagVisitor method onProject.

public Vertex onProject(ProjectPhysicalRel rel) {
    List<Expression<?>> projection = rel.projection(parameterMetadata);
    Vertex vertex = dag.newUniqueVertex("Project", mapUsingServiceP(ServiceFactories.nonSharedService(ctx -> ExpressionUtil.projectionFn(projection, ExpressionEvalContext.from(ctx))), (Function<JetSqlRow, JetSqlRow> projectionFn, JetSqlRow row) -> projectionFn.apply(row)));
    connectInputPreserveCollation(rel, vertex);
    return vertex;
}
Also used : Vertex(com.hazelcast.jet.core.Vertex) Expression(com.hazelcast.sql.impl.expression.Expression) ConstantExpression(com.hazelcast.sql.impl.expression.ConstantExpression) JetSqlRow(com.hazelcast.sql.impl.row.JetSqlRow)

Example 37 with JetSqlRow

use of com.hazelcast.sql.impl.row.JetSqlRow in project hazelcast by hazelcast.

the class CreateDagVisitor method onAccumulate.

public Vertex onAccumulate(AggregateAccumulatePhysicalRel rel) {
    AggregateOperation<?, JetSqlRow> aggregateOperation = rel.aggrOp();
    Vertex vertex = dag.newUniqueVertex("Accumulate", Processors.accumulateP(aggregateOperation));
    connectInput(rel.getInput(), vertex, null);
    return vertex;
}
Also used : Vertex(com.hazelcast.jet.core.Vertex) JetSqlRow(com.hazelcast.sql.impl.row.JetSqlRow)

Example 38 with JetSqlRow

use of com.hazelcast.sql.impl.row.JetSqlRow in project hazelcast by hazelcast.

the class CreateDagVisitor method onCombineByKey.

public Vertex onCombineByKey(AggregateCombineByKeyPhysicalRel rel) {
    AggregateOperation<?, JetSqlRow> aggregateOperation = rel.aggrOp();
    Vertex vertex = dag.newUniqueVertex("CombineByKey", Processors.combineByKeyP(aggregateOperation, (key, value) -> value));
    connectInput(rel.getInput(), vertex, edge -> edge.distributed().partitioned(entryKey()));
    return vertex;
}
Also used : Address(com.hazelcast.cluster.Address) Traverser(com.hazelcast.jet.Traverser) ExpressionValues(com.hazelcast.jet.sql.impl.opt.ExpressionValues) PlanObjectKey(com.hazelcast.sql.impl.optimizer.PlanObjectKey) Processors.mapP(com.hazelcast.jet.core.processor.Processors.mapP) SourceProcessors.convenientSourceP(com.hazelcast.jet.core.processor.SourceProcessors.convenientSourceP) QueryParameterMetadata(com.hazelcast.sql.impl.QueryParameterMetadata) Collections.singletonList(java.util.Collections.singletonList) BiFunctionEx(com.hazelcast.function.BiFunctionEx) HazelcastTable(com.hazelcast.jet.sql.impl.schema.HazelcastTable) ObjectArrayKey(com.hazelcast.jet.sql.impl.ObjectArrayKey) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) SqlConnectorUtil(com.hazelcast.jet.sql.impl.connector.SqlConnectorUtil) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) Functions.entryKey(com.hazelcast.function.Functions.entryKey) SqlConnectorUtil.getJetSqlConnector(com.hazelcast.jet.sql.impl.connector.SqlConnectorUtil.getJetSqlConnector) DAG(com.hazelcast.jet.core.DAG) RootResultConsumerSink.rootResultConsumerSink(com.hazelcast.jet.sql.impl.processors.RootResultConsumerSink.rootResultConsumerSink) ServiceFactories(com.hazelcast.jet.pipeline.ServiceFactories) FunctionEx(com.hazelcast.function.FunctionEx) KeyedWindowResultFunction(com.hazelcast.jet.core.function.KeyedWindowResultFunction) Predicate(java.util.function.Predicate) Collections.emptyList(java.util.Collections.emptyList) Set(java.util.Set) ConsumerEx(com.hazelcast.function.ConsumerEx) IMapSqlConnector(com.hazelcast.jet.sql.impl.connector.map.IMapSqlConnector) ExpressionUtil(com.hazelcast.jet.sql.impl.ExpressionUtil) Processors.filterUsingServiceP(com.hazelcast.jet.core.processor.Processors.filterUsingServiceP) List(java.util.List) ExpressionEvalContext(com.hazelcast.sql.impl.expression.ExpressionEvalContext) Processors.mapUsingServiceP(com.hazelcast.jet.core.processor.Processors.mapUsingServiceP) Table(com.hazelcast.sql.impl.schema.Table) ComparatorEx(com.hazelcast.function.ComparatorEx) QueryDataType(com.hazelcast.sql.impl.type.QueryDataType) Processors(com.hazelcast.jet.core.processor.Processors) Processors.flatMapUsingServiceP(com.hazelcast.jet.core.processor.Processors.flatMapUsingServiceP) TimestampKind(com.hazelcast.jet.core.TimestampKind) Function(java.util.function.Function) JetSqlRow(com.hazelcast.sql.impl.row.JetSqlRow) HashSet(java.util.HashSet) Edge.from(com.hazelcast.jet.core.Edge.from) Edge(com.hazelcast.jet.core.Edge) Expression(com.hazelcast.sql.impl.expression.Expression) ProcessorSupplier(com.hazelcast.jet.core.ProcessorSupplier) VertexWithInputConfig(com.hazelcast.jet.sql.impl.connector.SqlConnector.VertexWithInputConfig) SingleRel(org.apache.calcite.rel.SingleRel) Nullable(javax.annotation.Nullable) JetJoinInfo(com.hazelcast.jet.sql.impl.JetJoinInfo) NodeEngine(com.hazelcast.spi.impl.NodeEngine) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) RelNode(org.apache.calcite.rel.RelNode) Consumer(java.util.function.Consumer) Vertex(com.hazelcast.jet.core.Vertex) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) SqlHashJoinP(com.hazelcast.jet.sql.impl.processors.SqlHashJoinP) ConstantExpression(com.hazelcast.sql.impl.expression.ConstantExpression) Processors.sortP(com.hazelcast.jet.core.processor.Processors.sortP) DefaultSerializationServiceBuilder(com.hazelcast.internal.serialization.impl.DefaultSerializationServiceBuilder) WindowUtils(com.hazelcast.jet.sql.impl.aggregate.WindowUtils) Edge.between(com.hazelcast.jet.core.Edge.between) Vertex(com.hazelcast.jet.core.Vertex) JetSqlRow(com.hazelcast.sql.impl.row.JetSqlRow)

Example 39 with JetSqlRow

use of com.hazelcast.sql.impl.row.JetSqlRow in project hazelcast by hazelcast.

the class CreateDagVisitor method onAggregateByKey.

public Vertex onAggregateByKey(AggregateByKeyPhysicalRel rel) {
    FunctionEx<JetSqlRow, ?> groupKeyFn = rel.groupKeyFn();
    AggregateOperation<?, JetSqlRow> aggregateOperation = rel.aggrOp();
    Vertex vertex = dag.newUniqueVertex("AggregateByKey", Processors.aggregateByKeyP(singletonList(groupKeyFn), aggregateOperation, (key, value) -> value));
    connectInput(rel.getInput(), vertex, edge -> edge.distributed().partitioned(groupKeyFn));
    return vertex;
}
Also used : Address(com.hazelcast.cluster.Address) Traverser(com.hazelcast.jet.Traverser) ExpressionValues(com.hazelcast.jet.sql.impl.opt.ExpressionValues) PlanObjectKey(com.hazelcast.sql.impl.optimizer.PlanObjectKey) Processors.mapP(com.hazelcast.jet.core.processor.Processors.mapP) SourceProcessors.convenientSourceP(com.hazelcast.jet.core.processor.SourceProcessors.convenientSourceP) QueryParameterMetadata(com.hazelcast.sql.impl.QueryParameterMetadata) Collections.singletonList(java.util.Collections.singletonList) BiFunctionEx(com.hazelcast.function.BiFunctionEx) HazelcastTable(com.hazelcast.jet.sql.impl.schema.HazelcastTable) ObjectArrayKey(com.hazelcast.jet.sql.impl.ObjectArrayKey) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) SqlConnectorUtil(com.hazelcast.jet.sql.impl.connector.SqlConnectorUtil) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) Functions.entryKey(com.hazelcast.function.Functions.entryKey) SqlConnectorUtil.getJetSqlConnector(com.hazelcast.jet.sql.impl.connector.SqlConnectorUtil.getJetSqlConnector) DAG(com.hazelcast.jet.core.DAG) RootResultConsumerSink.rootResultConsumerSink(com.hazelcast.jet.sql.impl.processors.RootResultConsumerSink.rootResultConsumerSink) ServiceFactories(com.hazelcast.jet.pipeline.ServiceFactories) FunctionEx(com.hazelcast.function.FunctionEx) KeyedWindowResultFunction(com.hazelcast.jet.core.function.KeyedWindowResultFunction) Predicate(java.util.function.Predicate) Collections.emptyList(java.util.Collections.emptyList) Set(java.util.Set) ConsumerEx(com.hazelcast.function.ConsumerEx) IMapSqlConnector(com.hazelcast.jet.sql.impl.connector.map.IMapSqlConnector) ExpressionUtil(com.hazelcast.jet.sql.impl.ExpressionUtil) Processors.filterUsingServiceP(com.hazelcast.jet.core.processor.Processors.filterUsingServiceP) List(java.util.List) ExpressionEvalContext(com.hazelcast.sql.impl.expression.ExpressionEvalContext) Processors.mapUsingServiceP(com.hazelcast.jet.core.processor.Processors.mapUsingServiceP) Table(com.hazelcast.sql.impl.schema.Table) ComparatorEx(com.hazelcast.function.ComparatorEx) QueryDataType(com.hazelcast.sql.impl.type.QueryDataType) Processors(com.hazelcast.jet.core.processor.Processors) Processors.flatMapUsingServiceP(com.hazelcast.jet.core.processor.Processors.flatMapUsingServiceP) TimestampKind(com.hazelcast.jet.core.TimestampKind) Function(java.util.function.Function) JetSqlRow(com.hazelcast.sql.impl.row.JetSqlRow) HashSet(java.util.HashSet) Edge.from(com.hazelcast.jet.core.Edge.from) Edge(com.hazelcast.jet.core.Edge) Expression(com.hazelcast.sql.impl.expression.Expression) ProcessorSupplier(com.hazelcast.jet.core.ProcessorSupplier) VertexWithInputConfig(com.hazelcast.jet.sql.impl.connector.SqlConnector.VertexWithInputConfig) SingleRel(org.apache.calcite.rel.SingleRel) Nullable(javax.annotation.Nullable) JetJoinInfo(com.hazelcast.jet.sql.impl.JetJoinInfo) NodeEngine(com.hazelcast.spi.impl.NodeEngine) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) RelNode(org.apache.calcite.rel.RelNode) Consumer(java.util.function.Consumer) Vertex(com.hazelcast.jet.core.Vertex) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) SqlHashJoinP(com.hazelcast.jet.sql.impl.processors.SqlHashJoinP) ConstantExpression(com.hazelcast.sql.impl.expression.ConstantExpression) Processors.sortP(com.hazelcast.jet.core.processor.Processors.sortP) DefaultSerializationServiceBuilder(com.hazelcast.internal.serialization.impl.DefaultSerializationServiceBuilder) WindowUtils(com.hazelcast.jet.sql.impl.aggregate.WindowUtils) Edge.between(com.hazelcast.jet.core.Edge.between) Vertex(com.hazelcast.jet.core.Vertex) JetSqlRow(com.hazelcast.sql.impl.row.JetSqlRow)

Example 40 with JetSqlRow

use of com.hazelcast.sql.impl.row.JetSqlRow in project hazelcast by hazelcast.

the class CreateDagVisitor method onSlidingWindow.

public Vertex onSlidingWindow(SlidingWindowPhysicalRel rel) {
    int orderingFieldIndex = rel.orderingFieldIndex();
    FunctionEx<ExpressionEvalContext, SlidingWindowPolicy> windowPolicySupplier = rel.windowPolicyProvider();
    // this vertex is used only if there's no aggregation by a window bound
    Vertex vertex = dag.newUniqueVertex("Sliding-Window", flatMapUsingServiceP(ServiceFactories.nonSharedService(ctx -> {
        ExpressionEvalContext evalContext = ExpressionEvalContext.from(ctx);
        SlidingWindowPolicy windowPolicy = windowPolicySupplier.apply(evalContext);
        return row -> WindowUtils.addWindowBounds(row, orderingFieldIndex, windowPolicy);
    }), (BiFunctionEx<Function<JetSqlRow, Traverser<JetSqlRow>>, JetSqlRow, Traverser<JetSqlRow>>) Function::apply));
    connectInput(rel.getInput(), vertex, null);
    return vertex;
}
Also used : ExpressionEvalContext(com.hazelcast.sql.impl.expression.ExpressionEvalContext) Address(com.hazelcast.cluster.Address) Traverser(com.hazelcast.jet.Traverser) ExpressionValues(com.hazelcast.jet.sql.impl.opt.ExpressionValues) PlanObjectKey(com.hazelcast.sql.impl.optimizer.PlanObjectKey) Processors.mapP(com.hazelcast.jet.core.processor.Processors.mapP) SourceProcessors.convenientSourceP(com.hazelcast.jet.core.processor.SourceProcessors.convenientSourceP) QueryParameterMetadata(com.hazelcast.sql.impl.QueryParameterMetadata) Collections.singletonList(java.util.Collections.singletonList) BiFunctionEx(com.hazelcast.function.BiFunctionEx) HazelcastTable(com.hazelcast.jet.sql.impl.schema.HazelcastTable) ObjectArrayKey(com.hazelcast.jet.sql.impl.ObjectArrayKey) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) SqlConnectorUtil(com.hazelcast.jet.sql.impl.connector.SqlConnectorUtil) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) Functions.entryKey(com.hazelcast.function.Functions.entryKey) SqlConnectorUtil.getJetSqlConnector(com.hazelcast.jet.sql.impl.connector.SqlConnectorUtil.getJetSqlConnector) DAG(com.hazelcast.jet.core.DAG) RootResultConsumerSink.rootResultConsumerSink(com.hazelcast.jet.sql.impl.processors.RootResultConsumerSink.rootResultConsumerSink) ServiceFactories(com.hazelcast.jet.pipeline.ServiceFactories) FunctionEx(com.hazelcast.function.FunctionEx) KeyedWindowResultFunction(com.hazelcast.jet.core.function.KeyedWindowResultFunction) Predicate(java.util.function.Predicate) Collections.emptyList(java.util.Collections.emptyList) Set(java.util.Set) ConsumerEx(com.hazelcast.function.ConsumerEx) IMapSqlConnector(com.hazelcast.jet.sql.impl.connector.map.IMapSqlConnector) ExpressionUtil(com.hazelcast.jet.sql.impl.ExpressionUtil) Processors.filterUsingServiceP(com.hazelcast.jet.core.processor.Processors.filterUsingServiceP) List(java.util.List) ExpressionEvalContext(com.hazelcast.sql.impl.expression.ExpressionEvalContext) Processors.mapUsingServiceP(com.hazelcast.jet.core.processor.Processors.mapUsingServiceP) Table(com.hazelcast.sql.impl.schema.Table) ComparatorEx(com.hazelcast.function.ComparatorEx) QueryDataType(com.hazelcast.sql.impl.type.QueryDataType) Processors(com.hazelcast.jet.core.processor.Processors) Processors.flatMapUsingServiceP(com.hazelcast.jet.core.processor.Processors.flatMapUsingServiceP) TimestampKind(com.hazelcast.jet.core.TimestampKind) Function(java.util.function.Function) JetSqlRow(com.hazelcast.sql.impl.row.JetSqlRow) HashSet(java.util.HashSet) Edge.from(com.hazelcast.jet.core.Edge.from) Edge(com.hazelcast.jet.core.Edge) Expression(com.hazelcast.sql.impl.expression.Expression) ProcessorSupplier(com.hazelcast.jet.core.ProcessorSupplier) VertexWithInputConfig(com.hazelcast.jet.sql.impl.connector.SqlConnector.VertexWithInputConfig) SingleRel(org.apache.calcite.rel.SingleRel) Nullable(javax.annotation.Nullable) JetJoinInfo(com.hazelcast.jet.sql.impl.JetJoinInfo) NodeEngine(com.hazelcast.spi.impl.NodeEngine) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) RelNode(org.apache.calcite.rel.RelNode) Consumer(java.util.function.Consumer) Vertex(com.hazelcast.jet.core.Vertex) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) SqlHashJoinP(com.hazelcast.jet.sql.impl.processors.SqlHashJoinP) ConstantExpression(com.hazelcast.sql.impl.expression.ConstantExpression) Processors.sortP(com.hazelcast.jet.core.processor.Processors.sortP) DefaultSerializationServiceBuilder(com.hazelcast.internal.serialization.impl.DefaultSerializationServiceBuilder) WindowUtils(com.hazelcast.jet.sql.impl.aggregate.WindowUtils) Edge.between(com.hazelcast.jet.core.Edge.between) Vertex(com.hazelcast.jet.core.Vertex) KeyedWindowResultFunction(com.hazelcast.jet.core.function.KeyedWindowResultFunction) Function(java.util.function.Function) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) Traverser(com.hazelcast.jet.Traverser) JetSqlRow(com.hazelcast.sql.impl.row.JetSqlRow) BiFunctionEx(com.hazelcast.function.BiFunctionEx)

Aggregations

JetSqlRow (com.hazelcast.sql.impl.row.JetSqlRow)50 ParallelJVMTest (com.hazelcast.test.annotation.ParallelJVMTest)16 QuickTest (com.hazelcast.test.annotation.QuickTest)16 Test (org.junit.Test)16 ExpressionEvalContext (com.hazelcast.sql.impl.expression.ExpressionEvalContext)15 ArrayList (java.util.ArrayList)14 Vertex (com.hazelcast.jet.core.Vertex)13 List (java.util.List)9 IndexConfig (com.hazelcast.config.IndexConfig)8 JobConfig (com.hazelcast.jet.config.JobConfig)8 ProcessorMetaSupplier (com.hazelcast.jet.core.ProcessorMetaSupplier)8 NodeEngine (com.hazelcast.spi.impl.NodeEngine)8 Expression (com.hazelcast.sql.impl.expression.Expression)8 QueryDataType (com.hazelcast.sql.impl.type.QueryDataType)8 FunctionEx (com.hazelcast.function.FunctionEx)7 QueryParameterMetadata (com.hazelcast.sql.impl.QueryParameterMetadata)7 DAG (com.hazelcast.jet.core.DAG)6 ObjectArrayKey (com.hazelcast.jet.sql.impl.ObjectArrayKey)6 DefaultSerializationServiceBuilder (com.hazelcast.internal.serialization.impl.DefaultSerializationServiceBuilder)5 Traverser (com.hazelcast.jet.Traverser)5