Search in sources :

Example 11 with Transform

use of com.hazelcast.jet.impl.pipeline.transform.Transform in project hazelcast by hazelcast.

the class StageWithKeyAndWindowImpl method aggregate2.

@Nonnull
@Override
@SuppressWarnings("rawtypes")
public <T1, R> StreamStage<KeyedWindowResult<K, R>> aggregate2(@Nonnull StreamStageWithKey<T1, ? extends K> stage1, @Nonnull AggregateOperation2<? super T, ? super T1, ?, ? extends R> aggrOp) {
    ensureJetEvents(computeStage, "This pipeline stage");
    ComputeStageImplBase computeStage1 = ((StageWithGroupingBase) stage1).computeStage;
    ensureJetEvents(computeStage1, "stage1");
    Transform upstream1 = computeStage1.transform;
    FunctionAdapter fnAdapter = ADAPT_TO_JET_EVENT;
    return this.computeStage.attach(new WindowGroupTransform<K, R>(asList(this.computeStage.transform, upstream1), wDef, asList(fnAdapter.adaptKeyFn(keyFn()), fnAdapter.adaptKeyFn(stage1.keyFn())), adaptAggregateOperation2(aggrOp)), singletonList((GeneralStage<?>) computeStage1), fnAdapter);
}
Also used : GeneralStage(com.hazelcast.jet.pipeline.GeneralStage) WindowGroupTransform(com.hazelcast.jet.impl.pipeline.transform.WindowGroupTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) Nonnull(javax.annotation.Nonnull)

Example 12 with Transform

use of com.hazelcast.jet.impl.pipeline.transform.Transform in project hazelcast by hazelcast.

the class StageWithKeyAndWindowImpl method aggregate3.

@Nonnull
@Override
@SuppressWarnings("rawtypes")
public <T1, T2, R> StreamStage<KeyedWindowResult<K, R>> aggregate3(@Nonnull StreamStageWithKey<T1, ? extends K> stage1, @Nonnull StreamStageWithKey<T2, ? extends K> stage2, @Nonnull AggregateOperation3<? super T, ? super T1, ? super T2, ?, ? extends R> aggrOp) {
    ComputeStageImplBase computeStage1 = ((StageWithGroupingBase) stage1).computeStage;
    ComputeStageImplBase computeStage2 = ((StageWithGroupingBase) stage2).computeStage;
    ensureJetEvents(computeStage, "This pipeline stage");
    ensureJetEvents(computeStage1, "stage1");
    ensureJetEvents(computeStage2, "stage2");
    Transform transform1 = ((StageWithGroupingBase) stage1).computeStage.transform;
    Transform transform2 = ((StageWithGroupingBase) stage2).computeStage.transform;
    FunctionAdapter fnAdapter = ADAPT_TO_JET_EVENT;
    return computeStage.attach(new WindowGroupTransform<K, R>(asList(computeStage.transform, transform1, transform2), wDef, asList(fnAdapter.adaptKeyFn(keyFn()), fnAdapter.adaptKeyFn(stage1.keyFn()), fnAdapter.adaptKeyFn(stage2.keyFn())), adaptAggregateOperation3(aggrOp)), asList((GeneralStage<?>) computeStage1, (GeneralStage<?>) computeStage2), fnAdapter);
}
Also used : GeneralStage(com.hazelcast.jet.pipeline.GeneralStage) WindowGroupTransform(com.hazelcast.jet.impl.pipeline.transform.WindowGroupTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) Nonnull(javax.annotation.Nonnull)

Example 13 with Transform

use of com.hazelcast.jet.impl.pipeline.transform.Transform in project hazelcast-jet by hazelcast.

the class GrAggBuilder method buildBatch.

@SuppressWarnings("unchecked")
public <A, R, OUT> BatchStage<OUT> buildBatch(@Nonnull AggregateOperation<A, ? extends R> aggrOp, @Nonnull DistributedBiFunction<? super K, ? super R, OUT> mapToOutputFn) {
    List<Transform> upstreamTransforms = upstreamStages.stream().map(s -> s.transform).collect(toList());
    Transform transform = new GroupTransform<>(upstreamTransforms, keyFns, aggrOp, mapToOutputFn);
    pipelineImpl.connect(upstreamTransforms, transform);
    return new BatchStageImpl<>(transform, pipelineImpl);
}
Also used : WindowGroupTransform(com.hazelcast.jet.impl.pipeline.transform.WindowGroupTransform) DistributedBiFunction(com.hazelcast.jet.function.DistributedBiFunction) ArrayList(java.util.ArrayList) BatchStage(com.hazelcast.jet.pipeline.BatchStage) Tag.tag(com.hazelcast.jet.datamodel.Tag.tag) KeyedWindowResultFunction(com.hazelcast.jet.function.KeyedWindowResultFunction) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) WindowGroupAggregateBuilder(com.hazelcast.jet.pipeline.WindowGroupAggregateBuilder) StageWithGroupingAndWindow(com.hazelcast.jet.pipeline.StageWithGroupingAndWindow) DistributedFunction(com.hazelcast.jet.function.DistributedFunction) GroupTransform(com.hazelcast.jet.impl.pipeline.transform.GroupTransform) Nonnull(javax.annotation.Nonnull) StreamStage(com.hazelcast.jet.pipeline.StreamStage) WindowDefinition(com.hazelcast.jet.pipeline.WindowDefinition) StageWithGrouping(com.hazelcast.jet.pipeline.StageWithGrouping) JetEventFunctionAdapter.adaptAggregateOperation(com.hazelcast.jet.impl.pipeline.JetEventFunctionAdapter.adaptAggregateOperation) Tag(com.hazelcast.jet.datamodel.Tag) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) ADAPT_TO_JET_EVENT(com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.ADAPT_TO_JET_EVENT) List(java.util.List) ComputeStageImplBase.ensureJetEvents(com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.ensureJetEvents) Collectors.toList(java.util.stream.Collectors.toList) JetEventFunctionAdapter.adaptKeyFn(com.hazelcast.jet.impl.pipeline.JetEventFunctionAdapter.adaptKeyFn) StreamStageWithGrouping(com.hazelcast.jet.pipeline.StreamStageWithGrouping) GroupAggregateBuilder(com.hazelcast.jet.pipeline.GroupAggregateBuilder) WindowGroupTransform(com.hazelcast.jet.impl.pipeline.transform.WindowGroupTransform) GroupTransform(com.hazelcast.jet.impl.pipeline.transform.GroupTransform) WindowGroupTransform(com.hazelcast.jet.impl.pipeline.transform.WindowGroupTransform) GroupTransform(com.hazelcast.jet.impl.pipeline.transform.GroupTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform)

Example 14 with Transform

use of com.hazelcast.jet.impl.pipeline.transform.Transform in project hazelcast-jet by hazelcast.

the class Planner method addEdges.

public void addEdges(Transform transform, Vertex toVertex, BiConsumer<Edge, Integer> configureEdgeFn) {
    int destOrdinal = 0;
    for (Transform fromTransform : transform.upstream()) {
        PlannerVertex fromPv = xform2vertex.get(fromTransform);
        Edge edge = from(fromPv.v, fromPv.nextAvailableOrdinal()).to(toVertex, destOrdinal);
        dag.edge(edge);
        configureEdgeFn.accept(edge, destOrdinal);
        destOrdinal++;
    }
}
Also used : SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) TimestampTransform(com.hazelcast.jet.impl.pipeline.transform.TimestampTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) StreamSourceTransform(com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform) Edge(com.hazelcast.jet.core.Edge)

Example 15 with Transform

use of com.hazelcast.jet.impl.pipeline.transform.Transform in project hazelcast-jet by hazelcast.

the class AggBuilder method build.

@Nonnull
@SuppressWarnings("unchecked")
public <A, R, OUT, OUT_STAGE extends GeneralStage<OUT>> OUT_STAGE build(@Nonnull AggregateOperation<A, R> aggrOp, @Nonnull CreateOutStageFn<OUT, OUT_STAGE> createOutStageFn, @Nullable WindowResultFunction<? super R, ? extends OUT> mapToOutputFn) {
    AggregateOperation adaptedAggrOp = wDef != null ? adaptAggregateOperation(aggrOp) : aggrOp;
    List<Transform> upstreamTransforms = upstreamStages.stream().map(s -> ((AbstractStage) s).transform).collect(toList());
    final Transform transform;
    if (wDef != null) {
        requireNonNull(mapToOutputFn, "wDef != null but mapToOutputFn == null");
        transform = new WindowAggregateTransform<>(upstreamTransforms, wDef, adaptedAggrOp, mapToOutputFn);
    } else {
        transform = new AggregateTransform<>(upstreamTransforms, adaptedAggrOp);
    }
    OUT_STAGE attached = createOutStageFn.get(transform, DONT_ADAPT, pipelineImpl);
    pipelineImpl.connect(upstreamTransforms, transform);
    return attached;
}
Also used : GeneralStage(com.hazelcast.jet.pipeline.GeneralStage) DONT_ADAPT(com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.DONT_ADAPT) WindowDefinition(com.hazelcast.jet.pipeline.WindowDefinition) JetEventFunctionAdapter.adaptAggregateOperation(com.hazelcast.jet.impl.pipeline.JetEventFunctionAdapter.adaptAggregateOperation) Tag(com.hazelcast.jet.datamodel.Tag) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) ArrayList(java.util.ArrayList) Tag.tag(com.hazelcast.jet.datamodel.Tag.tag) List(java.util.List) ComputeStageImplBase.ensureJetEvents(com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.ensureJetEvents) Collectors.toList(java.util.stream.Collectors.toList) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) Objects.requireNonNull(java.util.Objects.requireNonNull) WindowResultFunction(com.hazelcast.jet.function.WindowResultFunction) Nonnull(javax.annotation.Nonnull) AggregateTransform(com.hazelcast.jet.impl.pipeline.transform.AggregateTransform) WindowAggregateTransform(com.hazelcast.jet.impl.pipeline.transform.WindowAggregateTransform) Nullable(javax.annotation.Nullable) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) AggregateTransform(com.hazelcast.jet.impl.pipeline.transform.AggregateTransform) WindowAggregateTransform(com.hazelcast.jet.impl.pipeline.transform.WindowAggregateTransform) JetEventFunctionAdapter.adaptAggregateOperation(com.hazelcast.jet.impl.pipeline.JetEventFunctionAdapter.adaptAggregateOperation) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) Nonnull(javax.annotation.Nonnull)

Aggregations

Transform (com.hazelcast.jet.impl.pipeline.transform.Transform)20 Nonnull (javax.annotation.Nonnull)15 SinkTransform (com.hazelcast.jet.impl.pipeline.transform.SinkTransform)12 List (java.util.List)12 StreamSourceTransform (com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform)9 TimestampTransform (com.hazelcast.jet.impl.pipeline.transform.TimestampTransform)9 ArrayList (java.util.ArrayList)9 Collectors.toList (java.util.stream.Collectors.toList)9 AbstractTransform (com.hazelcast.jet.impl.pipeline.transform.AbstractTransform)8 GeneralStage (com.hazelcast.jet.pipeline.GeneralStage)8 FlatMapTransform (com.hazelcast.jet.impl.pipeline.transform.FlatMapTransform)7 MapTransform (com.hazelcast.jet.impl.pipeline.transform.MapTransform)7 BatchStage (com.hazelcast.jet.pipeline.BatchStage)7 FunctionEx (com.hazelcast.function.FunctionEx)6 ProcessorMetaSupplier (com.hazelcast.jet.core.ProcessorMetaSupplier)6 StreamStage (com.hazelcast.jet.pipeline.StreamStage)6 SupplierEx (com.hazelcast.function.SupplierEx)5 Traverser (com.hazelcast.jet.Traverser)5 Traversers (com.hazelcast.jet.Traversers)5 DAG (com.hazelcast.jet.core.DAG)5