Search in sources :

Example 1 with Transform

use of com.hazelcast.jet.impl.pipeline.transform.Transform in project hazelcast-jet by hazelcast.

the class GrAggBuilder method buildStream.

@SuppressWarnings("unchecked")
public <A, R, OUT> StreamStage<OUT> buildStream(@Nonnull AggregateOperation<A, ? extends R> aggrOp, @Nonnull KeyedWindowResultFunction<? super K, ? super R, OUT> mapToOutputFn) {
    List<Transform> upstreamTransforms = upstreamStages.stream().map(s -> s.transform).collect(toList());
    JetEventFunctionAdapter fnAdapter = ADAPT_TO_JET_EVENT;
    // Avoided Stream API here due to static typing issues
    List<DistributedFunction<?, ? extends K>> adaptedKeyFns = new ArrayList<>();
    for (DistributedFunction keyFn : keyFns) {
        adaptedKeyFns.add(adaptKeyFn(keyFn));
    }
    Transform transform = new WindowGroupTransform<K, A, R, JetEvent<OUT>>(upstreamTransforms, wDef, adaptedKeyFns, adaptAggregateOperation(aggrOp), fnAdapter.adaptKeyedWindowResultFn(mapToOutputFn));
    pipelineImpl.connect(upstreamTransforms, transform);
    return new StreamStageImpl<>(transform, fnAdapter, pipelineImpl);
}
Also used : WindowGroupTransform(com.hazelcast.jet.impl.pipeline.transform.WindowGroupTransform) DistributedBiFunction(com.hazelcast.jet.function.DistributedBiFunction) ArrayList(java.util.ArrayList) BatchStage(com.hazelcast.jet.pipeline.BatchStage) Tag.tag(com.hazelcast.jet.datamodel.Tag.tag) KeyedWindowResultFunction(com.hazelcast.jet.function.KeyedWindowResultFunction) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) WindowGroupAggregateBuilder(com.hazelcast.jet.pipeline.WindowGroupAggregateBuilder) StageWithGroupingAndWindow(com.hazelcast.jet.pipeline.StageWithGroupingAndWindow) DistributedFunction(com.hazelcast.jet.function.DistributedFunction) GroupTransform(com.hazelcast.jet.impl.pipeline.transform.GroupTransform) Nonnull(javax.annotation.Nonnull) StreamStage(com.hazelcast.jet.pipeline.StreamStage) WindowDefinition(com.hazelcast.jet.pipeline.WindowDefinition) StageWithGrouping(com.hazelcast.jet.pipeline.StageWithGrouping) JetEventFunctionAdapter.adaptAggregateOperation(com.hazelcast.jet.impl.pipeline.JetEventFunctionAdapter.adaptAggregateOperation) Tag(com.hazelcast.jet.datamodel.Tag) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) ADAPT_TO_JET_EVENT(com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.ADAPT_TO_JET_EVENT) List(java.util.List) ComputeStageImplBase.ensureJetEvents(com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.ensureJetEvents) Collectors.toList(java.util.stream.Collectors.toList) JetEventFunctionAdapter.adaptKeyFn(com.hazelcast.jet.impl.pipeline.JetEventFunctionAdapter.adaptKeyFn) StreamStageWithGrouping(com.hazelcast.jet.pipeline.StreamStageWithGrouping) GroupAggregateBuilder(com.hazelcast.jet.pipeline.GroupAggregateBuilder) ArrayList(java.util.ArrayList) WindowGroupTransform(com.hazelcast.jet.impl.pipeline.transform.WindowGroupTransform) GroupTransform(com.hazelcast.jet.impl.pipeline.transform.GroupTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) WindowGroupTransform(com.hazelcast.jet.impl.pipeline.transform.WindowGroupTransform) DistributedFunction(com.hazelcast.jet.function.DistributedFunction)

Example 2 with Transform

use of com.hazelcast.jet.impl.pipeline.transform.Transform in project hazelcast-jet by hazelcast.

the class PipelineImpl method drainTo.

@Override
public <T> SinkStage drainTo(@Nonnull Sink<T> sink, GeneralStage<?>... stagesToDrain) {
    if (stagesToDrain == null || stagesToDrain.length == 0) {
        throw new IllegalArgumentException("No stages supplied to Pipeline.drainTo()");
    }
    List<Transform> upstream = Arrays.stream(stagesToDrain).map(s -> (AbstractStage) s).map(s -> s.transform).collect(toList());
    int[] ordinalsToAdapt = IntStream.range(0, stagesToDrain.length).filter(i -> ((ComputeStageImplBase) stagesToDrain[i]).fnAdapter == ADAPT_TO_JET_EVENT).toArray();
    SinkImpl sinkImpl = (SinkImpl) sink;
    SinkTransform sinkTransform = new SinkTransform(sinkImpl, upstream, ordinalsToAdapt);
    SinkStageImpl sinkStage = new SinkStageImpl(sinkTransform, this);
    sinkImpl.onAssignToStage();
    connect(upstream, sinkTransform);
    return sinkStage;
}
Also used : GeneralStage(com.hazelcast.jet.pipeline.GeneralStage) IntStream(java.util.stream.IntStream) DONT_ADAPT(com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.DONT_ADAPT) StreamStage(com.hazelcast.jet.pipeline.StreamStage) Arrays(java.util.Arrays) BatchSource(com.hazelcast.jet.pipeline.BatchSource) BatchSourceTransform(com.hazelcast.jet.impl.pipeline.transform.BatchSourceTransform) Pipeline(com.hazelcast.jet.pipeline.Pipeline) HashMap(java.util.HashMap) StreamSource(com.hazelcast.jet.pipeline.StreamSource) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) ADAPT_TO_JET_EVENT(com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.ADAPT_TO_JET_EVENT) ArrayList(java.util.ArrayList) BatchStage(com.hazelcast.jet.pipeline.BatchStage) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) SinkStage(com.hazelcast.jet.pipeline.SinkStage) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) Map(java.util.Map) StreamSourceTransform(com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform) DAG(com.hazelcast.jet.core.DAG) Nonnull(javax.annotation.Nonnull) Sink(com.hazelcast.jet.pipeline.Sink) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) BatchSourceTransform(com.hazelcast.jet.impl.pipeline.transform.BatchSourceTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) StreamSourceTransform(com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform)

Example 3 with Transform

use of com.hazelcast.jet.impl.pipeline.transform.Transform in project hazelcast-jet by hazelcast.

the class Planner method createDag.

DAG createDag() {
    Map<Transform, List<Transform>> adjacencyMap = pipeline.adjacencyMap();
    validateNoLeakage(adjacencyMap);
    // Calculate greatest common denominator of frame lengths from all transforms in the pipeline
    long frameSizeGcd = Util.gcd(adjacencyMap.keySet().stream().map(Transform::watermarkFrameSize).filter(frameSize -> frameSize > 0).mapToLong(i -> i).toArray());
    WatermarkEmissionPolicy emitPolicy = frameSizeGcd > 0 ? emitByFrame(tumblingWinPolicy(frameSizeGcd)) : noThrottling();
    // Replace emission policy
    for (Transform transform : adjacencyMap.keySet()) {
        if (transform instanceof StreamSourceTransform) {
            StreamSourceTransform t = (StreamSourceTransform) transform;
            if (t.getWmParams() != null) {
                t.setWmGenerationParams(t.getWmParams().withEmitPolicy(emitPolicy));
            }
        } else if (transform instanceof TimestampTransform) {
            TimestampTransform t = (TimestampTransform) transform;
            t.setWmGenerationParams(t.getWmGenParams().withEmitPolicy(emitPolicy));
        }
    }
    Iterable<Transform> sorted = topologicalSort(adjacencyMap, Object::toString);
    for (Transform transform : sorted) {
        transform.addToDag(this);
    }
    return dag;
}
Also used : Processor(com.hazelcast.jet.core.Processor) HashMap(java.util.HashMap) HashSet(java.util.HashSet) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) Map(java.util.Map) WatermarkEmissionPolicy.noThrottling(com.hazelcast.jet.core.WatermarkEmissionPolicy.noThrottling) BiConsumer(java.util.function.BiConsumer) TimestampTransform(com.hazelcast.jet.impl.pipeline.transform.TimestampTransform) Edge.from(com.hazelcast.jet.core.Edge.from) DAG(com.hazelcast.jet.core.DAG) Edge(com.hazelcast.jet.core.Edge) TopologicalSorter.topologicalSort(com.hazelcast.jet.impl.TopologicalSorter.topologicalSort) Nonnull(javax.annotation.Nonnull) ProcessorSupplier(com.hazelcast.jet.core.ProcessorSupplier) DistributedSupplier(com.hazelcast.jet.function.DistributedSupplier) WatermarkEmissionPolicy.emitByFrame(com.hazelcast.jet.core.WatermarkEmissionPolicy.emitByFrame) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) WatermarkEmissionPolicy(com.hazelcast.jet.core.WatermarkEmissionPolicy) Set(java.util.Set) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) Consumer(java.util.function.Consumer) Vertex(com.hazelcast.jet.core.Vertex) Util(com.hazelcast.jet.impl.util.Util) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) Entry(java.util.Map.Entry) StreamSourceTransform(com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform) SlidingWindowPolicy.tumblingWinPolicy(com.hazelcast.jet.core.SlidingWindowPolicy.tumblingWinPolicy) WatermarkEmissionPolicy(com.hazelcast.jet.core.WatermarkEmissionPolicy) StreamSourceTransform(com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) TimestampTransform(com.hazelcast.jet.impl.pipeline.transform.TimestampTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) StreamSourceTransform(com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform) TimestampTransform(com.hazelcast.jet.impl.pipeline.transform.TimestampTransform)

Example 4 with Transform

use of com.hazelcast.jet.impl.pipeline.transform.Transform in project hazelcast by hazelcast.

the class ComputeStageImplBase method attachMapUsingPartitionedServiceAsyncBatched.

@Nonnull
@SuppressWarnings({ "unchecked", "rawtypes" })
<S, K, R, RET> RET attachMapUsingPartitionedServiceAsyncBatched(@Nonnull ServiceFactory<?, S> serviceFactory, int maxBatchSize, @Nonnull FunctionEx<? super T, ? extends K> partitionKeyFn, @Nonnull BiFunctionEx<? super S, ? super List<T>, ? extends CompletableFuture<List<R>>> mapAsyncFn) {
    checkSerializable(mapAsyncFn, "mapAsyncFn");
    checkSerializable(partitionKeyFn, "partitionKeyFn");
    serviceFactory = moveAttachedFilesToPipeline(serviceFactory);
    BiFunctionEx<? super S, ? super List<T>, ? extends CompletableFuture<List<Traverser<R>>>> flatMapAsyncFn = (s, items) -> mapAsyncFn.apply(s, items).thenApply(list -> toList(list, Traversers::singleton));
    BiFunctionEx adaptedFlatMapFn = fnAdapter.adaptFlatMapUsingServiceAsyncBatchedFn(flatMapAsyncFn);
    FunctionEx adaptedPartitionKeyFn = fnAdapter.adaptKeyFn(partitionKeyFn);
    // Here we flatten the result from List<Traverser<R>> to Traverser<R>.
    // The former is used in pipeline API, the latter in core API.
    BiFunctionEx<? super S, ? super List<T>, ? extends CompletableFuture<Traverser<R>>> flattenedFn = (svc, items) -> {
        // R might actually be JetEvent<R> -- we can't represent this with static types
        CompletableFuture<List<Traverser<R>>> f = (CompletableFuture<List<Traverser<R>>>) adaptedFlatMapFn.apply(svc, items);
        return f.thenApply(res -> traverseIterable(res).flatMap(Function.identity()));
    };
    PartitionedProcessorTransform processorTransform = flatMapUsingServiceAsyncBatchedPartitionedTransform(transform, "map", serviceFactory, MAX_CONCURRENT_ASYNC_BATCHES, maxBatchSize, flattenedFn, adaptedPartitionKeyFn);
    return attach(processorTransform, fnAdapter);
}
Also used : Traverser(com.hazelcast.jet.Traverser) PartitionedProcessorTransform.partitionedCustomProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.partitionedCustomProcessorTransform) PartitionedProcessorTransform.flatMapUsingServiceAsyncBatchedPartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServiceAsyncBatchedPartitionedTransform) ProcessorTransform.flatMapUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceTransform) ProcessorTransform.mapUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.mapUsingServiceTransform) Collections.singletonList(java.util.Collections.singletonList) BiFunctionEx(com.hazelcast.function.BiFunctionEx) JoinClause(com.hazelcast.jet.pipeline.JoinClause) Traversers.traverseIterable(com.hazelcast.jet.Traversers.traverseIterable) Arrays.asList(java.util.Arrays.asList) ProcessorTransform.flatMapUsingServiceAsyncTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceAsyncTransform) Preconditions.checkTrue(com.hazelcast.internal.util.Preconditions.checkTrue) PredicateEx(com.hazelcast.function.PredicateEx) GlobalMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.GlobalMapStatefulTransform) PeekTransform(com.hazelcast.jet.impl.pipeline.transform.PeekTransform) GeneralStage(com.hazelcast.jet.pipeline.GeneralStage) FunctionEx(com.hazelcast.function.FunctionEx) PartitionedProcessorTransform.flatMapUsingServiceAsyncPartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServiceAsyncPartitionedTransform) Collections.emptyList(java.util.Collections.emptyList) SupplierEx(com.hazelcast.function.SupplierEx) List(java.util.List) JetEvent.jetEvent(com.hazelcast.jet.impl.JetEvent.jetEvent) ProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform) MergeTransform(com.hazelcast.jet.impl.pipeline.transform.MergeTransform) PartitionedProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform) Sink(com.hazelcast.jet.pipeline.Sink) MapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.MapStatefulTransform) ComparatorEx(com.hazelcast.function.ComparatorEx) Util.checkSerializable(com.hazelcast.jet.impl.util.Util.checkSerializable) CompletableFuture(java.util.concurrent.CompletableFuture) Function(java.util.function.Function) JetException(com.hazelcast.jet.JetException) BatchStage(com.hazelcast.jet.pipeline.BatchStage) GlobalFlatMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.GlobalFlatMapStatefulTransform) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) ProcessorTransform.filterUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.filterUsingServiceTransform) PartitionedProcessorTransform.flatMapUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServicePartitionedTransform) DEFAULT_IDLE_TIMEOUT(com.hazelcast.jet.core.EventTimePolicy.DEFAULT_IDLE_TIMEOUT) MapTransform(com.hazelcast.jet.impl.pipeline.transform.MapTransform) TimestampTransform(com.hazelcast.jet.impl.pipeline.transform.TimestampTransform) ServiceFactory(com.hazelcast.jet.pipeline.ServiceFactory) Nonnull(javax.annotation.Nonnull) Nullable(javax.annotation.Nullable) BiPredicateEx(com.hazelcast.function.BiPredicateEx) FlatMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.FlatMapStatefulTransform) StreamStage(com.hazelcast.jet.pipeline.StreamStage) ProcessorTransform.flatMapUsingServiceAsyncBatchedTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceAsyncBatchedTransform) Util.toList(com.hazelcast.jet.impl.util.Util.toList) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) PartitionedProcessorTransform.filterUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.filterUsingServicePartitionedTransform) WatermarkPolicy.limitingLag(com.hazelcast.jet.core.WatermarkPolicy.limitingLag) Traversers(com.hazelcast.jet.Traversers) ProcessorTransform.customProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.customProcessorTransform) AbstractTransform(com.hazelcast.jet.impl.pipeline.transform.AbstractTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) PartitionedProcessorTransform.mapUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.mapUsingServicePartitionedTransform) SinkStage(com.hazelcast.jet.pipeline.SinkStage) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) HashJoinTransform(com.hazelcast.jet.impl.pipeline.transform.HashJoinTransform) SortTransform(com.hazelcast.jet.impl.pipeline.transform.SortTransform) EventTimePolicy.eventTimePolicy(com.hazelcast.jet.core.EventTimePolicy.eventTimePolicy) FlatMapTransform(com.hazelcast.jet.impl.pipeline.transform.FlatMapTransform) TriFunction(com.hazelcast.jet.function.TriFunction) CompletableFuture(java.util.concurrent.CompletableFuture) DEFAULT_IDLE_TIMEOUT(com.hazelcast.jet.core.EventTimePolicy.DEFAULT_IDLE_TIMEOUT) Traverser(com.hazelcast.jet.Traverser) PartitionedProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform) BiFunctionEx(com.hazelcast.function.BiFunctionEx) FunctionEx(com.hazelcast.function.FunctionEx) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) Collections.singletonList(java.util.Collections.singletonList) Arrays.asList(java.util.Arrays.asList) Collections.emptyList(java.util.Collections.emptyList) List(java.util.List) Util.toList(com.hazelcast.jet.impl.util.Util.toList) BiFunctionEx(com.hazelcast.function.BiFunctionEx) Nonnull(javax.annotation.Nonnull)

Example 5 with Transform

use of com.hazelcast.jet.impl.pipeline.transform.Transform in project hazelcast by hazelcast.

the class ComputeStageImplBase method attachMapUsingServiceAsyncBatched.

@Nonnull
@SuppressWarnings({ "unchecked", "rawtypes" })
<S, R, RET> RET attachMapUsingServiceAsyncBatched(@Nonnull ServiceFactory<?, S> serviceFactory, int maxBatchSize, @Nonnull BiFunctionEx<? super S, ? super List<T>, ? extends CompletableFuture<List<Traverser<R>>>> flatMapAsyncBatchedFn) {
    checkSerializable(flatMapAsyncBatchedFn, "mapAsyncBatchedFn");
    serviceFactory = moveAttachedFilesToPipeline(serviceFactory);
    BiFunctionEx adaptedFn = fnAdapter.adaptFlatMapUsingServiceAsyncBatchedFn(flatMapAsyncBatchedFn);
    // Here we flatten the result from List<Traverser<R>> to Traverser<R>.
    // The former is used in pipeline API, the latter in core API.
    BiFunctionEx<? super S, ? super List<T>, ? extends CompletableFuture<Traverser<R>>> flattenedFn = (svc, items) -> {
        // R might actually be JetEvent<R> -- we can't represent this with static types
        CompletableFuture<List<Traverser<R>>> f = (CompletableFuture<List<Traverser<R>>>) adaptedFn.apply(svc, items);
        return f.thenApply(res -> traverseIterable(res).flatMap(Function.identity()));
    };
    ProcessorTransform processorTransform = flatMapUsingServiceAsyncBatchedTransform(transform, "map", serviceFactory, MAX_CONCURRENT_ASYNC_BATCHES, maxBatchSize, flattenedFn);
    return attach(processorTransform, fnAdapter);
}
Also used : Traverser(com.hazelcast.jet.Traverser) PartitionedProcessorTransform.partitionedCustomProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.partitionedCustomProcessorTransform) PartitionedProcessorTransform.flatMapUsingServiceAsyncBatchedPartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServiceAsyncBatchedPartitionedTransform) ProcessorTransform.flatMapUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceTransform) ProcessorTransform.mapUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.mapUsingServiceTransform) Collections.singletonList(java.util.Collections.singletonList) BiFunctionEx(com.hazelcast.function.BiFunctionEx) JoinClause(com.hazelcast.jet.pipeline.JoinClause) Traversers.traverseIterable(com.hazelcast.jet.Traversers.traverseIterable) Arrays.asList(java.util.Arrays.asList) ProcessorTransform.flatMapUsingServiceAsyncTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceAsyncTransform) Preconditions.checkTrue(com.hazelcast.internal.util.Preconditions.checkTrue) PredicateEx(com.hazelcast.function.PredicateEx) GlobalMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.GlobalMapStatefulTransform) PeekTransform(com.hazelcast.jet.impl.pipeline.transform.PeekTransform) GeneralStage(com.hazelcast.jet.pipeline.GeneralStage) FunctionEx(com.hazelcast.function.FunctionEx) PartitionedProcessorTransform.flatMapUsingServiceAsyncPartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServiceAsyncPartitionedTransform) Collections.emptyList(java.util.Collections.emptyList) SupplierEx(com.hazelcast.function.SupplierEx) List(java.util.List) JetEvent.jetEvent(com.hazelcast.jet.impl.JetEvent.jetEvent) ProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform) MergeTransform(com.hazelcast.jet.impl.pipeline.transform.MergeTransform) PartitionedProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform) Sink(com.hazelcast.jet.pipeline.Sink) MapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.MapStatefulTransform) ComparatorEx(com.hazelcast.function.ComparatorEx) Util.checkSerializable(com.hazelcast.jet.impl.util.Util.checkSerializable) CompletableFuture(java.util.concurrent.CompletableFuture) Function(java.util.function.Function) JetException(com.hazelcast.jet.JetException) BatchStage(com.hazelcast.jet.pipeline.BatchStage) GlobalFlatMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.GlobalFlatMapStatefulTransform) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) ProcessorTransform.filterUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.filterUsingServiceTransform) PartitionedProcessorTransform.flatMapUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServicePartitionedTransform) DEFAULT_IDLE_TIMEOUT(com.hazelcast.jet.core.EventTimePolicy.DEFAULT_IDLE_TIMEOUT) MapTransform(com.hazelcast.jet.impl.pipeline.transform.MapTransform) TimestampTransform(com.hazelcast.jet.impl.pipeline.transform.TimestampTransform) ServiceFactory(com.hazelcast.jet.pipeline.ServiceFactory) Nonnull(javax.annotation.Nonnull) Nullable(javax.annotation.Nullable) BiPredicateEx(com.hazelcast.function.BiPredicateEx) FlatMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.FlatMapStatefulTransform) StreamStage(com.hazelcast.jet.pipeline.StreamStage) ProcessorTransform.flatMapUsingServiceAsyncBatchedTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceAsyncBatchedTransform) Util.toList(com.hazelcast.jet.impl.util.Util.toList) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) PartitionedProcessorTransform.filterUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.filterUsingServicePartitionedTransform) WatermarkPolicy.limitingLag(com.hazelcast.jet.core.WatermarkPolicy.limitingLag) Traversers(com.hazelcast.jet.Traversers) ProcessorTransform.customProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.customProcessorTransform) AbstractTransform(com.hazelcast.jet.impl.pipeline.transform.AbstractTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) PartitionedProcessorTransform.mapUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.mapUsingServicePartitionedTransform) SinkStage(com.hazelcast.jet.pipeline.SinkStage) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) HashJoinTransform(com.hazelcast.jet.impl.pipeline.transform.HashJoinTransform) SortTransform(com.hazelcast.jet.impl.pipeline.transform.SortTransform) EventTimePolicy.eventTimePolicy(com.hazelcast.jet.core.EventTimePolicy.eventTimePolicy) FlatMapTransform(com.hazelcast.jet.impl.pipeline.transform.FlatMapTransform) TriFunction(com.hazelcast.jet.function.TriFunction) CompletableFuture(java.util.concurrent.CompletableFuture) DEFAULT_IDLE_TIMEOUT(com.hazelcast.jet.core.EventTimePolicy.DEFAULT_IDLE_TIMEOUT) PartitionedProcessorTransform.partitionedCustomProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.partitionedCustomProcessorTransform) ProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform) PartitionedProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform) ProcessorTransform.customProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.customProcessorTransform) Traverser(com.hazelcast.jet.Traverser) Collections.singletonList(java.util.Collections.singletonList) Arrays.asList(java.util.Arrays.asList) Collections.emptyList(java.util.Collections.emptyList) List(java.util.List) Util.toList(com.hazelcast.jet.impl.util.Util.toList) BiFunctionEx(com.hazelcast.function.BiFunctionEx) Nonnull(javax.annotation.Nonnull)

Aggregations

Transform (com.hazelcast.jet.impl.pipeline.transform.Transform)20 Nonnull (javax.annotation.Nonnull)15 SinkTransform (com.hazelcast.jet.impl.pipeline.transform.SinkTransform)12 List (java.util.List)12 StreamSourceTransform (com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform)9 TimestampTransform (com.hazelcast.jet.impl.pipeline.transform.TimestampTransform)9 ArrayList (java.util.ArrayList)9 Collectors.toList (java.util.stream.Collectors.toList)9 AbstractTransform (com.hazelcast.jet.impl.pipeline.transform.AbstractTransform)8 GeneralStage (com.hazelcast.jet.pipeline.GeneralStage)8 FlatMapTransform (com.hazelcast.jet.impl.pipeline.transform.FlatMapTransform)7 MapTransform (com.hazelcast.jet.impl.pipeline.transform.MapTransform)7 BatchStage (com.hazelcast.jet.pipeline.BatchStage)7 FunctionEx (com.hazelcast.function.FunctionEx)6 ProcessorMetaSupplier (com.hazelcast.jet.core.ProcessorMetaSupplier)6 StreamStage (com.hazelcast.jet.pipeline.StreamStage)6 SupplierEx (com.hazelcast.function.SupplierEx)5 Traverser (com.hazelcast.jet.Traverser)5 Traversers (com.hazelcast.jet.Traversers)5 DAG (com.hazelcast.jet.core.DAG)5