Search in sources :

Example 1 with SinkStage

use of com.hazelcast.jet.pipeline.SinkStage in project hazelcast-jet by hazelcast.

the class PipelineImpl method drainTo.

@Override
public <T> SinkStage drainTo(@Nonnull Sink<T> sink, GeneralStage<?>... stagesToDrain) {
    if (stagesToDrain == null || stagesToDrain.length == 0) {
        throw new IllegalArgumentException("No stages supplied to Pipeline.drainTo()");
    }
    List<Transform> upstream = Arrays.stream(stagesToDrain).map(s -> (AbstractStage) s).map(s -> s.transform).collect(toList());
    int[] ordinalsToAdapt = IntStream.range(0, stagesToDrain.length).filter(i -> ((ComputeStageImplBase) stagesToDrain[i]).fnAdapter == ADAPT_TO_JET_EVENT).toArray();
    SinkImpl sinkImpl = (SinkImpl) sink;
    SinkTransform sinkTransform = new SinkTransform(sinkImpl, upstream, ordinalsToAdapt);
    SinkStageImpl sinkStage = new SinkStageImpl(sinkTransform, this);
    sinkImpl.onAssignToStage();
    connect(upstream, sinkTransform);
    return sinkStage;
}
Also used : GeneralStage(com.hazelcast.jet.pipeline.GeneralStage) IntStream(java.util.stream.IntStream) DONT_ADAPT(com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.DONT_ADAPT) StreamStage(com.hazelcast.jet.pipeline.StreamStage) Arrays(java.util.Arrays) BatchSource(com.hazelcast.jet.pipeline.BatchSource) BatchSourceTransform(com.hazelcast.jet.impl.pipeline.transform.BatchSourceTransform) Pipeline(com.hazelcast.jet.pipeline.Pipeline) HashMap(java.util.HashMap) StreamSource(com.hazelcast.jet.pipeline.StreamSource) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) ADAPT_TO_JET_EVENT(com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.ADAPT_TO_JET_EVENT) ArrayList(java.util.ArrayList) BatchStage(com.hazelcast.jet.pipeline.BatchStage) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) SinkStage(com.hazelcast.jet.pipeline.SinkStage) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) Map(java.util.Map) StreamSourceTransform(com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform) DAG(com.hazelcast.jet.core.DAG) Nonnull(javax.annotation.Nonnull) Sink(com.hazelcast.jet.pipeline.Sink) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) BatchSourceTransform(com.hazelcast.jet.impl.pipeline.transform.BatchSourceTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) StreamSourceTransform(com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform)

Example 2 with SinkStage

use of com.hazelcast.jet.pipeline.SinkStage in project hazelcast by hazelcast.

the class PipelineImpl method writeTo.

@Nonnull
@Override
@SuppressWarnings({ "rawtypes", "unchecked" })
public <T> SinkStage writeTo(@Nonnull Sink<? super T> sink, @Nonnull GeneralStage<? extends T> stage0, @Nonnull GeneralStage<? extends T> stage1, @Nonnull GeneralStage<? extends T>... moreStages) {
    List<GeneralStage> stages = new ArrayList<>(asList(moreStages));
    stages.add(0, stage0);
    stages.add(1, stage1);
    List<Transform> upstream = stages.stream().map(s -> (AbstractStage) s).map(s -> s.transform).collect(toList());
    int[] ordinalsToAdapt = IntStream.range(0, stages.size()).filter(i -> ((ComputeStageImplBase) stages.get(i)).fnAdapter == ADAPT_TO_JET_EVENT).toArray();
    SinkImpl sinkImpl = (SinkImpl) sink;
    SinkTransform sinkTransform = new SinkTransform(sinkImpl, upstream, ordinalsToAdapt);
    SinkStageImpl sinkStage = new SinkStageImpl(sinkTransform, this);
    sinkImpl.onAssignToStage();
    connectGeneralStages(stages, sinkTransform);
    return sinkStage;
}
Also used : IntStream(java.util.stream.IntStream) Util.escapeGraphviz(com.hazelcast.jet.impl.util.Util.escapeGraphviz) HashMap(java.util.HashMap) StreamSource(com.hazelcast.jet.pipeline.StreamSource) ArrayList(java.util.ArrayList) Collections.singletonList(java.util.Collections.singletonList) BatchStage(com.hazelcast.jet.pipeline.BatchStage) HashSet(java.util.HashSet) LinkedHashMap(java.util.LinkedHashMap) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) Arrays.asList(java.util.Arrays.asList) Map(java.util.Map) DAG(com.hazelcast.jet.core.DAG) Nonnull(javax.annotation.Nonnull) GeneralStage(com.hazelcast.jet.pipeline.GeneralStage) BatchSource(com.hazelcast.jet.pipeline.BatchSource) BatchSourceTransform(com.hazelcast.jet.impl.pipeline.transform.BatchSourceTransform) Pipeline(com.hazelcast.jet.pipeline.Pipeline) Set(java.util.Set) AbstractTransform(com.hazelcast.jet.impl.pipeline.transform.AbstractTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) File(java.io.File) ADAPT_TO_JET_EVENT(com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.ADAPT_TO_JET_EVENT) SinkStage(com.hazelcast.jet.pipeline.SinkStage) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) Entry(java.util.Map.Entry) Util.addOrIncrementIndexInName(com.hazelcast.jet.impl.util.Util.addOrIncrementIndexInName) StreamSourceTransform(com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform) StreamSourceStage(com.hazelcast.jet.pipeline.StreamSourceStage) Collections(java.util.Collections) Sink(com.hazelcast.jet.pipeline.Sink) GeneralStage(com.hazelcast.jet.pipeline.GeneralStage) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) ArrayList(java.util.ArrayList) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) BatchSourceTransform(com.hazelcast.jet.impl.pipeline.transform.BatchSourceTransform) AbstractTransform(com.hazelcast.jet.impl.pipeline.transform.AbstractTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) StreamSourceTransform(com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform) Nonnull(javax.annotation.Nonnull)

Aggregations

DAG (com.hazelcast.jet.core.DAG)2 ADAPT_TO_JET_EVENT (com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.ADAPT_TO_JET_EVENT)2 BatchSourceTransform (com.hazelcast.jet.impl.pipeline.transform.BatchSourceTransform)2 SinkTransform (com.hazelcast.jet.impl.pipeline.transform.SinkTransform)2 StreamSourceTransform (com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform)2 Transform (com.hazelcast.jet.impl.pipeline.transform.Transform)2 BatchSource (com.hazelcast.jet.pipeline.BatchSource)2 BatchStage (com.hazelcast.jet.pipeline.BatchStage)2 GeneralStage (com.hazelcast.jet.pipeline.GeneralStage)2 Pipeline (com.hazelcast.jet.pipeline.Pipeline)2 Sink (com.hazelcast.jet.pipeline.Sink)2 SinkStage (com.hazelcast.jet.pipeline.SinkStage)2 StreamSource (com.hazelcast.jet.pipeline.StreamSource)2 ArrayList (java.util.ArrayList)2 HashMap (java.util.HashMap)2 List (java.util.List)2 Map (java.util.Map)2 Collectors.toList (java.util.stream.Collectors.toList)2 IntStream (java.util.stream.IntStream)2 Nonnull (javax.annotation.Nonnull)2