Search in sources :

Example 1 with TSetEnvironment

use of edu.iu.dsc.tws.tset.env.TSetEnvironment in project beam by apache.

the class FlattenTranslatorBatch method translateNode.

@Override
public void translateNode(Flatten.PCollections<T> transform, Twister2BatchTranslationContext context) {
    Collection<PCollection<?>> pcs = context.getInputs().values();
    List<BatchTSetImpl<WindowedValue<T>>> tSets = new ArrayList<>();
    BatchTSetImpl<WindowedValue<T>> unionTSet;
    if (pcs.isEmpty()) {
        final TSetEnvironment tsetEnv = context.getEnvironment();
        unionTSet = ((BatchTSetEnvironment) tsetEnv).createSource(new Twister2EmptySource(), context.getOptions().getParallelism());
    } else {
        for (PValue pc : pcs) {
            BatchTSetImpl<WindowedValue<T>> curr = context.getInputDataSet(pc);
            tSets.add(curr);
        }
        BatchTSetImpl<WindowedValue<T>> first = tSets.remove(0);
        Collection<TSet<WindowedValue<T>>> others = new ArrayList<>();
        others.addAll(tSets);
        if (tSets.size() > 0) {
            unionTSet = first.union(others);
        } else {
            unionTSet = first;
        }
    }
    context.setOutputDataSet(context.getOutput(transform), unionTSet);
}
Also used : Twister2EmptySource(org.apache.beam.runners.twister2.translation.wrappers.Twister2EmptySource) TSetEnvironment(edu.iu.dsc.tws.tset.env.TSetEnvironment) BatchTSetEnvironment(edu.iu.dsc.tws.tset.env.BatchTSetEnvironment) ArrayList(java.util.ArrayList) TSet(edu.iu.dsc.tws.api.tset.sets.TSet) PValue(org.apache.beam.sdk.values.PValue) PCollection(org.apache.beam.sdk.values.PCollection) WindowedValue(org.apache.beam.sdk.util.WindowedValue) BatchTSetImpl(edu.iu.dsc.tws.tset.sets.batch.BatchTSetImpl)

Example 2 with TSetEnvironment

use of edu.iu.dsc.tws.tset.env.TSetEnvironment in project twister2 by DSC-SPIDAL.

the class ReadSourceTranslatorBatch method translateNode.

@Override
public void translateNode(Read.Bounded<T> transform, Twister2BatchTranslationContext context) {
    BoundedSource<T> boundedSource = transform.getSource();
    Twister2BoundedSource<T> twister2BoundedSource = new Twister2BoundedSource<T>(boundedSource, context, context.getOptions());
    final TSetEnvironment tsetEnv = context.getEnvironment();
    // TODO: need to set paralliem value
    SourceTSet<WindowedValue<T>> sourceTSet = ((BatchEnvironment) tsetEnv).createSource(twister2BoundedSource, 1);
    PCollection<T> output = context.getOutput(transform);
    context.setOutputDataSet(output, sourceTSet);
}
Also used : Twister2BoundedSource(org.apache.beam.runners.twister2.translation.wrappers.Twister2BoundedSource) WindowedValue(org.apache.beam.sdk.util.WindowedValue) BatchEnvironment(edu.iu.dsc.tws.tset.env.BatchEnvironment) TSetEnvironment(edu.iu.dsc.tws.tset.env.TSetEnvironment)

Example 3 with TSetEnvironment

use of edu.iu.dsc.tws.tset.env.TSetEnvironment in project beam by apache.

the class ReadSourceTranslatorBatch method translateNode.

@Override
public void translateNode(SplittableParDo.PrimitiveBoundedRead<T> transform, Twister2BatchTranslationContext context) {
    BoundedSource<T> boundedSource = transform.getSource();
    Twister2BoundedSource<T> twister2BoundedSource = new Twister2BoundedSource<T>(boundedSource, context, context.getOptions());
    final TSetEnvironment tsetEnv = context.getEnvironment();
    SourceTSet<WindowedValue<T>> sourceTSet = ((BatchTSetEnvironment) tsetEnv).createSource(twister2BoundedSource, context.getOptions().getParallelism());
    PCollection<T> output = context.getOutput(transform);
    context.setOutputDataSet(output, sourceTSet);
}
Also used : Twister2BoundedSource(org.apache.beam.runners.twister2.translation.wrappers.Twister2BoundedSource) BatchTSetEnvironment(edu.iu.dsc.tws.tset.env.BatchTSetEnvironment) WindowedValue(org.apache.beam.sdk.util.WindowedValue) TSetEnvironment(edu.iu.dsc.tws.tset.env.TSetEnvironment) BatchTSetEnvironment(edu.iu.dsc.tws.tset.env.BatchTSetEnvironment)

Example 4 with TSetEnvironment

use of edu.iu.dsc.tws.tset.env.TSetEnvironment in project beam by apache.

the class ImpulseTranslatorBatch method translateNode.

@Override
public void translateNode(Impulse transform, Twister2BatchTranslationContext context) {
    final TSetEnvironment tsetEnv = context.getEnvironment();
    SourceTSet<WindowedValue<byte[]>> sourceTSet = ((BatchTSetEnvironment) tsetEnv).createSource(new ImpulseSource(), 1);
    PCollection<byte[]> output = context.getOutput(transform);
    context.setOutputDataSet(output, sourceTSet);
}
Also used : BatchTSetEnvironment(edu.iu.dsc.tws.tset.env.BatchTSetEnvironment) WindowedValue(org.apache.beam.sdk.util.WindowedValue) TSetEnvironment(edu.iu.dsc.tws.tset.env.TSetEnvironment) BatchTSetEnvironment(edu.iu.dsc.tws.tset.env.BatchTSetEnvironment) ImpulseSource(org.apache.beam.runners.twister2.translators.functions.ImpulseSource)

Aggregations

TSetEnvironment (edu.iu.dsc.tws.tset.env.TSetEnvironment)4 WindowedValue (org.apache.beam.sdk.util.WindowedValue)4 BatchTSetEnvironment (edu.iu.dsc.tws.tset.env.BatchTSetEnvironment)3 Twister2BoundedSource (org.apache.beam.runners.twister2.translation.wrappers.Twister2BoundedSource)2 TSet (edu.iu.dsc.tws.api.tset.sets.TSet)1 BatchEnvironment (edu.iu.dsc.tws.tset.env.BatchEnvironment)1 BatchTSetImpl (edu.iu.dsc.tws.tset.sets.batch.BatchTSetImpl)1 ArrayList (java.util.ArrayList)1 Twister2EmptySource (org.apache.beam.runners.twister2.translation.wrappers.Twister2EmptySource)1 ImpulseSource (org.apache.beam.runners.twister2.translators.functions.ImpulseSource)1 PCollection (org.apache.beam.sdk.values.PCollection)1 PValue (org.apache.beam.sdk.values.PValue)1