Search in sources :

Example 1 with SKeyedPartition

use of edu.iu.dsc.tws.comms.stream.SKeyedPartition in project twister2 by DSC-SPIDAL.

the class SKeyedPartitionExample method compute.

@Override
protected void compute(WorkerEnvironment workerEnv) {
    LogicalPlanBuilder logicalPlanBuilder = LogicalPlanBuilder.plan(jobParameters.getSources(), jobParameters.getTargets(), workerEnv).withFairDistribution();
    // create the communication
    partition = new SKeyedPartition(workerEnv.getCommunicator(), logicalPlanBuilder, MessageTypes.INTEGER, MessageTypes.INTEGER_ARRAY, new PartitionReceiver(), new HashingSelector());
    this.resultsVerifier = new ResultsVerifier<>(inputDataArray, (ints, args) -> new Tuple<>(-1, ints), new TupleComparator<>(// any int
    (d1, d2) -> true, IntArrayComparator.getInstance()));
    Set<Integer> tasksOfExecutor = logicalPlanBuilder.getSourcesOnThisWorker();
    // now initialize the workers
    for (int t : tasksOfExecutor) {
        // the map thread where data is produced
        Thread mapThread = new Thread(new MapWorker(t));
        mapThread.start();
    }
}
Also used : IntArrayComparator(edu.iu.dsc.tws.examples.verification.comparators.IntArrayComparator) Tuple(edu.iu.dsc.tws.api.comms.structs.Tuple) SingularReceiver(edu.iu.dsc.tws.api.comms.SingularReceiver) Set(java.util.Set) LogicalPlanBuilder(edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder) Config(edu.iu.dsc.tws.api.config.Config) MessageTypes(edu.iu.dsc.tws.api.comms.messaging.types.MessageTypes) Logger(java.util.logging.Logger) Level(java.util.logging.Level) TupleComparator(edu.iu.dsc.tws.examples.verification.comparators.TupleComparator) WorkerEnvironment(edu.iu.dsc.tws.api.resource.WorkerEnvironment) KeyedBenchWorker(edu.iu.dsc.tws.examples.comms.KeyedBenchWorker) SKeyedPartition(edu.iu.dsc.tws.comms.stream.SKeyedPartition) ResultsVerifier(edu.iu.dsc.tws.examples.verification.ResultsVerifier) HashingSelector(edu.iu.dsc.tws.comms.selectors.HashingSelector) SKeyedPartition(edu.iu.dsc.tws.comms.stream.SKeyedPartition) LogicalPlanBuilder(edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder) TupleComparator(edu.iu.dsc.tws.examples.verification.comparators.TupleComparator) HashingSelector(edu.iu.dsc.tws.comms.selectors.HashingSelector) Tuple(edu.iu.dsc.tws.api.comms.structs.Tuple)

Aggregations

SingularReceiver (edu.iu.dsc.tws.api.comms.SingularReceiver)1 MessageTypes (edu.iu.dsc.tws.api.comms.messaging.types.MessageTypes)1 Tuple (edu.iu.dsc.tws.api.comms.structs.Tuple)1 Config (edu.iu.dsc.tws.api.config.Config)1 WorkerEnvironment (edu.iu.dsc.tws.api.resource.WorkerEnvironment)1 HashingSelector (edu.iu.dsc.tws.comms.selectors.HashingSelector)1 SKeyedPartition (edu.iu.dsc.tws.comms.stream.SKeyedPartition)1 LogicalPlanBuilder (edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder)1 KeyedBenchWorker (edu.iu.dsc.tws.examples.comms.KeyedBenchWorker)1 ResultsVerifier (edu.iu.dsc.tws.examples.verification.ResultsVerifier)1 IntArrayComparator (edu.iu.dsc.tws.examples.verification.comparators.IntArrayComparator)1 TupleComparator (edu.iu.dsc.tws.examples.verification.comparators.TupleComparator)1 Set (java.util.Set)1 Level (java.util.logging.Level)1 Logger (java.util.logging.Logger)1