Search in sources :

Example 1 with SGather

use of edu.iu.dsc.tws.comms.stream.SGather in project twister2 by DSC-SPIDAL.

the class SGatherExample method compute.

@Override
protected void compute(WorkerEnvironment workerEnv) {
    if (jobParameters.getTargets() != 1) {
        LOG.warning("Setting targets to 1. Found, " + jobParameters.getTargets());
        jobParameters.getTaskStages().set(1, 1);
    }
    LogicalPlanBuilder logicalPlanBuilder = LogicalPlanBuilder.plan(jobParameters.getSources(), jobParameters.getTargets(), workerEnv);
    // create the communication
    gather = new SGather(workerEnv.getCommunicator(), logicalPlanBuilder, MessageTypes.INTEGER_ARRAY, new FinalReduceReceiver());
    Set<Integer> tasksOfExecutor = logicalPlanBuilder.getSourcesOnThisWorker();
    for (int t : tasksOfExecutor) {
        finishedSources.put(t, false);
    }
    if (tasksOfExecutor.size() == 0) {
        sourcesDone = true;
    }
    if (!logicalPlan.getLogicalIdsOfWorker(workerId).contains(logicalPlanBuilder.getTargets().iterator().next())) {
        gatherDone = true;
    }
    this.resultsVerifier = new ResultsVerifier<>(inputDataArray, (dataArray, args) -> {
        List<Tuple<Integer, int[]>> listOfArrays = new ArrayList<>();
        for (int i = 0; i < logicalPlanBuilder.getSources().size(); i++) {
            listOfArrays.add(new Tuple<>(i, dataArray));
        }
        return listOfArrays.iterator();
    }, new IteratorComparator<>(new TupleComparator<>(IntComparator.getInstance(), IntArrayComparator.getInstance())));
    // now initialize the workers
    for (int t : tasksOfExecutor) {
        // the map thread where data is produced
        Thread mapThread = new Thread(new BenchWorker.MapWorker(t));
        mapThread.start();
    }
}
Also used : IntArrayComparator(edu.iu.dsc.tws.examples.verification.comparators.IntArrayComparator) TIMING_MESSAGE_RECV(edu.iu.dsc.tws.examples.utils.bench.BenchmarkConstants.TIMING_MESSAGE_RECV) Tuple(edu.iu.dsc.tws.api.comms.structs.Tuple) BulkReceiver(edu.iu.dsc.tws.api.comms.BulkReceiver) Iterator(java.util.Iterator) Set(java.util.Set) LogicalPlanBuilder(edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder) Config(edu.iu.dsc.tws.api.config.Config) Timing(edu.iu.dsc.tws.examples.utils.bench.Timing) MessageTypes(edu.iu.dsc.tws.api.comms.messaging.types.MessageTypes) Logger(java.util.logging.Logger) SGather(edu.iu.dsc.tws.comms.stream.SGather) BenchWorker(edu.iu.dsc.tws.examples.comms.BenchWorker) TIMING_ALL_RECV(edu.iu.dsc.tws.examples.utils.bench.BenchmarkConstants.TIMING_ALL_RECV) ArrayList(java.util.ArrayList) TupleComparator(edu.iu.dsc.tws.examples.verification.comparators.TupleComparator) List(java.util.List) WorkerEnvironment(edu.iu.dsc.tws.api.resource.WorkerEnvironment) IteratorComparator(edu.iu.dsc.tws.examples.verification.comparators.IteratorComparator) IntComparator(edu.iu.dsc.tws.examples.verification.comparators.IntComparator) BenchmarkUtils(edu.iu.dsc.tws.examples.utils.bench.BenchmarkUtils) ResultsVerifier(edu.iu.dsc.tws.examples.verification.ResultsVerifier) LogicalPlanBuilder(edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder) SGather(edu.iu.dsc.tws.comms.stream.SGather) BenchWorker(edu.iu.dsc.tws.examples.comms.BenchWorker) IteratorComparator(edu.iu.dsc.tws.examples.verification.comparators.IteratorComparator) ArrayList(java.util.ArrayList) List(java.util.List) Tuple(edu.iu.dsc.tws.api.comms.structs.Tuple)

Aggregations

BulkReceiver (edu.iu.dsc.tws.api.comms.BulkReceiver)1 MessageTypes (edu.iu.dsc.tws.api.comms.messaging.types.MessageTypes)1 Tuple (edu.iu.dsc.tws.api.comms.structs.Tuple)1 Config (edu.iu.dsc.tws.api.config.Config)1 WorkerEnvironment (edu.iu.dsc.tws.api.resource.WorkerEnvironment)1 SGather (edu.iu.dsc.tws.comms.stream.SGather)1 LogicalPlanBuilder (edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder)1 BenchWorker (edu.iu.dsc.tws.examples.comms.BenchWorker)1 TIMING_ALL_RECV (edu.iu.dsc.tws.examples.utils.bench.BenchmarkConstants.TIMING_ALL_RECV)1 TIMING_MESSAGE_RECV (edu.iu.dsc.tws.examples.utils.bench.BenchmarkConstants.TIMING_MESSAGE_RECV)1 BenchmarkUtils (edu.iu.dsc.tws.examples.utils.bench.BenchmarkUtils)1 Timing (edu.iu.dsc.tws.examples.utils.bench.Timing)1 ResultsVerifier (edu.iu.dsc.tws.examples.verification.ResultsVerifier)1 IntArrayComparator (edu.iu.dsc.tws.examples.verification.comparators.IntArrayComparator)1 IntComparator (edu.iu.dsc.tws.examples.verification.comparators.IntComparator)1 IteratorComparator (edu.iu.dsc.tws.examples.verification.comparators.IteratorComparator)1 TupleComparator (edu.iu.dsc.tws.examples.verification.comparators.TupleComparator)1 ArrayList (java.util.ArrayList)1 Iterator (java.util.Iterator)1 List (java.util.List)1