Examples with SPartition edu.iu.dsc.tws.comms.stream.SPartition used on opensource projects

Example 1 with SPartition

use of edu.iu.dsc.tws.comms.stream.SPartition in project twister2 by DSC-SPIDAL.

the class WordCountWorker method execute.

@Override
public void execute(Config cfg, JobAPI.Job job, IWorkerController workerController, IPersistentVolume persistentVolume, IVolatileVolume volatileVolume) {
    this.config = cfg;
    this.id = workerController.getWorkerInfo().getWorkerID();
    this.noOfTasksPerExecutor = NO_OF_TASKS / workerController.getNumberOfWorkers();
    setupTasks(workerController);
    setupNetwork(workerController);
    // create the communication
    keyedPartition = new SPartition(channel, logicalPlan, sources, destinations, MessageTypes.OBJECT, new WordAggregate(), new HashingSelector());
    scheduleTasks();
    progress();
}

Example 2 with SPartition

use of edu.iu.dsc.tws.comms.stream.SPartition in project twister2 by DSC-SPIDAL.

the class SPartitionExample method compute.

@Override
protected void compute(WorkerEnvironment workerEnv) {
    LogicalPlanBuilder logicalPlanBuilder = LogicalPlanBuilder.plan(jobParameters.getSources(), jobParameters.getTargets(), workerEnv).withFairDistribution();
    // create the communication
    partition = new SPartition(workerEnv.getCommunicator(), logicalPlanBuilder, MessageTypes.INTEGER_ARRAY, new PartitionReceiver(), new LoadBalanceSelector());
    this.resultsVerifier = new ResultsVerifier<>(inputDataArray, (ints, args) -> ints, IntArrayComparator.getInstance());
    logicalPlanBuilder.getTargetsOnThisWorker().forEach(t -> {
        if (workerId == 0) {
            recvrInWorker0 = t;
        }
    });
    Set<Integer> tasksOfExecutor = logicalPlanBuilder.getSourcesOnThisWorker();
    // now initialize the workers
    for (int t : tasksOfExecutor) {
        // the map thread where data is produced
        Thread mapThread = new Thread(new MapWorker(t));
        mapThread.start();
    }
}