Search in sources :

Example 1 with ReduceOperationFunction

use of edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction in project twister2 by DSC-SPIDAL.

the class SAllReduceExample method compute.

@Override
protected void compute(WorkerEnvironment workerEnv) {
    LogicalPlanBuilder logicalPlanBuilder = LogicalPlanBuilder.plan(jobParameters.getSources(), jobParameters.getTargets(), workerEnv).withFairDistribution();
    // create the communication
    reduce = new SAllReduce(workerEnv.getCommunicator(), logicalPlanBuilder, MessageTypes.INTEGER_ARRAY, new ReduceOperationFunction(Op.SUM, MessageTypes.INTEGER_ARRAY), new FinalSingularReceiver());
    Set<Integer> tasksOfExecutor = logicalPlanBuilder.getSourcesOnThisWorker();
    for (int t : tasksOfExecutor) {
        finishedSources.put(t, false);
    }
    sourcesDone = tasksOfExecutor.size() == 0;
    Set<Integer> targetTasksOfExecutor = logicalPlanBuilder.getTargetsOnThisWorker();
    for (int taskId : targetTasksOfExecutor) {
        if (logicalPlanBuilder.getTargets().contains(taskId)) {
            reduceDone = false;
        }
        if (workerId == 0) {
            receiverInWorker0 = taskId;
        }
    }
    this.resultsVerifier = new ResultsVerifier<>(inputDataArray, (array, args) -> {
        int sourcesCount = jobParameters.getTaskStages().get(0);
        return GeneratorUtils.multiplyIntArray(array, sourcesCount);
    }, IntArrayComparator.getInstance());
    // now initialize the workers
    for (int t : tasksOfExecutor) {
        // the map thread where data is produced
        Thread mapThread = new Thread(new MapWorker(t));
        mapThread.start();
    }
}
Also used : IntArrayComparator(edu.iu.dsc.tws.examples.verification.comparators.IntArrayComparator) TIMING_MESSAGE_RECV(edu.iu.dsc.tws.examples.utils.bench.BenchmarkConstants.TIMING_MESSAGE_RECV) SingularReceiver(edu.iu.dsc.tws.api.comms.SingularReceiver) GeneratorUtils(edu.iu.dsc.tws.examples.verification.GeneratorUtils) Set(java.util.Set) LogicalPlanBuilder(edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder) Config(edu.iu.dsc.tws.api.config.Config) Timing(edu.iu.dsc.tws.examples.utils.bench.Timing) MessageTypes(edu.iu.dsc.tws.api.comms.messaging.types.MessageTypes) Logger(java.util.logging.Logger) BenchWorker(edu.iu.dsc.tws.examples.comms.BenchWorker) TIMING_ALL_RECV(edu.iu.dsc.tws.examples.utils.bench.BenchmarkConstants.TIMING_ALL_RECV) WorkerEnvironment(edu.iu.dsc.tws.api.resource.WorkerEnvironment) Op(edu.iu.dsc.tws.api.comms.Op) ReduceOperationFunction(edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction) BenchmarkUtils(edu.iu.dsc.tws.examples.utils.bench.BenchmarkUtils) ResultsVerifier(edu.iu.dsc.tws.examples.verification.ResultsVerifier) SAllReduce(edu.iu.dsc.tws.comms.stream.SAllReduce) ReduceOperationFunction(edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction) LogicalPlanBuilder(edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder) SAllReduce(edu.iu.dsc.tws.comms.stream.SAllReduce)

Example 2 with ReduceOperationFunction

use of edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction in project twister2 by DSC-SPIDAL.

the class SKeyedReduceExample method compute.

@Override
protected void compute(WorkerEnvironment workerEnv) {
    LogicalPlanBuilder logicalPlanBuilder = LogicalPlanBuilder.plan(jobParameters.getSources(), jobParameters.getTargets(), workerEnv).withFairDistribution();
    keyedReduce = new SKeyedReduce(workerEnv.getCommunicator(), logicalPlanBuilder, MessageTypes.INTEGER, MessageTypes.INTEGER_ARRAY, new ReduceOperationFunction(Op.SUM, MessageTypes.INTEGER_ARRAY), new FinalSingularReceiver(jobParameters.getIterations()), new SimpleKeyBasedSelector());
    Set<Integer> tasksOfExecutor = logicalPlanBuilder.getSourcesOnThisWorker();
    for (int t : tasksOfExecutor) {
        finishedSources.put(t, false);
    }
    if (tasksOfExecutor.size() == 0) {
        sourcesDone = true;
    }
    LOG.log(Level.INFO, String.format("%d Sources %s target %d this %s", workerId, logicalPlanBuilder.getSources(), 1, tasksOfExecutor));
    // now initialize the workers
    for (int t : tasksOfExecutor) {
        // the map thread where data is produced
        Thread mapThread = new Thread(new KeyedBenchWorker.MapWorker(t));
        mapThread.start();
    }
}
Also used : SimpleKeyBasedSelector(edu.iu.dsc.tws.comms.selectors.SimpleKeyBasedSelector) ReduceOperationFunction(edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction) KeyedBenchWorker(edu.iu.dsc.tws.examples.comms.KeyedBenchWorker) LogicalPlanBuilder(edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder) SKeyedReduce(edu.iu.dsc.tws.comms.stream.SKeyedReduce)

Example 3 with ReduceOperationFunction

use of edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction in project twister2 by DSC-SPIDAL.

the class BReduceExample method compute.

@Override
protected void compute(WorkerEnvironment workerEnv) {
    LogicalPlanBuilder logicalPlanBuilder = LogicalPlanBuilder.plan(jobParameters.getSources(), jobParameters.getTargets(), workerEnv).withFairDistribution();
    // create the communication
    reduce = new BReduce(workerEnv.getCommunicator(), logicalPlanBuilder, new ReduceOperationFunction(Op.SUM, MessageTypes.INTEGER_ARRAY), new FinalSingularReceiver(), MessageTypes.INTEGER_ARRAY);
    Set<Integer> tasksOfExecutor = logicalPlanBuilder.getSourcesOnThisWorker();
    for (int t : tasksOfExecutor) {
        finishedSources.put(t, false);
    }
    if (tasksOfExecutor.size() == 0) {
        sourcesDone = true;
    }
    this.resultsVerifier = new ResultsVerifier<>(inputDataArray, (ints, args) -> GeneratorUtils.multiplyIntArray(ints, jobParameters.getTotalIterations() * logicalPlanBuilder.getSources().size()), IntArrayComparator.getInstance());
    LOG.log(Level.INFO, String.format("%d Sources %s target %d this %s", workerId, logicalPlanBuilder.getSources(), logicalPlanBuilder.getTargets().iterator().next(), tasksOfExecutor));
    // now initialize the workers
    for (int t : tasksOfExecutor) {
        // the map thread where data is produced
        Thread mapThread = new Thread(new MapWorker(t));
        mapThread.start();
    }
}
Also used : BReduce(edu.iu.dsc.tws.comms.batch.BReduce) IntArrayComparator(edu.iu.dsc.tws.examples.verification.comparators.IntArrayComparator) SingularReceiver(edu.iu.dsc.tws.api.comms.SingularReceiver) GeneratorUtils(edu.iu.dsc.tws.examples.verification.GeneratorUtils) Set(java.util.Set) LogicalPlanBuilder(edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder) Config(edu.iu.dsc.tws.api.config.Config) Timing(edu.iu.dsc.tws.examples.utils.bench.Timing) MessageTypes(edu.iu.dsc.tws.api.comms.messaging.types.MessageTypes) Logger(java.util.logging.Logger) BenchWorker(edu.iu.dsc.tws.examples.comms.BenchWorker) BenchmarkConstants(edu.iu.dsc.tws.examples.utils.bench.BenchmarkConstants) Level(java.util.logging.Level) WorkerEnvironment(edu.iu.dsc.tws.api.resource.WorkerEnvironment) Op(edu.iu.dsc.tws.api.comms.Op) ReduceOperationFunction(edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction) BenchmarkUtils(edu.iu.dsc.tws.examples.utils.bench.BenchmarkUtils) ResultsVerifier(edu.iu.dsc.tws.examples.verification.ResultsVerifier) Comparator(java.util.Comparator) ReduceOperationFunction(edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction) LogicalPlanBuilder(edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder) BReduce(edu.iu.dsc.tws.comms.batch.BReduce)

Example 4 with ReduceOperationFunction

use of edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction in project twister2 by DSC-SPIDAL.

the class SVMCommsReducer method execute.

@Override
protected void execute(WorkerEnvironment workerEnv) {
    Set<Integer> sources = new HashSet<>();
    Integer noOfSourceTasks = svmJobParameters.getParallelism();
    for (int i = 0; i < noOfSourceTasks; i++) {
        sources.add(i);
    }
    int target = noOfSourceTasks;
    reduce = new BReduce(workerEnv.getCommunicator(), logicalPlan, sources, target, new ReduceOperationFunction(Op.SUM, MessageTypes.DOUBLE), new FinalSingularReceiver(), MessageTypes.DOUBLE);
    Set<Integer> tasksOfExecutor = Utils.getTasksOfExecutor(workerId, logicalPlan, taskStages, 0);
    for (int t : tasksOfExecutor) {
        finishedSources.put(t, false);
    }
    if (tasksOfExecutor.size() == 0) {
        sourcesDone = true;
    }
    if (!logicalPlan.getLogicalIdsOfWorker(workerId).contains(target)) {
        reduceDone = true;
    }
    LOG.log(Level.INFO, String.format("%d Sources %s target %d this %s", workerId, sources, target, tasksOfExecutor));
    for (int t : tasksOfExecutor) {
        Thread mapThread = new Thread(new DataStreamer(t));
        mapThread.start();
    }
}
Also used : ReduceOperationFunction(edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction) HashSet(java.util.HashSet) BReduce(edu.iu.dsc.tws.comms.batch.BReduce)

Example 5 with ReduceOperationFunction

use of edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction in project twister2 by DSC-SPIDAL.

the class BAllReduceExample method compute.

@Override
protected void compute(WorkerEnvironment workerEnv) {
    Integer noOfSourceTasks = jobParameters.getTaskStages().get(0);
    Set<Integer> sources = IntStream.range(0, noOfSourceTasks).boxed().collect(Collectors.toSet());
    int noOfTargetTasks = jobParameters.getTaskStages().get(1);
    Set<Integer> targets = IntStream.range(noOfSourceTasks, noOfTargetTasks + noOfSourceTasks).boxed().collect(Collectors.toSet());
    LogicalPlanBuilder logicalPlanBuilder = LogicalPlanBuilder.plan(jobParameters.getSources(), jobParameters.getTargets(), workerEnv).withFairDistribution();
    // create the communication
    reduce = new BAllReduce(workerEnv.getCommunicator(), logicalPlanBuilder, new ReduceOperationFunction(Op.SUM, MessageTypes.INTEGER_ARRAY), new FinalSingularReceiver(), MessageTypes.INTEGER_ARRAY);
    Set<Integer> tasksOfExecutor = logicalPlanBuilder.getSourcesOnThisWorker();
    for (int t : tasksOfExecutor) {
        finishedSources.put(t, false);
    }
    if (tasksOfExecutor.size() == 0) {
        sourcesDone = true;
    }
    this.resultsVerifier = new ResultsVerifier<>(inputDataArray, (ints, args) -> GeneratorUtils.multiplyIntArray(ints, jobParameters.getIterations() * sources.size()), IntArrayComparator.getInstance());
    LOG.log(Level.INFO, String.format("%d Sources %s target %s this %s", workerId, sources, targets, tasksOfExecutor));
    // now initialize the workers
    for (int t : tasksOfExecutor) {
        // the map thread where data is produced
        Thread mapThread = new Thread(new BenchWorker.MapWorker(t));
        mapThread.start();
    }
}
Also used : IntStream(java.util.stream.IntStream) IntArrayComparator(edu.iu.dsc.tws.examples.verification.comparators.IntArrayComparator) SingularReceiver(edu.iu.dsc.tws.api.comms.SingularReceiver) BAllReduce(edu.iu.dsc.tws.comms.batch.BAllReduce) GeneratorUtils(edu.iu.dsc.tws.examples.verification.GeneratorUtils) Set(java.util.Set) LogicalPlanBuilder(edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder) Config(edu.iu.dsc.tws.api.config.Config) Timing(edu.iu.dsc.tws.examples.utils.bench.Timing) MessageTypes(edu.iu.dsc.tws.api.comms.messaging.types.MessageTypes) Logger(java.util.logging.Logger) BenchWorker(edu.iu.dsc.tws.examples.comms.BenchWorker) Collectors(java.util.stream.Collectors) BenchmarkConstants(edu.iu.dsc.tws.examples.utils.bench.BenchmarkConstants) Level(java.util.logging.Level) WorkerEnvironment(edu.iu.dsc.tws.api.resource.WorkerEnvironment) Op(edu.iu.dsc.tws.api.comms.Op) ReduceOperationFunction(edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction) BenchmarkUtils(edu.iu.dsc.tws.examples.utils.bench.BenchmarkUtils) ResultsVerifier(edu.iu.dsc.tws.examples.verification.ResultsVerifier) Comparator(java.util.Comparator) ReduceOperationFunction(edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction) BenchWorker(edu.iu.dsc.tws.examples.comms.BenchWorker) LogicalPlanBuilder(edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder) BAllReduce(edu.iu.dsc.tws.comms.batch.BAllReduce)

Aggregations

ReduceOperationFunction (edu.iu.dsc.tws.comms.functions.reduction.ReduceOperationFunction)7 LogicalPlanBuilder (edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder)6 Op (edu.iu.dsc.tws.api.comms.Op)5 MessageTypes (edu.iu.dsc.tws.api.comms.messaging.types.MessageTypes)5 Config (edu.iu.dsc.tws.api.config.Config)5 WorkerEnvironment (edu.iu.dsc.tws.api.resource.WorkerEnvironment)5 BenchmarkUtils (edu.iu.dsc.tws.examples.utils.bench.BenchmarkUtils)5 Timing (edu.iu.dsc.tws.examples.utils.bench.Timing)5 ResultsVerifier (edu.iu.dsc.tws.examples.verification.ResultsVerifier)5 IntArrayComparator (edu.iu.dsc.tws.examples.verification.comparators.IntArrayComparator)5 Set (java.util.Set)5 Logger (java.util.logging.Logger)5 SingularReceiver (edu.iu.dsc.tws.api.comms.SingularReceiver)4 BenchWorker (edu.iu.dsc.tws.examples.comms.BenchWorker)4 GeneratorUtils (edu.iu.dsc.tws.examples.verification.GeneratorUtils)4 BenchmarkConstants (edu.iu.dsc.tws.examples.utils.bench.BenchmarkConstants)3 Comparator (java.util.Comparator)3 Level (java.util.logging.Level)3 BReduce (edu.iu.dsc.tws.comms.batch.BReduce)2 SimpleKeyBasedSelector (edu.iu.dsc.tws.comms.selectors.SimpleKeyBasedSelector)2