Search in sources :

Example 1 with ComputeConnection

use of edu.iu.dsc.tws.task.impl.ComputeConnection in project twister2 by DSC-SPIDAL.

the class KMeansConnectedDataflowExample method generateThirdJob.

private static DataFlowGraph generateThirdJob(Config config, int parallelismValue, int instances, int iterations, int dimension, DataFlowJobConfig jobConfig) {
    KMeansSourceTask kMeansSourceTask = new KMeansSourceTask(dimension);
    KMeansAllReduceTask kMeansAllReduceTask = new KMeansAllReduceTask();
    ComputeGraphBuilder kmeansComputeGraphBuilder = ComputeGraphBuilder.newBuilder(config);
    // Add source, and sink tasks to the task graph builder for the third task graph
    kmeansComputeGraphBuilder.addSource("kmeanssource", kMeansSourceTask, parallelismValue);
    ComputeConnection kMeanscomputeConnection = kmeansComputeGraphBuilder.addCompute("kmeanssink", kMeansAllReduceTask, parallelismValue);
    // Creating the communication edges between the tasks for the third task graph
    kMeanscomputeConnection.allreduce("kmeanssource").viaEdge("all-reduce").withReductionFunction(new CentroidAggregator()).withDataType(MessageTypes.OBJECT);
    kmeansComputeGraphBuilder.setMode(OperationMode.BATCH);
    kmeansComputeGraphBuilder.setTaskGraphName("kmeansTG");
    ComputeGraph thirdGraph = kmeansComputeGraphBuilder.build();
    DataFlowGraph job = DataFlowGraph.newSubGraphJob("kmeansTG", thirdGraph).setWorkers(instances).addDataFlowJobConfig(jobConfig).setGraphType("iterative").setIterations(iterations);
    return job;
}
Also used : ComputeGraph(edu.iu.dsc.tws.api.compute.graph.ComputeGraph) ComputeGraphBuilder(edu.iu.dsc.tws.task.impl.ComputeGraphBuilder) ComputeConnection(edu.iu.dsc.tws.task.impl.ComputeConnection) DataFlowGraph(edu.iu.dsc.tws.task.cdfw.DataFlowGraph)

Example 2 with ComputeConnection

use of edu.iu.dsc.tws.task.impl.ComputeConnection in project twister2 by DSC-SPIDAL.

the class KMeansConnectedDataflowExample method generateFirstJob.

private static DataFlowGraph generateFirstJob(Config config, int parallelismValue, String dataDirectory, int dimension, int dsize, int instances, DataFlowJobConfig jobConfig) {
    DataObjectSource dataObjectSource = new DataObjectSource(Context.TWISTER2_DIRECT_EDGE, dataDirectory);
    KMeansDataObjectCompute dataObjectCompute = new KMeansDataObjectCompute(Context.TWISTER2_DIRECT_EDGE, dsize, parallelismValue, dimension);
    KMeansDataObjectDirectSink dataObjectSink = new KMeansDataObjectDirectSink("points");
    ComputeGraphBuilder datapointsComputeGraphBuilder = ComputeGraphBuilder.newBuilder(config);
    // Add source, compute, and sink tasks to the task graph builder for the first task graph
    datapointsComputeGraphBuilder.addSource("datapointsource", dataObjectSource, parallelismValue);
    ComputeConnection datapointComputeConnection = datapointsComputeGraphBuilder.addCompute("datapointcompute", dataObjectCompute, parallelismValue);
    ComputeConnection firstGraphComputeConnection = datapointsComputeGraphBuilder.addCompute("datapointsink", dataObjectSink, parallelismValue);
    // Creating the communication edges between the tasks for the second task graph
    datapointComputeConnection.direct("datapointsource").viaEdge(Context.TWISTER2_DIRECT_EDGE).withDataType(MessageTypes.OBJECT);
    firstGraphComputeConnection.direct("datapointcompute").viaEdge(Context.TWISTER2_DIRECT_EDGE).withDataType(MessageTypes.OBJECT);
    datapointsComputeGraphBuilder.setMode(OperationMode.BATCH);
    datapointsComputeGraphBuilder.setTaskGraphName("datapointsTG");
    ComputeGraph firstGraph = datapointsComputeGraphBuilder.build();
    DataFlowGraph job = DataFlowGraph.newSubGraphJob("datapointsTG", firstGraph).setWorkers(instances).addDataFlowJobConfig(jobConfig).setGraphType("non-iterative");
    return job;
}
Also used : ComputeGraph(edu.iu.dsc.tws.api.compute.graph.ComputeGraph) ComputeGraphBuilder(edu.iu.dsc.tws.task.impl.ComputeGraphBuilder) DataObjectSource(edu.iu.dsc.tws.task.dataobjects.DataObjectSource) ComputeConnection(edu.iu.dsc.tws.task.impl.ComputeConnection) DataFlowGraph(edu.iu.dsc.tws.task.cdfw.DataFlowGraph)

Example 3 with ComputeConnection

use of edu.iu.dsc.tws.task.impl.ComputeConnection in project twister2 by DSC-SPIDAL.

the class TaskGraphBuildTest method createGraphWithEdgeName.

private ComputeGraph createGraphWithEdgeName(String edgeName) {
    TestSource testSource = new TestSource();
    TestSink1 testCompute = new TestSink1();
    TestSink2 testSink = new TestSink2();
    ComputeGraphBuilder computeGraphBuilder = ComputeGraphBuilder.newBuilder(getConfig());
    computeGraphBuilder.addSource("source", testSource, 4);
    ComputeConnection computeConnection = computeGraphBuilder.addCompute("compute", testCompute, 4);
    computeConnection.partition("source").viaEdge(edgeName).withDataType(MessageTypes.OBJECT);
    ComputeConnection rc = computeGraphBuilder.addCompute("sink", testSink, 1);
    rc.allreduce("compute").viaEdge(edgeName).withReductionFunction(new Aggregator()).withDataType(MessageTypes.OBJECT);
    ComputeGraph graph = computeGraphBuilder.build();
    return graph;
}
Also used : ComputeGraph(edu.iu.dsc.tws.api.compute.graph.ComputeGraph) ComputeGraphBuilder(edu.iu.dsc.tws.task.impl.ComputeGraphBuilder) ComputeConnection(edu.iu.dsc.tws.task.impl.ComputeConnection)

Example 4 with ComputeConnection

use of edu.iu.dsc.tws.task.impl.ComputeConnection in project twister2 by DSC-SPIDAL.

the class SvmSgdIterativeRunner method buildSvmSgdTestingTG.

private ComputeGraph buildSvmSgdTestingTG() {
    iterativePredictionDataStreamer = new IterativePredictionDataStreamer(this.svmJobParameters.getFeatures(), this.operationMode, this.svmJobParameters.isDummy(), this.binaryBatchModel);
    iterativeSVMAccuracyReduce = new IterativeSVMAccuracyReduce(this.operationMode);
    testingBuilder.addSource(Constants.SimpleGraphConfig.PREDICTION_SOURCE_TASK, iterativePredictionDataStreamer, dataStreamerParallelism);
    ComputeConnection svmComputeConnection = testingBuilder.addCompute(Constants.SimpleGraphConfig.PREDICTION_REDUCE_TASK, iterativeSVMAccuracyReduce, dataStreamerParallelism);
    svmComputeConnection.allreduce(Constants.SimpleGraphConfig.PREDICTION_SOURCE_TASK).viaEdge(Constants.SimpleGraphConfig.PREDICTION_EDGE).withReductionFunction(new IterativeAccuracyReduceFunction()).withDataType(MessageTypes.DOUBLE);
    testingBuilder.setMode(operationMode);
    testingBuilder.setTaskGraphName(IterativeSVMConstants.ITERATIVE_PREDICTION_TASK_GRAPH);
    return testingBuilder.build();
}
Also used : IterativeSVMAccuracyReduce(edu.iu.dsc.tws.examples.ml.svm.aggregate.IterativeSVMAccuracyReduce) IterativePredictionDataStreamer(edu.iu.dsc.tws.examples.ml.svm.streamer.IterativePredictionDataStreamer) ComputeConnection(edu.iu.dsc.tws.task.impl.ComputeConnection) IterativeAccuracyReduceFunction(edu.iu.dsc.tws.examples.ml.svm.aggregate.IterativeAccuracyReduceFunction)

Example 5 with ComputeConnection

use of edu.iu.dsc.tws.task.impl.ComputeConnection in project twister2 by DSC-SPIDAL.

the class SourceTaskDataLoader method execute.

@Override
public void execute() {
    getParams();
    /*
     * First data is loaded from files
     * */
    ComputeGraphBuilder computeGraphBuilder = ComputeGraphBuilder.newBuilder(config);
    // DataObjectSource sourceTask = new DataObjectSource(Context.TWISTER2_DIRECT_EDGE,
    // dataSource);
    // DataObjectSink sinkTask = new DataObjectSink();
    // computeGraphBuilder.addSource("datapointsource", sourceTask, parallelism);
    // ComputeConnection firstGraphComputeConnection = computeGraphBuilder.addSink(
    // "datapointsink", sinkTask, parallelism);
    // firstGraphComputeConnection.direct("datapointsource",
    // Context.TWISTER2_DIRECT_EDGE, DataType.OBJECT);
    // computeGraphBuilder.setMode(OperationMode.BATCH);
    // 
    // ComputeGraph datapointsTaskGraph = computeGraphBuilder.build();
    // ExecutionPlan firstGraphExecutionPlan = taskExecutor.plan(datapointsTaskGraph);
    // taskExecutor.execute(datapointsTaskGraph, firstGraphExecutionPlan);
    // DataObject<Object> dataPointsObject = taskExecutor.getOutput(
    // datapointsTaskGraph, firstGraphExecutionPlan, "datapointsink");
    // LOG.info("Total Partitions : " + dataPointsObject.getPartitions().length);
    /*
     * Second Task
     * */
    DataSourceTask kMeansSourceTask = new DataSourceTask();
    SimpleDataAllReduceTask kMeansAllReduceTask = new SimpleDataAllReduceTask();
    computeGraphBuilder.addSource("kmeanssource", kMeansSourceTask, parallelism);
    ComputeConnection computeConnection = computeGraphBuilder.addCompute("kmeanssink", kMeansAllReduceTask, parallelism);
    computeConnection.allreduce("kmeanssource").viaEdge("all-reduce").withReductionFunction(new SimpleDataAggregator()).withDataType(MessageTypes.OBJECT);
    computeGraphBuilder.setMode(OperationMode.BATCH);
    ComputeGraph simpleTaskGraph = computeGraphBuilder.build();
    ExecutionPlan plan = taskExecutor.plan(simpleTaskGraph);
    // taskExecutor.addInput(
    // simpleTaskGraph, plan, "kmeanssource", "points", dataPointsObject);
    taskExecutor.execute(simpleTaskGraph, plan);
    DataObject<double[][]> dataSet = taskExecutor.getOutput(simpleTaskGraph, plan, "kmeanssink");
// DataObject<Object> dataSet = taskExecutor.getOutput(simpleTaskGraph, plan, "kmeanssink");
// DataPartition<Object> values = dataSet.getPartitions()[0];
// Object lastObject = values.getConsumer().next();
// LOG.info(String.format("Last Object : %s", lastObject.getClass().getGraphName()));
}
Also used : ExecutionPlan(edu.iu.dsc.tws.api.compute.executor.ExecutionPlan) ComputeGraph(edu.iu.dsc.tws.api.compute.graph.ComputeGraph) ComputeGraphBuilder(edu.iu.dsc.tws.task.impl.ComputeGraphBuilder) ComputeConnection(edu.iu.dsc.tws.task.impl.ComputeConnection)

Aggregations

ComputeConnection (edu.iu.dsc.tws.task.impl.ComputeConnection)65 ComputeGraphBuilder (edu.iu.dsc.tws.task.impl.ComputeGraphBuilder)55 ComputeGraph (edu.iu.dsc.tws.api.compute.graph.ComputeGraph)40 TaskSchedulerClassTest (edu.iu.dsc.tws.tsched.utils.TaskSchedulerClassTest)16 ExecutionPlan (edu.iu.dsc.tws.api.compute.executor.ExecutionPlan)13 DataFlowGraph (edu.iu.dsc.tws.task.cdfw.DataFlowGraph)8 DataObject (edu.iu.dsc.tws.api.dataset.DataObject)6 GraphDataSource (edu.iu.dsc.tws.graphapi.partition.GraphDataSource)6 DataObjectSource (edu.iu.dsc.tws.task.dataobjects.DataObjectSource)6 DataObjectSink (edu.iu.dsc.tws.task.dataobjects.DataObjectSink)5 ReduceAggregator (edu.iu.dsc.tws.examples.ml.svm.aggregate.ReduceAggregator)4 ConnectedSink (edu.iu.dsc.tws.task.cdfw.task.ConnectedSink)4 SVMReduce (edu.iu.dsc.tws.examples.ml.svm.aggregate.SVMReduce)3 DataFileReplicatedReadSource (edu.iu.dsc.tws.task.dataobjects.DataFileReplicatedReadSource)3 IExecutor (edu.iu.dsc.tws.api.compute.executor.IExecutor)2 Config (edu.iu.dsc.tws.api.config.Config)2 TextInputSplit (edu.iu.dsc.tws.data.api.splits.TextInputSplit)2 IterativeAccuracyReduceFunction (edu.iu.dsc.tws.examples.ml.svm.aggregate.IterativeAccuracyReduceFunction)2 IterativeSVMCompute (edu.iu.dsc.tws.examples.ml.svm.compute.IterativeSVMCompute)2 SVMCompute (edu.iu.dsc.tws.examples.ml.svm.compute.SVMCompute)2