Search in sources :

Example 16 with BaseSource

use of edu.iu.dsc.tws.api.compute.nodes.BaseSource in project twister2 by DSC-SPIDAL.

the class BTGatherExample method buildTaskGraph.

@Override
public ComputeGraphBuilder buildTaskGraph() {
    List<Integer> taskStages = jobParameters.getTaskStages();
    int sourceParallelism = taskStages.get(0);
    int sinkParallelism = taskStages.get(1);
    MessageType dataType = MessageTypes.INTEGER_ARRAY;
    String edge = "edge";
    BaseSource g = new SourceTask(edge);
    ICompute r = new GatherSinkTask();
    computeGraphBuilder.addSource(SOURCE, g, sourceParallelism);
    computeConnection = computeGraphBuilder.addCompute(SINK, r, sinkParallelism);
    computeConnection.gather(SOURCE).viaEdge(edge).withDataType(dataType);
    return computeGraphBuilder;
}
Also used : BaseSource(edu.iu.dsc.tws.api.compute.nodes.BaseSource) ICompute(edu.iu.dsc.tws.api.compute.nodes.ICompute) MessageType(edu.iu.dsc.tws.api.comms.messaging.types.MessageType)

Example 17 with BaseSource

use of edu.iu.dsc.tws.api.compute.nodes.BaseSource in project twister2 by DSC-SPIDAL.

the class BTJoinExample method buildTaskGraph.

@Override
public ComputeGraphBuilder buildTaskGraph() {
    List<Integer> taskStages = jobParameters.getTaskStages();
    int sourceParallelism = taskStages.get(0);
    int sinkParallelism = taskStages.get(1);
    MessageType keyType = MessageTypes.INTEGER;
    MessageType dataType = MessageTypes.INTEGER_ARRAY;
    BaseSource source1 = new JoinSource(JoinRelation.LEFT);
    BaseSource source2 = new JoinSource(JoinRelation.RIGHT);
    ICompute r = new JoinSinkTask();
    computeGraphBuilder.addSource(SOURCE, source1, sourceParallelism);
    computeGraphBuilder.addSource(SOURCE2, source2, sourceParallelism);
    computeConnection = computeGraphBuilder.addCompute(SINK, r, sinkParallelism);
    computeConnection.innerJoin(SOURCE, SOURCE2, CommunicationContext.JoinAlgorithm.SORT).viaLeftEdge(LEFT_EDGE).viaRightEdge(RIGHT_EDGE).withKeyType(keyType).withLeftDataType(dataType).withRightDataType(dataType).withTaskPartitioner(new TaskPartitioner() {

        private List<Integer> dst;

        @Override
        public void prepare(Set sources, Set destinations) {
            this.dst = new ArrayList<>(destinations);
            Collections.sort(this.dst);
        }

        @Override
        public int partition(int source, Object data) {
            return dst.get((Integer) data % dst.size());
        }

        @Override
        public void commit(int source, int partition) {
        }
    }).withComparator(Integer::compareTo);
    return computeGraphBuilder;
}
Also used : BaseSource(edu.iu.dsc.tws.api.compute.nodes.BaseSource) Set(java.util.Set) ArrayList(java.util.ArrayList) List(java.util.List) ICompute(edu.iu.dsc.tws.api.compute.nodes.ICompute) TaskPartitioner(edu.iu.dsc.tws.api.compute.TaskPartitioner) MessageType(edu.iu.dsc.tws.api.comms.messaging.types.MessageType)

Example 18 with BaseSource

use of edu.iu.dsc.tws.api.compute.nodes.BaseSource in project twister2 by DSC-SPIDAL.

the class BTKeyedGatherExample method buildTaskGraph.

@Override
public ComputeGraphBuilder buildTaskGraph() {
    List<Integer> taskStages = jobParameters.getTaskStages();
    int sourceParallelism = taskStages.get(0);
    int sinkParallelism = taskStages.get(1);
    MessageType keyType = MessageTypes.INTEGER;
    MessageType dataType = MessageTypes.INTEGER_ARRAY;
    String edge = "edge";
    BaseSource g = new SourceTask(edge, true);
    ICompute r = new KeyedGatherGroupedSinkTask();
    computeGraphBuilder.addSource(SOURCE, g, sourceParallelism);
    computeConnection = computeGraphBuilder.addCompute(SINK, r, sinkParallelism);
    computeConnection.keyedGather(SOURCE).viaEdge(edge).withKeyType(keyType).withTaskPartitioner(new HashingPartitioner()).withDataType(dataType);
    return computeGraphBuilder;
}
Also used : BaseSource(edu.iu.dsc.tws.api.compute.nodes.BaseSource) HashingPartitioner(edu.iu.dsc.tws.tset.fn.HashingPartitioner) ICompute(edu.iu.dsc.tws.api.compute.nodes.ICompute) MessageType(edu.iu.dsc.tws.api.comms.messaging.types.MessageType)

Example 19 with BaseSource

use of edu.iu.dsc.tws.api.compute.nodes.BaseSource in project twister2 by DSC-SPIDAL.

the class BTPartitionExample method buildTaskGraph.

@Override
public ComputeGraphBuilder buildTaskGraph() {
    List<Integer> taskStages = jobParameters.getTaskStages();
    int sourceParallelism = taskStages.get(0);
    int sinkParallelism = taskStages.get(1);
    MessageType dataType = MessageTypes.INTEGER_ARRAY;
    String edge = "edge";
    BaseSource g = new SourceTask(edge);
    ICompute r = new PartitionSinkTask();
    computeGraphBuilder.addSource(SOURCE, g, sourceParallelism);
    computeConnection = computeGraphBuilder.addCompute(SINK, r, sinkParallelism);
    computeConnection.partition(SOURCE).viaEdge(edge).withDataType(dataType);
    return computeGraphBuilder;
}
Also used : BaseSource(edu.iu.dsc.tws.api.compute.nodes.BaseSource) ICompute(edu.iu.dsc.tws.api.compute.nodes.ICompute) MessageType(edu.iu.dsc.tws.api.comms.messaging.types.MessageType)

Example 20 with BaseSource

use of edu.iu.dsc.tws.api.compute.nodes.BaseSource in project twister2 by DSC-SPIDAL.

the class BTBroadCastExample method buildTaskGraph.

@Override
public ComputeGraphBuilder buildTaskGraph() {
    List<Integer> taskStages = jobParameters.getTaskStages();
    int sourceParallelism = taskStages.get(0);
    int sinkParallelism = taskStages.get(1);
    String edge = "edge";
    BaseSource g = new SourceTask(edge);
    ICompute r = new BroadcastSinkTask();
    computeGraphBuilder.addSource(SOURCE, g, sourceParallelism);
    computeConnection = computeGraphBuilder.addCompute(SINK, r, sinkParallelism);
    computeConnection.broadcast(SOURCE).viaEdge(edge);
    return computeGraphBuilder;
}
Also used : BaseSource(edu.iu.dsc.tws.api.compute.nodes.BaseSource) ICompute(edu.iu.dsc.tws.api.compute.nodes.ICompute)

Aggregations

BaseSource (edu.iu.dsc.tws.api.compute.nodes.BaseSource)20 ICompute (edu.iu.dsc.tws.api.compute.nodes.ICompute)19 MessageType (edu.iu.dsc.tws.api.comms.messaging.types.MessageType)13 Op (edu.iu.dsc.tws.api.comms.Op)2 TaskPartitioner (edu.iu.dsc.tws.api.compute.TaskPartitioner)2 JobConfig (edu.iu.dsc.tws.api.JobConfig)1 ExecutionPlan (edu.iu.dsc.tws.api.compute.executor.ExecutionPlan)1 ComputeGraph (edu.iu.dsc.tws.api.compute.graph.ComputeGraph)1 Config (edu.iu.dsc.tws.api.config.Config)1 BenchmarkResultsRecorder (edu.iu.dsc.tws.examples.utils.bench.BenchmarkResultsRecorder)1 DeterministicTaskPartitioner (edu.iu.dsc.tws.examples.utils.partitioners.DeterministicTaskPartitioner)1 ComputeEnvironment (edu.iu.dsc.tws.task.ComputeEnvironment)1 ComputeGraphBuilder (edu.iu.dsc.tws.task.impl.ComputeGraphBuilder)1 KeyedGatherConfig (edu.iu.dsc.tws.task.impl.ops.KeyedGatherConfig)1 HashingPartitioner (edu.iu.dsc.tws.tset.fn.HashingPartitioner)1 ArrayList (java.util.ArrayList)1 List (java.util.List)1 Set (java.util.Set)1