Search in sources :

Example 31 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class MPIDataFlowCommunication method partition.

@Override
public DataFlowOperation partition(Map<String, Object> properties, MessageType type, int edge1, Set<Integer> sourceTasks, Set<Integer> destTasks, MessageReceiver receiver, CompletionListener cmpListener) {
    // merge with the user specified configuration, user specified will take precedence
    Config mergedCfg = Config.newBuilder().putAll(config).putAll(properties).build();
    MPIDataFlowPartition dataFlowOperation = new MPIDataFlowPartition(channel, sourceTasks, destTasks, receiver, MPIDataFlowPartition.PartitionStratergy.DIRECT, cmpListener);
    dataFlowOperation.init(mergedCfg, type, instancePlan, edge1);
    return dataFlowOperation;
}
Also used : Config(edu.iu.dsc.tws.common.config.Config)

Example 32 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class MPIDataFlowCommunication method broadCast.

public DataFlowOperation broadCast(Map<String, Object> properties, MessageType type, int edge, int sourceTask, Set<Integer> destTasks, MessageReceiver receiver) {
    // merge with the user specified configuration, user specified will take precedence
    Config mergedCfg = Config.newBuilder().putAll(config).putAll(properties).build();
    // create the dataflow operation
    MPIDataFlowBroadcast dataFlowOperation = new MPIDataFlowBroadcast(channel, sourceTask, destTasks, receiver);
    // intialize the operation
    dataFlowOperation.init(mergedCfg, type, instancePlan, edge);
    return dataFlowOperation;
}
Also used : Config(edu.iu.dsc.tws.common.config.Config)

Example 33 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class MPIDataFlowCommunication method keyedReduce.

public DataFlowOperation keyedReduce(Map<String, Object> properties, MessageType type, Set<Integer> edge, Set<Integer> sourceTasks, Set<Integer> destTasks, MultiMessageReceiver receiver, MultiMessageReceiver partial) {
    // merge with the user specified configuration, user specified will take precedence
    Config mergedCfg = Config.newBuilder().putAll(config).putAll(properties).build();
    // create the dataflow operation
    MPIDataFlowMultiReduce dataFlowOperation = new MPIDataFlowMultiReduce(channel, sourceTasks, destTasks, receiver, partial, edge);
    // intialize the operation
    dataFlowOperation.init(mergedCfg, type, instancePlan, 0);
    return dataFlowOperation;
}
Also used : Config(edu.iu.dsc.tws.common.config.Config)

Example 34 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class MPIDataFlowCommunication method gather.

@Override
public DataFlowOperation gather(Map<String, Object> properties, MessageType type, MessageType keyType, int edge1, Set<Integer> sourceTasks, int destTask, MessageReceiver receiver) {
    // merge with the user specified configuration, user specified will take precedence
    Config mergedCfg = Config.newBuilder().putAll(config).putAll(properties).build();
    // create the dataflow operation
    MPIDataFlowGather dataFlowOperation = new MPIDataFlowGather(channel, sourceTasks, destTask, receiver, 0, 0, mergedCfg, type, keyType, instancePlan, edge1);
    // intialize the operation
    dataFlowOperation.init(mergedCfg, type, instancePlan, edge1);
    return dataFlowOperation;
}
Also used : Config(edu.iu.dsc.tws.common.config.Config)

Example 35 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class MPIDataFlowCommunication method loadBalance.

public DataFlowOperation loadBalance(Map<String, Object> properties, MessageType type, int edge, Set<Integer> sourceTasks, Set<Integer> destTasks, MessageReceiver receiver) {
    // merge with the user specified configuration, user specified will take precedence
    Config mergedCfg = Config.newBuilder().putAll(config).putAll(properties).build();
    // create the dataflow operation
    MPIDataFlowLoadBalance dataFlowOperation = new MPIDataFlowLoadBalance(channel, sourceTasks, destTasks, receiver);
    // intialize the operation
    dataFlowOperation.init(mergedCfg, type, instancePlan, edge);
    return dataFlowOperation;
}
Also used : Config(edu.iu.dsc.tws.common.config.Config)

Aggregations

Config (edu.iu.dsc.tws.common.config.Config)59 JobConfig (edu.iu.dsc.tws.api.JobConfig)31 BasicJob (edu.iu.dsc.tws.api.basic.job.BasicJob)30 ResourceContainer (edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer)30 InputFormat (edu.iu.dsc.tws.data.api.InputFormat)3 Path (edu.iu.dsc.tws.data.fs.Path)3 InputSplit (edu.iu.dsc.tws.data.fs.io.InputSplit)3 InputSplitAssigner (edu.iu.dsc.tws.data.fs.io.InputSplitAssigner)3 TextInputFormatter (edu.iu.dsc.tws.data.api.formatters.TextInputFormatter)2 URI (java.net.URI)2 CommandLine (org.apache.commons.cli.CommandLine)2 CommandLineParser (org.apache.commons.cli.CommandLineParser)2 DefaultParser (org.apache.commons.cli.DefaultParser)2 HelpFormatter (org.apache.commons.cli.HelpFormatter)2 Options (org.apache.commons.cli.Options)2 ParseException (org.apache.commons.cli.ParseException)2 BinaryInputFormatter (edu.iu.dsc.tws.data.api.formatters.BinaryInputFormatter)1 FileInputSplit (edu.iu.dsc.tws.data.fs.FileInputSplit)1 JobAPI (edu.iu.dsc.tws.proto.system.job.JobAPI)1 IController (edu.iu.dsc.tws.rsched.spi.scheduler.IController)1