Search in sources :

Example 41 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class BasicCxMultiTaskGraphJob method main.

public static void main(String[] args) {
    // first load the configurations from command line and config files
    Config config = ResourceAllocator.loadConfig(new HashMap<>());
    // build JobConfig
    JobConfig jobConfig = new JobConfig();
    // Task 0 provide input to Task 1 & Task 2 independent Parallel Tasks.
    // build the job
    BasicJob basicJob = BasicJob.newBuilder().setName("basic-cxmultitaskgraph").setContainerClass(// will be updated soon
    SimpleCxMultiTaskGraph.class.getName()).setRequestResource(new ResourceContainer(2, 1024), 3).setConfig(jobConfig).build();
    // now submit the job
    Twister2Submitter.submitContainerJob(basicJob, config);
}
Also used : BasicJob(edu.iu.dsc.tws.api.basic.job.BasicJob) Config(edu.iu.dsc.tws.common.config.Config) JobConfig(edu.iu.dsc.tws.api.JobConfig) JobConfig(edu.iu.dsc.tws.api.JobConfig) ResourceContainer(edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer)

Example 42 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class BasicTGraphJobExample method main.

public static void main(String[] args) {
    // first load the configurations from command line and config files
    Config config = ResourceAllocator.loadConfig(new HashMap<>());
    // build JobConfig
    JobConfig jobConfig = new JobConfig();
    // build the job
    BasicJob basicJob = BasicJob.newBuilder().setName("basic-tgraphExampleJob").setContainerClass(SimpleTGraphExample.class.getName()).setRequestResource(new ResourceContainer(4, 1024), 4).setConfig(jobConfig).build();
    // now submit the job
    Twister2Submitter.submitContainerJob(basicJob, config);
}
Also used : BasicJob(edu.iu.dsc.tws.api.basic.job.BasicJob) Config(edu.iu.dsc.tws.common.config.Config) JobConfig(edu.iu.dsc.tws.api.JobConfig) JobConfig(edu.iu.dsc.tws.api.JobConfig) ResourceContainer(edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer)

Example 43 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class BasicTaskGraphJob method main.

public static void main(String[] args) {
    // first load the configurations from command line and config files
    Config config = ResourceAllocator.loadConfig(new HashMap<>());
    // build JobConfig
    JobConfig jobConfig = new JobConfig();
    // build the job
    BasicJob basicJob = BasicJob.newBuilder().setName("basic-taskgraph").setContainerClass(SimpleTaskGraph.class.getName()).setRequestResource(new ResourceContainer(2, 1024), 2).setConfig(jobConfig).build();
    // now submit the job
    Twister2Submitter.submitContainerJob(basicJob, config);
}
Also used : BasicJob(edu.iu.dsc.tws.api.basic.job.BasicJob) Config(edu.iu.dsc.tws.common.config.Config) JobConfig(edu.iu.dsc.tws.api.JobConfig) JobConfig(edu.iu.dsc.tws.api.JobConfig) ResourceContainer(edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer)

Example 44 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class BasicTaskQueueJob method main.

public static void main(String[] args) {
    // first load the configurations from command line and config files
    Config config = ResourceAllocator.loadConfig(new HashMap<>());
    // build JobConfig
    JobConfig jobConfig = new JobConfig();
    // build the job
    BasicJob basicJob = BasicJob.newBuilder().setName("basic-taskqueue").setContainerClass(SimpleTaskQueue.class.getName()).setRequestResource(new ResourceContainer(2, 1024), 2).setConfig(jobConfig).build();
    // now submit the job
    Twister2Submitter.submitContainerJob(basicJob, config);
}
Also used : BasicJob(edu.iu.dsc.tws.api.basic.job.BasicJob) Config(edu.iu.dsc.tws.common.config.Config) JobConfig(edu.iu.dsc.tws.api.JobConfig) JobConfig(edu.iu.dsc.tws.api.JobConfig) ResourceContainer(edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer)

Example 45 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class KubernetesWorker method loadConfig.

/**
 * loadConfig from config files
 * @return
 */
public static Config loadConfig(String configDir) {
    // first lets read the essential properties from java system properties
    String twister2Home = Paths.get("").toAbsolutePath().toString();
    LOG.log(Level.INFO, String.format("Loading configuration with twister2_home: %s and " + "configuration: %s", twister2Home, configDir));
    Config conf1 = ConfigLoader.loadConfig(twister2Home, configDir);
    Config conf2 = Config.newBuilder().putAll(conf1).put(Context.TWISTER2_HOME.getKey(), twister2Home).put(Context.TWISTER2_CONF.getKey(), configDir).put(Context.TWISTER2_CLUSTER_TYPE, KUBERNETES_CLUSTER_TYPE).build();
    LOG.log(Level.INFO, "Config files are read from directory: " + configDir);
    return conf2;
}
Also used : Config(edu.iu.dsc.tws.common.config.Config)

Aggregations

Config (edu.iu.dsc.tws.common.config.Config)59 JobConfig (edu.iu.dsc.tws.api.JobConfig)31 BasicJob (edu.iu.dsc.tws.api.basic.job.BasicJob)30 ResourceContainer (edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer)30 InputFormat (edu.iu.dsc.tws.data.api.InputFormat)3 Path (edu.iu.dsc.tws.data.fs.Path)3 InputSplit (edu.iu.dsc.tws.data.fs.io.InputSplit)3 InputSplitAssigner (edu.iu.dsc.tws.data.fs.io.InputSplitAssigner)3 TextInputFormatter (edu.iu.dsc.tws.data.api.formatters.TextInputFormatter)2 URI (java.net.URI)2 CommandLine (org.apache.commons.cli.CommandLine)2 CommandLineParser (org.apache.commons.cli.CommandLineParser)2 DefaultParser (org.apache.commons.cli.DefaultParser)2 HelpFormatter (org.apache.commons.cli.HelpFormatter)2 Options (org.apache.commons.cli.Options)2 ParseException (org.apache.commons.cli.ParseException)2 BinaryInputFormatter (edu.iu.dsc.tws.data.api.formatters.BinaryInputFormatter)1 FileInputSplit (edu.iu.dsc.tws.data.fs.FileInputSplit)1 JobAPI (edu.iu.dsc.tws.proto.system.job.JobAPI)1 IController (edu.iu.dsc.tws.rsched.spi.scheduler.IController)1