Search in sources :

Example 56 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class BaseReduceHLCommunication method main.

public static void main(String[] args) {
    // first load the configurations from command line and config files
    Config config = ResourceAllocator.loadConfig(new HashMap<>());
    // build JobConfig
    JobConfig jobConfig = new JobConfig();
    // build the job
    BasicJob basicJob = BasicJob.newBuilder().setName("basic-hl-reduce").setContainerClass(BaseReduceHLCommunication.class.getName()).setRequestResource(new ResourceContainer(2, 1024), 4).setConfig(jobConfig).build();
    // now submit the job
    Twister2Submitter.submitContainerJob(basicJob, config);
}
Also used : BasicJob(edu.iu.dsc.tws.api.basic.job.BasicJob) JobConfig(edu.iu.dsc.tws.api.JobConfig) Config(edu.iu.dsc.tws.common.config.Config) JobConfig(edu.iu.dsc.tws.api.JobConfig) ResourceContainer(edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer)

Example 57 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class MPIProcess method main.

public static void main(String[] args) {
    Options cmdOptions = null;
    try {
        MPI.Init(args);
        int rank = MPI.COMM_WORLD.getRank();
        int size = MPI.COMM_WORLD.getSize();
        cmdOptions = setupOptions();
        CommandLineParser parser = new DefaultParser();
        // parse the help options first.
        CommandLine cmd = parser.parse(cmdOptions, args);
        // load the configuration
        // we are loading the configuration for all the components
        Config config = loadConfigurations(cmd, rank);
        // normal worker
        LOG.log(Level.FINE, "A worker process is starting...");
        worker(config, rank);
    } catch (MPIException e) {
        LOG.log(Level.SEVERE, "Failed the MPI process", e);
        throw new RuntimeException(e);
    } catch (ParseException e) {
        HelpFormatter formatter = new HelpFormatter();
        formatter.printHelp("SubmitterMain", cmdOptions);
        throw new RuntimeException("Error parsing command line options: ", e);
    } finally {
        try {
            MPI.Finalize();
        } catch (MPIException ignore) {
        }
    }
}
Also used : MPIException(mpi.MPIException) HelpFormatter(org.apache.commons.cli.HelpFormatter) Options(org.apache.commons.cli.Options) CommandLine(org.apache.commons.cli.CommandLine) Config(edu.iu.dsc.tws.common.config.Config) CommandLineParser(org.apache.commons.cli.CommandLineParser) ParseException(org.apache.commons.cli.ParseException) DefaultParser(org.apache.commons.cli.DefaultParser)

Example 58 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class BasicTaskSchedulingJob method main.

public static void main(String[] args) {
    // first load the configurations from command line and config files
    Config config = ResourceAllocator.loadConfig(new HashMap<>());
    // build JobConfig
    JobConfig jobConfig = new JobConfig();
    // build the job
    BasicJob basicJob = BasicJob.newBuilder().setName("basic-testing").setContainerClass(TaskScheduling.class.getName()).setRequestResource(new ResourceContainer(2, 1024), 2).setConfig(jobConfig).build();
    // now submit the job
    Twister2Submitter.submitContainerJob(basicJob, config);
}
Also used : BasicJob(edu.iu.dsc.tws.api.basic.job.BasicJob) Config(edu.iu.dsc.tws.common.config.Config) JobConfig(edu.iu.dsc.tws.api.JobConfig) JobConfig(edu.iu.dsc.tws.api.JobConfig) ResourceContainer(edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer)

Example 59 with Config

use of edu.iu.dsc.tws.common.config.Config in project twister2 by DSC-SPIDAL.

the class ScpTest method main.

public static void main(String[] args) {
    Config config = Config.newBuilder().put(ScpContext.UPLOAD_DIRECTORY, "/vagrant").put(ScpContext.TWISTER2_UPLOADER_SCP_OPTIONS, "").put(ScpContext.TWISTER2_UPLOADER_SCP_CONNECTION, "root@149.165.150.81").put(ScpContext.TWISTER2_UPLOADER_SSH_OPTIONS, "-i /Users/user1/.ssh/id_rsa").put(ScpContext.TWISTER2_UPLOADER_SSH_CONNECTION, "root@149.165.150.81").build();
    ScpUploader uploader = new ScpUploader();
    uploader.initialize(config);
    // upload
    URI destURI = uploader.uploadPackage("/Users/user1/Desktop/tobecopied");
    System.out.println("File path in remote machine is " + destURI);
// delete
// uploader.undo();
// System.out.println("File deleted in remote machine");
}
Also used : Config(edu.iu.dsc.tws.common.config.Config) URI(java.net.URI)

Aggregations

Config (edu.iu.dsc.tws.common.config.Config)59 JobConfig (edu.iu.dsc.tws.api.JobConfig)31 BasicJob (edu.iu.dsc.tws.api.basic.job.BasicJob)30 ResourceContainer (edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer)30 InputFormat (edu.iu.dsc.tws.data.api.InputFormat)3 Path (edu.iu.dsc.tws.data.fs.Path)3 InputSplit (edu.iu.dsc.tws.data.fs.io.InputSplit)3 InputSplitAssigner (edu.iu.dsc.tws.data.fs.io.InputSplitAssigner)3 TextInputFormatter (edu.iu.dsc.tws.data.api.formatters.TextInputFormatter)2 URI (java.net.URI)2 CommandLine (org.apache.commons.cli.CommandLine)2 CommandLineParser (org.apache.commons.cli.CommandLineParser)2 DefaultParser (org.apache.commons.cli.DefaultParser)2 HelpFormatter (org.apache.commons.cli.HelpFormatter)2 Options (org.apache.commons.cli.Options)2 ParseException (org.apache.commons.cli.ParseException)2 BinaryInputFormatter (edu.iu.dsc.tws.data.api.formatters.BinaryInputFormatter)1 FileInputSplit (edu.iu.dsc.tws.data.fs.FileInputSplit)1 JobAPI (edu.iu.dsc.tws.proto.system.job.JobAPI)1 IController (edu.iu.dsc.tws.rsched.spi.scheduler.IController)1