Examples with ResourceContainer edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer used on opensource projects

Example 31 with ResourceContainer

use of edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer in project twister2 by DSC-SPIDAL.

the class ResourceAllocator method buildRequestedResources.

private RequestedResources buildRequestedResources(JobAPI.Job job) {
    JobAPI.JobResources jobResources = job.getJobResources();
    int noOfContainers = jobResources.getNoOfContainers();
    ResourceContainer container = new ResourceContainer((int) jobResources.getContainer().getAvailableCPU(), (int) jobResources.getContainer().getAvailableMemory(), (int) jobResources.getContainer().getAvailableDisk());
    return new RequestedResources(noOfContainers, container);
}

Example 32 with ResourceContainer

use of edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer in project twister2 by DSC-SPIDAL.

the class AuroraLauncher method launch.

/**
 * Launch the processes according to the resource plan.
 *
 * @param resourceRequest requested resources
 * @return true if the request is granted
 */
@Override
public boolean launch(RequestedResources resourceRequest, JobAPI.Job job) {
    String jobName = job.getJobName();
    // first check whether there is an active job running with same name on ZooKeeper
    if (ZKUtil.isThereAnActiveJob(jobName, config)) {
        throw new RuntimeException("There is an active job in ZooKeeper with same name." + "\nFirst try to kill that job. Run terminate job command." + "\nThis job is not submitted to Aurora Server");
    }
    // construct the controller to submit the job to Aurora Scheduler
    String cluster = AuroraContext.auroraClusterName(config);
    String role = AuroraContext.role(config);
    String env = AuroraContext.environment(config);
    AuroraClientController controller = new AuroraClientController(cluster, role, env, jobName, true);
    // get aurora file name to execute when submitting the job
    String auroraFilename = AuroraContext.auroraScript(config);
    // get environment variables from config
    Map<AuroraField, String> bindings = constructEnvVariables(config);
    // convert RequestedResources to environment variables, override previous values from config
    ResourceContainer container = resourceRequest.getContainer();
    bindings.put(AuroraField.JOB_NAME, jobName);
    bindings.put(AuroraField.AURORA_WORKER_CLASS, AuroraContext.auroraWorkerClass(config));
    bindings.put(AuroraField.CPUS_PER_CONTAINER, container.getNoOfCpus() + "");
    bindings.put(AuroraField.RAM_PER_CONTAINER, container.getMemoryInBytes() + "");
    bindings.put(AuroraField.DISK_PER_CONTAINER, container.getDiskInBytes() + "");
    bindings.put(AuroraField.NUMBER_OF_CONTAINERS, resourceRequest.getNoOfContainers() + "");
    printEnvs(bindings);
    return controller.createJob(bindings, auroraFilename);
}

Example 33 with ResourceContainer

use of edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer in project twister2 by DSC-SPIDAL.

the class WordCountJob method main.

public static void main(String[] args) {
    // first load the configurations from command line and config files
    Config config = ResourceAllocator.loadConfig(new HashMap<>());
    // build JobConfig
    JobConfig jobConfig = new JobConfig();
    BasicJob.BasicJobBuilder jobBuilder = BasicJob.newBuilder();
    jobBuilder.setName("batch-wordcount");
    jobBuilder.setContainerClass(WordCountContainer.class.getName());
    jobBuilder.setRequestResource(new ResourceContainer(2, 1024), 4);
    jobBuilder.setConfig(jobConfig);
    // now submit the job
    Twister2Submitter.submitContainerJob(jobBuilder.build(), config);
}

Example 34 with ResourceContainer

use of edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer in project twister2 by DSC-SPIDAL.

the class BaseAllReduceCommunication method main.

public static void main(String[] args) {
    // first load the configurations from command line and config files
    Config config = ResourceAllocator.loadConfig(new HashMap<>());
    // build JobConfig
    JobConfig jobConfig = new JobConfig();
    // build the job
    BasicJob basicJob = BasicJob.newBuilder().setName("basic-all-reduce").setContainerClass(BaseAllReduceCommunication.class.getName()).setRequestResource(new ResourceContainer(2, 1024), 4).setConfig(jobConfig).build();
    // now submit the job
    Twister2Submitter.submitContainerJob(basicJob, config);
}

Example 35 with ResourceContainer

use of edu.iu.dsc.tws.rsched.spi.resource.ResourceContainer in project twister2 by DSC-SPIDAL.

the class BaseBroadcastCommunication method main.

public static void main(String[] args) {
    // first load the configurations from command line and config files
    Config config = ResourceAllocator.loadConfig(new HashMap<>());
    // build JobConfig
    JobConfig jobConfig = new JobConfig();
    // build the job
    BasicJob basicJob = BasicJob.newBuilder().setName("basic-broadcast").setContainerClass(BaseBroadcastCommunication.class.getName()).setRequestResource(new ResourceContainer(2, 1024), 4).setConfig(jobConfig).build();
    // now submit the job
    Twister2Submitter.submitContainerJob(basicJob, config);
}