Examples with DataNodeLocatorUtils edu.iu.dsc.tws.data.utils.DataNodeLocatorUtils used on opensource projects

Example 1 with DataNodeLocatorUtils

use of edu.iu.dsc.tws.data.utils.DataNodeLocatorUtils in project twister2 by DSC-SPIDAL.

the class DataLocalityBatchTaskScheduler method initialize.

/**
 * This method first initialize the task instance values with default task instance ram, disk, and
 * cpu values from the task scheduler context.
 */
@Override
public void initialize(Config cfg) {
    this.config = cfg;
    this.instanceRAM = TaskSchedulerContext.taskInstanceRam(this.config);
    this.instanceDisk = TaskSchedulerContext.taskInstanceDisk(this.config);
    this.instanceCPU = TaskSchedulerContext.taskInstanceCpu(this.config);
    this.dataNodeLocatorUtils = new DataNodeLocatorUtils(config);
    this.dataLocalityAwareAllocation = new HashMap<>();
    this.taskAttributes = new TaskAttributes();
}

Example 2 with DataNodeLocatorUtils

use of edu.iu.dsc.tws.data.utils.DataNodeLocatorUtils in project twister2 by DSC-SPIDAL.

the class DataLocalityStreamingTaskScheduler method dTTimecalculatorList.

/**
 * Data transfer time calculator list
 */
private List<DataTransferTimeCalculator> dTTimecalculatorList(int index, WorkerPlan workerPlan, Map<Integer, List<TaskInstanceId>> map, int containerIndex, int maxTaskPerContainer) {
    List<String> inputDataList = getInputFilesList();
    Map<String, List<DataTransferTimeCalculator>> workerDatanodeDistanceMap;
    List<DataTransferTimeCalculator> dataTransferTimeCalculatorList = null;
    List<String> datanodesList;
    /*If the index is zero, simply calculate the distance between the worker node and the
    datanodes. Else, if the index values is greater than 0, check the container has reached
    the maximum task instances per container. If it is yes, then calculationList the container to
    the allocatedWorkers list which will not be considered for the next scheduling cycle.*/
    DataNodeLocatorUtils dataNodeLocatorUtils = new DataNodeLocatorUtils(config);
    if (inputDataList.size() > 0) {
        if (index == 0) {
            datanodesList = dataNodeLocatorUtils.findDataNodesLocation(inputDataList);
            workerDatanodeDistanceMap = distanceCalculator(datanodesList, workerPlan, index, allocatedWorkers);
            dataTransferTimeCalculatorList = findBestWorkerNode(workerDatanodeDistanceMap);
        } else {
            datanodesList = dataNodeLocatorUtils.findDataNodesLocation(inputDataList);
            Worker worker = workerPlan.getWorker(containerIndex);
            if (map.get(containerIndex).size() >= maxTaskPerContainer) {
                allocatedWorkers.add(worker.getId());
            }
            workerDatanodeDistanceMap = distanceCalculator(datanodesList, workerPlan, index, allocatedWorkers);
            dataTransferTimeCalculatorList = findBestWorkerNode(workerDatanodeDistanceMap);
        }
    }
    return dataTransferTimeCalculatorList;
}