Examples with ObjectResourcesSummary org.apache.storm.scheduler.resource.strategies.scheduling.ObjectResourcesSummary used on opensource projects

Example 11 with ObjectResourcesSummary

use of org.apache.storm.scheduler.resource.strategies.scheduling.ObjectResourcesSummary in project storm by apache.

the class NodeSorter method sortObjectResourcesGeneric.

/**
 * Sort objects by the following two criteria.
 *
 * <li>the number executors of the topology that needs to be scheduled is already on the
 * object (node or rack) in descending order. The reasoning to sort based on criterion 1 is so we schedule the rest
 * of a topology on the same object (node or rack) as the existing executors of the topology.</li>
 *
 * <li>the subordinate/subservient resource availability percentage of a rack in descending order We calculate the
 * resource availability percentage by dividing the resource availability of the object (node or rack) by the
 * resource availability of the entire rack or cluster depending on if object references a node or a rack.
 * How this differs from the DefaultResourceAwareStrategy is that the percentage boosts the node or rack if it is
 * requested by the executor that the sorting is being done for and pulls it down if it is not.
 * By doing this calculation, objects (node or rack) that have exhausted or little of one of the resources mentioned
 * above will be ranked after racks that have more balanced resource availability and nodes or racks that have
 * resources that are not requested will be ranked below . So we will be less likely to pick a rack that
 * have a lot of one resource but a low amount of another and have a lot of resources that are not requested by the executor.</li>
 *
 * @param allResources         contains all individual ObjectResources as well as cumulative stats
 * @param exec                 executor for which the sorting is done
 * @param existingScheduleFunc a function to get existing executors already scheduled on this object
 * @return a sorted list of ObjectResources
 */
@Deprecated
private List<ObjectResourcesItem> sortObjectResourcesGeneric(final ObjectResourcesSummary allResources, ExecutorDetails exec, final ExistingScheduleFunc existingScheduleFunc) {
    ObjectResourcesSummary affinityBasedAllResources = new ObjectResourcesSummary(allResources);
    NormalizedResourceRequest requestedResources = topologyDetails.getTotalResources(exec);
    affinityBasedAllResources.getObjectResources().forEach(x -> x.availableResources.updateForRareResourceAffinity(requestedResources));
    final NormalizedResourceOffer availableResourcesOverall = allResources.getAvailableResourcesOverall();
    List<ObjectResourcesItem> sortedObjectResources = new ArrayList<>();
    Comparator<ObjectResourcesItem> comparator = (o1, o2) -> {
        int execsScheduled1 = existingScheduleFunc.getNumExistingSchedule(o1.id);
        int execsScheduled2 = existingScheduleFunc.getNumExistingSchedule(o2.id);
        if (execsScheduled1 > execsScheduled2) {
            return -1;
        } else if (execsScheduled1 < execsScheduled2) {
            return 1;
        }
        double o1Avg = availableResourcesOverall.calculateAveragePercentageUsedBy(o1.availableResources);
        double o2Avg = availableResourcesOverall.calculateAveragePercentageUsedBy(o2.availableResources);
        if (o1Avg > o2Avg) {
            return -1;
        } else if (o1Avg < o2Avg) {
            return 1;
        }
        return o1.id.compareTo(o2.id);
    };
    sortedObjectResources.addAll(affinityBasedAllResources.getObjectResources());
    sortedObjectResources.sort(comparator);
    LOG.debug("Sorted Object Resources: {}", sortedObjectResources);
    return sortedObjectResources;
}

Example 12 with ObjectResourcesSummary

use of org.apache.storm.scheduler.resource.strategies.scheduling.ObjectResourcesSummary in project storm by apache.

the class NodeSorter method sortObjectResourcesDefault.

/**
 * Sort objects by the following two criteria.
 *
 * <li>the number executors of the topology that needs to be scheduled is already on the
 * object (node or rack) in descending order. The reasoning to sort based on criterion 1 is so we schedule the rest
 * of a topology on the same object (node or rack) as the existing executors of the topology.</li>
 *
 * <li>the subordinate/subservient resource availability percentage of a rack in descending order We calculate the
 * resource availability percentage by dividing the resource availability of the object (node or rack) by the
 * resource availability of the entire rack or cluster depending on if object references a node or a rack.
 * By doing this calculation, objects (node or rack) that have exhausted or little of one of the resources mentioned
 * above will be ranked after racks that have more balanced resource availability. So we will be less likely to pick
 * a rack that have a lot of one resource but a low amount of another.</li>
 *
 * @param allResources         contains all individual ObjectResources as well as cumulative stats
 * @param existingScheduleFunc a function to get existing executors already scheduled on this object
 * @return a sorted list of ObjectResources
 */
@Deprecated
private List<ObjectResourcesItem> sortObjectResourcesDefault(final ObjectResourcesSummary allResources, final ExistingScheduleFunc existingScheduleFunc) {
    final NormalizedResourceOffer availableResourcesOverall = allResources.getAvailableResourcesOverall();
    for (ObjectResourcesItem objectResources : allResources.getObjectResources()) {
        objectResources.minResourcePercent = availableResourcesOverall.calculateMinPercentageUsedBy(objectResources.availableResources);
        objectResources.avgResourcePercent = availableResourcesOverall.calculateAveragePercentageUsedBy(objectResources.availableResources);
        LOG.trace("for {}: minResourcePercent={}, avgResourcePercent={}, numExistingSchedule={}", objectResources.id, objectResources.minResourcePercent, objectResources.avgResourcePercent, existingScheduleFunc.getNumExistingSchedule(objectResources.id));
    }
    List<ObjectResourcesItem> sortedObjectResources = new ArrayList<>();
    Comparator<ObjectResourcesItem> comparator = (o1, o2) -> {
        int execsScheduled1 = existingScheduleFunc.getNumExistingSchedule(o1.id);
        int execsScheduled2 = existingScheduleFunc.getNumExistingSchedule(o2.id);
        if (execsScheduled1 > execsScheduled2) {
            return -1;
        } else if (execsScheduled1 < execsScheduled2) {
            return 1;
        }
        if (o1.minResourcePercent > o2.minResourcePercent) {
            return -1;
        } else if (o1.minResourcePercent < o2.minResourcePercent) {
            return 1;
        }
        double diff = o1.avgResourcePercent - o2.avgResourcePercent;
        if (diff > 0.0) {
            return -1;
        } else if (diff < 0.0) {
            return 1;
        }
        return o1.id.compareTo(o2.id);
    };
    sortedObjectResources.addAll(allResources.getObjectResources());
    sortedObjectResources.sort(comparator);
    LOG.debug("Sorted Object Resources: {}", sortedObjectResources);
    return sortedObjectResources;
}

Example 13 with ObjectResourcesSummary

use of org.apache.storm.scheduler.resource.strategies.scheduling.ObjectResourcesSummary in project storm by apache.

the class NodeSorter method sortNodes.

/**
 * Nodes are sorted by two criteria.
 *
 * <p>1) the number executors of the topology that needs to be scheduled is already on the node in
 * descending order. The reasoning to sort based on criterion 1 is so we schedule the rest of a topology on the same node as the
 * existing executors of the topology.
 *
 * <p>2) the subordinate/subservient resource availability percentage of a node in descending
 * order We calculate the resource availability percentage by dividing the resource availability that have exhausted or little of one of
 * the resources mentioned above will be ranked after on the node by the resource availability of the entire rack By doing this
 * calculation, nodes nodes that have more balanced resource availability. So we will be less likely to pick a node that have a lot of
 * one resource but a low amount of another.
 *
 * @param availRasNodes a list of all the nodes we want to sort
 * @param rackId     the rack id availNodes are a part of
 * @return a sorted list of nodes.
 */
private List<ObjectResourcesItem> sortNodes(List<RasNode> availRasNodes, ExecutorDetails exec, String rackId, Map<String, AtomicInteger> scheduledCount) {
    ObjectResourcesSummary rackResourcesSummary = new ObjectResourcesSummary("RACK");
    availRasNodes.forEach(x -> rackResourcesSummary.addObjectResourcesItem(new ObjectResourcesItem(x.getId(), x.getTotalAvailableResources(), x.getTotalResources(), 0, 0)));
    LOG.debug("Rack {}: Overall Avail [ {} ] Total [ {} ]", rackId, rackResourcesSummary.getAvailableResourcesOverall(), rackResourcesSummary.getTotalResourcesOverall());
    return sortObjectResources(rackResourcesSummary, exec, (superId) -> {
        AtomicInteger count = scheduledCount.get(superId);
        if (count == null) {
            return 0;
        }
        return count.get();
    });
}