Examples with Schedulable org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.Schedulable used on opensource projects

Example 1 with Schedulable

use of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.Schedulable in project hadoop by apache.

the class ComputeFairShares method computeSharesInternal.

/**
   * Given a set of Schedulables and a number of slots, compute their weighted
   * fair shares. The min and max shares and of the Schedulables are assumed to
   * be set beforehand. We compute the fairest possible allocation of shares to
   * the Schedulables that respects their min and max shares.
   * <p>
   * To understand what this method does, we must first define what weighted
   * fair sharing means in the presence of min and max shares. If there
   * were no minimum or maximum shares, then weighted fair sharing would be
   * achieved if the ratio of slotsAssigned / weight was equal for each
   * Schedulable and all slots were assigned. Minimum and maximum shares add a
   * further twist - Some Schedulables may have a min share higher than their
   * assigned share or a max share lower than their assigned share.
   * <p>
   * To deal with these possibilities, we define an assignment of slots as being
   * fair if there exists a ratio R such that: Schedulables S where S.minShare
   * {@literal >} R * S.weight are given share S.minShare - Schedulables S
   * where S.maxShare {@literal <} R * S.weight are given S.maxShare -
   * All other Schedulables S are assigned share R * S.weight -
   * The sum of all the shares is totalSlots.
   * <p>
   * We call R the weight-to-slots ratio because it converts a Schedulable's
   * weight to the number of slots it is assigned.
   * <p>
   * We compute a fair allocation by finding a suitable weight-to-slot ratio R.
   * To do this, we use binary search. Given a ratio R, we compute the number of
   * slots that would be used in total with this ratio (the sum of the shares
   * computed using the conditions above). If this number of slots is less than
   * totalSlots, then R is too small and more slots could be assigned. If the
   * number of slots is more than totalSlots, then R is too large.
   * <p>
   * We begin the binary search with a lower bound on R of 0 (which means that
   * all Schedulables are only given their minShare) and an upper bound computed
   * to be large enough that too many slots are given (by doubling R until we
   * use more than totalResources resources). The helper method
   * resourceUsedWithWeightToResourceRatio computes the total resources used with a
   * given value of R.
   * <p>
   * The running time of this algorithm is linear in the number of Schedulables,
   * because resourceUsedWithWeightToResourceRatio is linear-time and the number of
   * iterations of binary search is a constant (dependent on desired precision).
   */
private static void computeSharesInternal(Collection<? extends Schedulable> allSchedulables, Resource totalResources, ResourceType type, boolean isSteadyShare) {
    Collection<Schedulable> schedulables = new ArrayList<Schedulable>();
    int takenResources = handleFixedFairShares(allSchedulables, schedulables, isSteadyShare, type);
    if (schedulables.isEmpty()) {
        return;
    }
    // Find an upper bound on R that we can use in our binary search. We start
    // at R = 1 and double it until we have either used all the resources or we
    // have met all Schedulables' max shares.
    int totalMaxShare = 0;
    for (Schedulable sched : schedulables) {
        long maxShare = getResourceValue(sched.getMaxShare(), type);
        totalMaxShare = (int) Math.min(maxShare + (long) totalMaxShare, Integer.MAX_VALUE);
        if (totalMaxShare == Integer.MAX_VALUE) {
            break;
        }
    }
    long totalResource = Math.max((getResourceValue(totalResources, type) - takenResources), 0);
    totalResource = Math.min(totalMaxShare, totalResource);
    double rMax = 1.0;
    while (resourceUsedWithWeightToResourceRatio(rMax, schedulables, type) < totalResource) {
        rMax *= 2.0;
    }
    // Perform the binary search for up to COMPUTE_FAIR_SHARES_ITERATIONS steps
    double left = 0;
    double right = rMax;
    for (int i = 0; i < COMPUTE_FAIR_SHARES_ITERATIONS; i++) {
        double mid = (left + right) / 2.0;
        int plannedResourceUsed = resourceUsedWithWeightToResourceRatio(mid, schedulables, type);
        if (plannedResourceUsed == totalResource) {
            right = mid;
            break;
        } else if (plannedResourceUsed < totalResource) {
            left = mid;
        } else {
            right = mid;
        }
    }
    // Set the fair shares based on the value of R we've converged to
    for (Schedulable sched : schedulables) {
        if (isSteadyShare) {
            setResourceValue(computeShare(sched, right, type), ((FSQueue) sched).getSteadyFairShare(), type);
        } else {
            setResourceValue(computeShare(sched, right, type), sched.getFairShare(), type);
        }
    }
}

Example 2 with Schedulable

use of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.Schedulable in project hadoop by apache.

the class TestDominantResourceFairnessPolicy method testCompareSchedulablesWithClusterResourceChanges.

@Test
public void testCompareSchedulablesWithClusterResourceChanges() {
    Schedulable schedulable1 = createSchedulable(2000, 1);
    Schedulable schedulable2 = createSchedulable(1000, 2);
    // schedulable1 has share weights [1/2, 1/5], schedulable2 has share
    // weights [1/4, 2/5], schedulable1 > schedulable2 since 1/2 > 2/5
    assertTrue(createComparator(4000, 5).compare(schedulable1, schedulable2) > 0);
    // share weights have changed because of the cluster resource change.
    // schedulable1 has share weights [1/4, 1/6], schedulable2 has share
    // weights [1/8, 1/3], schedulable1 < schedulable2 since 1/4 < 1/3
    assertTrue(createComparator(8000, 6).compare(schedulable1, schedulable2) < 0);
}

Example 3 with Schedulable

use of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.Schedulable in project hadoop by apache.

the class ComputeFairShares method resourceUsedWithWeightToResourceRatio.

/**
   * Compute the resources that would be used given a weight-to-resource ratio
   * w2rRatio, for use in the computeFairShares algorithm as described in #
   */
private static int resourceUsedWithWeightToResourceRatio(double w2rRatio, Collection<? extends Schedulable> schedulables, ResourceType type) {
    int resourcesTaken = 0;
    for (Schedulable sched : schedulables) {
        int share = computeShare(sched, w2rRatio, type);
        resourcesTaken += share;
    }
    return resourcesTaken;
}

Example 4 with Schedulable

use of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.Schedulable in project hadoop by apache.

the class ComputeFairShares method handleFixedFairShares.

/**
   * Helper method to handle Schedulabes with fixed fairshares.
   * Returns the resources taken by fixed fairshare schedulables,
   * and adds the remaining to the passed nonFixedSchedulables.
   */
private static int handleFixedFairShares(Collection<? extends Schedulable> schedulables, Collection<Schedulable> nonFixedSchedulables, boolean isSteadyShare, ResourceType type) {
    int totalResource = 0;
    for (Schedulable sched : schedulables) {
        long fixedShare = getFairShareIfFixed(sched, isSteadyShare, type);
        if (fixedShare < 0) {
            nonFixedSchedulables.add(sched);
        } else {
            setResourceValue(fixedShare, isSteadyShare ? ((FSQueue) sched).getSteadyFairShare() : sched.getFairShare(), type);
            totalResource = (int) Math.min((long) totalResource + (long) fixedShare, Integer.MAX_VALUE);
        }
    }
    return totalResource;
}