Search in sources :

Example 1 with Tier

use of com.netflix.titus.api.model.Tier in project titus-control-plane by Netflix.

the class JobAndTaskMetrics method updateTaskMetrics.

private void updateTaskMetrics(TaskUpdateEvent event) {
    Job<?> job = event.getCurrentJob();
    Task task = event.getCurrentTask();
    Pair<Tier, String> assignment = JobManagerUtil.getTierAssignment(job, applicationSlaManagementService);
    registry.counter(TASK_STATE_CHANGE_METRIC_NAME, "tier", assignment.getLeft().name(), "capacityGroup", assignment.getRight(), "applicationName", job.getJobDescriptor().getApplicationName(), "state", task.getStatus().getState().name(), "kubeScheduler", "true").increment();
}
Also used : Task(com.netflix.titus.api.jobmanager.model.job.Task) Tier(com.netflix.titus.api.model.Tier)

Example 2 with Tier

use of com.netflix.titus.api.model.Tier in project titus-control-plane by Netflix.

the class ResourceConsumptionEvaluator method buildSystemConsumption.

private CompositeResourceConsumption buildSystemConsumption(Map<String, Map<String, ResourceConsumption>> capacityGroupConsumptionMap) {
    // Capacity group level
    Map<Tier, List<CompositeResourceConsumption>> tierConsumptions = new HashMap<>();
    capacityGroupConsumptionMap.forEach((capacityGroup, appConsumptions) -> {
        ApplicationSLA sla = applicationSlaMap.get(capacityGroup);
        ResourceDimension allowedConsumption = ResourceDimensions.multiply(sla.getResourceDimension(), sla.getInstanceCount());
        ResourceDimension maxConsumption = ResourceConsumptions.addMaxConsumptions(appConsumptions.values());
        List<Map<String, Object>> attrsList = appConsumptions.values().stream().map(ResourceConsumption::getAttributes).collect(Collectors.toList());
        CompositeResourceConsumption capacityGroupConsumption = new CompositeResourceConsumption(capacityGroup, ConsumptionLevel.CapacityGroup, ResourceConsumptions.addCurrentConsumptions(appConsumptions.values()), maxConsumption, allowedConsumption, ResourceConsumptions.mergeAttributes(attrsList), appConsumptions, !ResourceDimensions.isBigger(allowedConsumption, maxConsumption));
        tierConsumptions.computeIfAbsent(sla.getTier(), t -> new ArrayList<>()).add(capacityGroupConsumption);
    });
    // Tier level
    List<CompositeResourceConsumption> aggregatedTierConsumptions = new ArrayList<>();
    tierConsumptions.forEach((tier, consumptions) -> aggregatedTierConsumptions.add(ResourceConsumptions.aggregate(tier.name(), ConsumptionLevel.Tier, consumptions)));
    // System level
    return ResourceConsumptions.aggregate(SYSTEM_CONSUMER, ConsumptionLevel.System, aggregatedTierConsumptions);
}
Also used : DEFAULT_APPLICATION(com.netflix.titus.master.service.management.ApplicationSlaManagementService.DEFAULT_APPLICATION) ApplicationSlaManagementService(com.netflix.titus.master.service.management.ApplicationSlaManagementService) Task(com.netflix.titus.api.jobmanager.model.job.Task) CollectionsExt(com.netflix.titus.common.util.CollectionsExt) LoggerFactory(org.slf4j.LoggerFactory) Collectors.groupingBy(java.util.stream.Collectors.groupingBy) HashMap(java.util.HashMap) ResourceDimensions(com.netflix.titus.master.model.ResourceDimensions) Function(java.util.function.Function) Supplier(java.util.function.Supplier) CollectionsExt.copyAndRemove(com.netflix.titus.common.util.CollectionsExt.copyAndRemove) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) ResourceConsumption(com.netflix.titus.master.service.management.ResourceConsumption) Pair(com.netflix.titus.common.util.tuple.Pair) Map(java.util.Map) BatchJobExt(com.netflix.titus.api.jobmanager.model.job.ext.BatchJobExt) ApplicationSLA(com.netflix.titus.api.model.ApplicationSLA) ConsumptionLevel(com.netflix.titus.master.service.management.ResourceConsumption.ConsumptionLevel) ResourceDimension(com.netflix.titus.api.model.ResourceDimension) SYSTEM_CONSUMER(com.netflix.titus.master.service.management.ResourceConsumption.SYSTEM_CONSUMER) Job(com.netflix.titus.api.jobmanager.model.job.Job) Logger(org.slf4j.Logger) ServiceJobExt(com.netflix.titus.api.jobmanager.model.job.ext.ServiceJobExt) Collection(java.util.Collection) Set(java.util.Set) JobFunctions(com.netflix.titus.api.jobmanager.model.job.JobFunctions) Tier(com.netflix.titus.api.model.Tier) Collectors(java.util.stream.Collectors) TaskState(com.netflix.titus.api.jobmanager.model.job.TaskState) List(java.util.List) V3JobOperations(com.netflix.titus.api.jobmanager.service.V3JobOperations) ContainerResources(com.netflix.titus.api.jobmanager.model.job.ContainerResources) TaskAttributes(com.netflix.titus.api.jobmanager.TaskAttributes) VisibleForTesting(com.google.common.annotations.VisibleForTesting) CompositeResourceConsumption(com.netflix.titus.master.service.management.CompositeResourceConsumption) Evaluators(com.netflix.titus.common.util.Evaluators) Collections(java.util.Collections) ResourceConsumptions(com.netflix.titus.master.service.management.ResourceConsumptions) CompositeResourceConsumption(com.netflix.titus.master.service.management.CompositeResourceConsumption) Tier(com.netflix.titus.api.model.Tier) HashMap(java.util.HashMap) ApplicationSLA(com.netflix.titus.api.model.ApplicationSLA) ArrayList(java.util.ArrayList) ArrayList(java.util.ArrayList) List(java.util.List) HashMap(java.util.HashMap) Map(java.util.Map) ResourceDimension(com.netflix.titus.api.model.ResourceDimension)

Example 3 with Tier

use of com.netflix.titus.api.model.Tier in project titus-control-plane by Netflix.

the class Representation2ModelConvertions method asCoreEntity.

public static ApplicationSLA asCoreEntity(ApplicationSlaRepresentation representation) {
    ResourceDimension resourceDimension = ResourceDimension.newBuilder().withCpus(representation.getInstanceCPU()).withMemoryMB(representation.getInstanceMemoryMB()).withDiskMB(representation.getInstanceDiskMB()).withNetworkMbs(representation.getInstanceNetworkMbs()).withGpu(representation.getInstanceGPU()).build();
    Tier tier;
    if (representation.getTier() != null) {
        tier = Tier.valueOf(representation.getTier().name());
    } else {
        tier = Tier.Flex;
    }
    return ApplicationSLA.newBuilder().withAppName(representation.getAppName()).withTier(tier).withResourceDimension(resourceDimension).withInstanceCount(representation.getInstanceCount()).withSchedulerName(representation.getSchedulerName()).withResourcePool(representation.getResourcePool()).build();
}
Also used : Tier(com.netflix.titus.api.model.Tier) ResourceDimension(com.netflix.titus.api.model.ResourceDimension)

Example 4 with Tier

use of com.netflix.titus.api.model.Tier in project titus-control-plane by Netflix.

the class ConsumptionModelGenerator method getEvaluation.

DefaultResourceConsumptionService.ConsumptionEvaluationResult getEvaluation() {
    Map<String, CompositeResourceConsumption> groupConsumptionMap = new HashMap<>();
    Set<String> definedCapacityGroups = new HashSet<>(capacityGroupMap.keySet());
    // Used capacity groups
    Set<String> capacityGroupNames = actualConsumptionByGroupAndApp.keySet();
    for (String capacityGroupName : capacityGroupNames) {
        List<ResourceConsumption> appConsumptions = buildApplicationConsumptions(capacityGroupName);
        CompositeResourceConsumption groupConsumption = ResourceConsumptions.aggregate(capacityGroupName, ConsumptionLevel.CapacityGroup, appConsumptions, capacityGroupLimit(capacityGroupMap.getOrDefault(capacityGroupName, DEFAULT_SLA)));
        groupConsumptionMap.put(capacityGroupName, groupConsumption);
    }
    // Unused capacity groups
    CollectionsExt.copyAndRemove(definedCapacityGroups, capacityGroupNames).forEach(capacityGroup -> {
        ApplicationSLA sla = capacityGroupMap.getOrDefault(capacityGroup, DEFAULT_SLA);
        ResourceDimension limit = capacityGroupLimit(sla);
        groupConsumptionMap.put(capacityGroup, new CompositeResourceConsumption(capacityGroup, ConsumptionLevel.CapacityGroup, ResourceDimension.empty(), ResourceDimension.empty(), limit, Collections.emptyMap(), Collections.emptyMap(), false));
    });
    // Undefined capacity groups
    Set<String> undefinedCapacityGroups = CollectionsExt.copyAndRemove(capacityGroupNames, definedCapacityGroups);
    // Tier consumption
    Map<Tier, List<CompositeResourceConsumption>> tierCapacityGroups = groupConsumptionMap.values().stream().collect(Collectors.groupingBy(rc -> {
        ApplicationSLA sla = capacityGroupMap.get(rc.getConsumerName());
        if (sla == null) {
            sla = capacityGroupMap.get(DEFAULT_SLA.getAppName());
        }
        return sla.getTier();
    }));
    Map<String, CompositeResourceConsumption> tierConsumptions = new HashMap<>();
    tierCapacityGroups.forEach((tier, consumptions) -> tierConsumptions.put(tier.name(), ResourceConsumptions.aggregate(tier.name(), ConsumptionLevel.Tier, consumptions)));
    // System consumption
    CompositeResourceConsumption systemConsumption = ResourceConsumptions.aggregate(ResourceConsumption.SYSTEM_CONSUMER, ConsumptionLevel.System, tierConsumptions.values());
    return new DefaultResourceConsumptionService.ConsumptionEvaluationResult(definedCapacityGroups, undefinedCapacityGroups, systemConsumption);
}
Also used : ResourceConsumptionEvent(com.netflix.titus.master.service.management.ResourceConsumptionEvents.ResourceConsumptionEvent) CollectionsExt(com.netflix.titus.common.util.CollectionsExt) Set(java.util.Set) HashMap(java.util.HashMap) Tier(com.netflix.titus.api.model.Tier) ResourceDimensions(com.netflix.titus.master.model.ResourceDimensions) Collectors(java.util.stream.Collectors) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) ApplicationSlaSample(com.netflix.titus.testkit.data.core.ApplicationSlaSample) List(java.util.List) ResourceConsumption(com.netflix.titus.master.service.management.ResourceConsumption) Map(java.util.Map) Optional(java.util.Optional) Preconditions(com.google.common.base.Preconditions) ApplicationSLA(com.netflix.titus.api.model.ApplicationSLA) CompositeResourceConsumption(com.netflix.titus.master.service.management.CompositeResourceConsumption) ConsumptionLevel(com.netflix.titus.master.service.management.ResourceConsumption.ConsumptionLevel) Collections(java.util.Collections) ResourceDimension(com.netflix.titus.api.model.ResourceDimension) ResourceConsumptions(com.netflix.titus.master.service.management.ResourceConsumptions) HashMap(java.util.HashMap) Tier(com.netflix.titus.api.model.Tier) CompositeResourceConsumption(com.netflix.titus.master.service.management.CompositeResourceConsumption) ResourceConsumption(com.netflix.titus.master.service.management.ResourceConsumption) CompositeResourceConsumption(com.netflix.titus.master.service.management.CompositeResourceConsumption) ApplicationSLA(com.netflix.titus.api.model.ApplicationSLA) ArrayList(java.util.ArrayList) List(java.util.List) ResourceDimension(com.netflix.titus.api.model.ResourceDimension) HashSet(java.util.HashSet)

Aggregations

Tier (com.netflix.titus.api.model.Tier)4 ResourceDimension (com.netflix.titus.api.model.ResourceDimension)3 Task (com.netflix.titus.api.jobmanager.model.job.Task)2 ApplicationSLA (com.netflix.titus.api.model.ApplicationSLA)2 CollectionsExt (com.netflix.titus.common.util.CollectionsExt)2 ResourceDimensions (com.netflix.titus.master.model.ResourceDimensions)2 CompositeResourceConsumption (com.netflix.titus.master.service.management.CompositeResourceConsumption)2 ResourceConsumption (com.netflix.titus.master.service.management.ResourceConsumption)2 ConsumptionLevel (com.netflix.titus.master.service.management.ResourceConsumption.ConsumptionLevel)2 ResourceConsumptions (com.netflix.titus.master.service.management.ResourceConsumptions)2 ArrayList (java.util.ArrayList)2 Collections (java.util.Collections)2 HashMap (java.util.HashMap)2 HashSet (java.util.HashSet)2 List (java.util.List)2 Map (java.util.Map)2 Set (java.util.Set)2 Collectors (java.util.stream.Collectors)2 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 Preconditions (com.google.common.base.Preconditions)1