Search in sources :

Example 1 with EvictionConfiguration

use of com.netflix.titus.runtime.connector.eviction.EvictionConfiguration in project titus-control-plane by Netflix.

the class TitusQuotasManagerTest method isJobExemptFromSystemDisruptionWindow.

@Test
public void isJobExemptFromSystemDisruptionWindow() {
    Job<BatchJobExt> job1 = JobGenerator.oneBatchJob().but(withApplicationName("app1Test"));
    EvictionConfiguration config1 = mock(EvictionConfiguration.class);
    when(config1.getAppsExemptFromSystemDisruptionWindow()).thenReturn("app1.*");
    TitusQuotasManager titusQuotasManager = new TitusQuotasManager(null, null, null, null, config1, null);
    boolean jobExemptFromSystemDisruptionBudget = titusQuotasManager.isJobExemptFromSystemDisruptionWindow(job1);
    assertThat(jobExemptFromSystemDisruptionBudget).isTrue();
    EvictionConfiguration config2 = mock(EvictionConfiguration.class);
    when(config2.getAppsExemptFromSystemDisruptionWindow()).thenReturn("app2.*");
    TitusQuotasManager titusQuotasManager2 = new TitusQuotasManager(null, null, null, null, config2, null);
    boolean jobExemptFromSystemDisruptionBudget2 = titusQuotasManager2.isJobExemptFromSystemDisruptionWindow(job1);
    assertThat(jobExemptFromSystemDisruptionBudget2).isFalse();
}
Also used : BatchJobExt(com.netflix.titus.api.jobmanager.model.job.ext.BatchJobExt) EvictionConfiguration(com.netflix.titus.runtime.connector.eviction.EvictionConfiguration) Test(org.junit.Test)

Example 2 with EvictionConfiguration

use of com.netflix.titus.runtime.connector.eviction.EvictionConfiguration in project titus-control-plane by Netflix.

the class TitusQuotasManagerTest method tryConsumeSystemAndJobQuota.

@Test
public void tryConsumeSystemAndJobQuota() {
    String taskId = "job1Task1";
    String jobQuotaRejectionReason = "Job does not allow any more terminations";
    Job<BatchJobExt> job1 = JobGenerator.oneBatchJob().but(withApplicationName("app1Test"));
    EvictionConfiguration config1 = mock(EvictionConfiguration.class);
    when(config1.getAppsExemptFromSystemDisruptionWindow()).thenReturn("app1.*");
    SystemQuotaController systemQuotaController = mock(SystemQuotaController.class);
    when(systemQuotaController.consume(taskId)).thenReturn(ConsumptionResult.approved());
    JobQuotaController jobQuotaController = mock(JobQuotaController.class);
    when(jobQuotaController.consume(taskId)).thenReturn(ConsumptionResult.rejected(jobQuotaRejectionReason));
    TitusQuotasManager titusQuotasManager = new TitusQuotasManager(null, null, null, systemQuotaController, config1, null);
    ConsumptionResult consumptionResult = titusQuotasManager.tryConsumeSystemAndJobQuota(jobQuotaController, job1, taskId);
    assertThat(consumptionResult.isApproved()).isFalse();
    assertThat(consumptionResult.getRejectionReason()).isPresent();
    assertThat(consumptionResult.getRejectionReason().get()).isEqualTo(jobQuotaRejectionReason);
    JobQuotaController jobQuotaController2 = mock(JobQuotaController.class);
    when(jobQuotaController2.consume(taskId)).thenReturn(ConsumptionResult.approved());
    ConsumptionResult consumptionResult2 = titusQuotasManager.tryConsumeSystemAndJobQuota(jobQuotaController2, job1, taskId);
    assertThat(consumptionResult2.isApproved()).isTrue();
    String quotaLimitExceededReason = SystemQuotaConsumptionResults.QUOTA_LIMIT_EXCEEDED.getRejectionReason().get();
    when(systemQuotaController.consume(taskId)).thenReturn(ConsumptionResult.rejected(quotaLimitExceededReason));
    ConsumptionResult consumptionResult3 = titusQuotasManager.tryConsumeSystemAndJobQuota(jobQuotaController2, job1, taskId);
    assertThat(consumptionResult3.isApproved()).isFalse();
    assertThat(consumptionResult3.getRejectionReason()).isPresent();
    assertThat(consumptionResult3.getRejectionReason().get()).isEqualTo(quotaLimitExceededReason);
    String outsideSystemWindowReason = SystemQuotaConsumptionResults.OUTSIDE_SYSTEM_TIME_WINDOW.getRejectionReason().get();
    when(systemQuotaController.consume(taskId)).thenReturn(ConsumptionResult.rejected(outsideSystemWindowReason));
    ConsumptionResult consumptionResult4 = titusQuotasManager.tryConsumeSystemAndJobQuota(jobQuotaController2, job1, taskId);
    assertThat(consumptionResult4.isApproved()).isTrue();
}
Also used : BatchJobExt(com.netflix.titus.api.jobmanager.model.job.ext.BatchJobExt) EvictionConfiguration(com.netflix.titus.runtime.connector.eviction.EvictionConfiguration) JobQuotaController(com.netflix.titus.master.eviction.service.quota.job.JobQuotaController) SystemQuotaController(com.netflix.titus.master.eviction.service.quota.system.SystemQuotaController) Test(org.junit.Test)

Example 3 with EvictionConfiguration

use of com.netflix.titus.runtime.connector.eviction.EvictionConfiguration in project titus-control-plane by Netflix.

the class DefaultDeschedulerService method deschedule.

@Override
public List<DeschedulingResult> deschedule(Map<String, TaskRelocationPlan> plannedAheadTaskRelocationPlans) {
    List<Pair<Job, List<Task>>> allJobsAndTasks = jobOperations.getJobsAndTasks();
    Map<String, Job<?>> jobs = allJobsAndTasks.stream().map(Pair::getLeft).collect(Collectors.toMap(Job::getId, j -> j));
    Map<String, Task> tasksById = allJobsAndTasks.stream().flatMap(p -> p.getRight().stream()).collect(Collectors.toMap(Task::getId, t -> t));
    EvacuatedAgentsAllocationTracker evacuatedAgentsAllocationTracker = new EvacuatedAgentsAllocationTracker(nodeDataResolver.resolve(), tasksById);
    EvictionQuotaTracker evictionQuotaTracker = new EvictionQuotaTracker(evictionOperations, jobs);
    TaskMigrationDescheduler taskMigrationDescheduler = new TaskMigrationDescheduler(plannedAheadTaskRelocationPlans, evacuatedAgentsAllocationTracker, evictionQuotaTracker, evictionConfiguration, jobs, tasksById, titusRuntime);
    Map<String, DeschedulingResult> requestedImmediateEvictions = taskMigrationDescheduler.findAllImmediateEvictions();
    Map<String, DeschedulingResult> requestedEvictions = taskMigrationDescheduler.findRequestedJobOrTaskMigrations();
    Map<String, DeschedulingResult> allRequestedEvictions = CollectionsExt.merge(requestedImmediateEvictions, requestedEvictions);
    Map<String, DeschedulingResult> regularEvictions = new HashMap<>();
    Optional<Pair<TitusNode, List<Task>>> bestMatch;
    while ((bestMatch = taskMigrationDescheduler.nextBestMatch()).isPresent()) {
        TitusNode agent = bestMatch.get().getLeft();
        List<Task> tasks = bestMatch.get().getRight();
        tasks.forEach(task -> {
            if (!allRequestedEvictions.containsKey(task.getId())) {
                Optional<TaskRelocationPlan> relocationPlanForTask = getRelocationPlanForTask(agent, task, plannedAheadTaskRelocationPlans);
                relocationPlanForTask.ifPresent(rp -> regularEvictions.put(task.getId(), DeschedulingResult.newBuilder().withTask(task).withAgentInstance(agent).withTaskRelocationPlan(rp).build()));
            }
        });
    }
    // Find eviction which could not be scheduled now.
    for (Task task : tasksById.values()) {
        if (allRequestedEvictions.containsKey(task.getId()) || regularEvictions.containsKey(task.getId())) {
            continue;
        }
        if (evacuatedAgentsAllocationTracker.isEvacuated(task)) {
            DeschedulingFailure failure = taskMigrationDescheduler.getDeschedulingFailure(task);
            TaskRelocationPlan relocationPlan = plannedAheadTaskRelocationPlans.get(task.getId());
            if (relocationPlan == null) {
                relocationPlan = newNotDelayedRelocationPlan(task, false);
            }
            TitusNode agent = evacuatedAgentsAllocationTracker.getRemovableAgent(task);
            regularEvictions.put(task.getId(), DeschedulingResult.newBuilder().withTask(task).withAgentInstance(agent).withTaskRelocationPlan(relocationPlan).withFailure(failure).build());
        }
    }
    return CollectionsExt.merge(new ArrayList<>(allRequestedEvictions.values()), new ArrayList<>(regularEvictions.values()));
}
Also used : Task(com.netflix.titus.api.jobmanager.model.job.Task) CollectionsExt(com.netflix.titus.common.util.CollectionsExt) HashMap(java.util.HashMap) RelocationPredicates(com.netflix.titus.supplementary.relocation.util.RelocationPredicates) Singleton(javax.inject.Singleton) AtomicReference(java.util.concurrent.atomic.AtomicReference) ArrayList(java.util.ArrayList) Inject(javax.inject.Inject) Pair(com.netflix.titus.common.util.tuple.Pair) Map(java.util.Map) EvictionConfiguration(com.netflix.titus.runtime.connector.eviction.EvictionConfiguration) NodeDataResolver(com.netflix.titus.supplementary.relocation.connector.NodeDataResolver) ReadOnlyJobOperations(com.netflix.titus.api.jobmanager.service.ReadOnlyJobOperations) TaskRelocationReason(com.netflix.titus.api.relocation.model.TaskRelocationPlan.TaskRelocationReason) TaskRelocationPlan(com.netflix.titus.api.relocation.model.TaskRelocationPlan) DeschedulingFailure(com.netflix.titus.supplementary.relocation.model.DeschedulingFailure) DeschedulingResult(com.netflix.titus.supplementary.relocation.model.DeschedulingResult) Job(com.netflix.titus.api.jobmanager.model.job.Job) Collectors(java.util.stream.Collectors) List(java.util.List) ReadOnlyEvictionOperations(com.netflix.titus.api.eviction.service.ReadOnlyEvictionOperations) Optional(java.util.Optional) RelocationUtil(com.netflix.titus.supplementary.relocation.util.RelocationUtil) VisibleForTesting(com.google.common.annotations.VisibleForTesting) TitusRuntime(com.netflix.titus.common.runtime.TitusRuntime) Clock(com.netflix.titus.common.util.time.Clock) TitusNode(com.netflix.titus.supplementary.relocation.connector.TitusNode) JobFunctions.hasDisruptionBudget(com.netflix.titus.api.jobmanager.model.job.JobFunctions.hasDisruptionBudget) Task(com.netflix.titus.api.jobmanager.model.job.Task) HashMap(java.util.HashMap) DeschedulingResult(com.netflix.titus.supplementary.relocation.model.DeschedulingResult) TaskRelocationPlan(com.netflix.titus.api.relocation.model.TaskRelocationPlan) DeschedulingFailure(com.netflix.titus.supplementary.relocation.model.DeschedulingFailure) Job(com.netflix.titus.api.jobmanager.model.job.Job) TitusNode(com.netflix.titus.supplementary.relocation.connector.TitusNode) Pair(com.netflix.titus.common.util.tuple.Pair)

Aggregations

EvictionConfiguration (com.netflix.titus.runtime.connector.eviction.EvictionConfiguration)3 BatchJobExt (com.netflix.titus.api.jobmanager.model.job.ext.BatchJobExt)2 Test (org.junit.Test)2 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 ReadOnlyEvictionOperations (com.netflix.titus.api.eviction.service.ReadOnlyEvictionOperations)1 Job (com.netflix.titus.api.jobmanager.model.job.Job)1 JobFunctions.hasDisruptionBudget (com.netflix.titus.api.jobmanager.model.job.JobFunctions.hasDisruptionBudget)1 Task (com.netflix.titus.api.jobmanager.model.job.Task)1 ReadOnlyJobOperations (com.netflix.titus.api.jobmanager.service.ReadOnlyJobOperations)1 TaskRelocationPlan (com.netflix.titus.api.relocation.model.TaskRelocationPlan)1 TaskRelocationReason (com.netflix.titus.api.relocation.model.TaskRelocationPlan.TaskRelocationReason)1 TitusRuntime (com.netflix.titus.common.runtime.TitusRuntime)1 CollectionsExt (com.netflix.titus.common.util.CollectionsExt)1 Clock (com.netflix.titus.common.util.time.Clock)1 Pair (com.netflix.titus.common.util.tuple.Pair)1 JobQuotaController (com.netflix.titus.master.eviction.service.quota.job.JobQuotaController)1 SystemQuotaController (com.netflix.titus.master.eviction.service.quota.system.SystemQuotaController)1 NodeDataResolver (com.netflix.titus.supplementary.relocation.connector.NodeDataResolver)1 TitusNode (com.netflix.titus.supplementary.relocation.connector.TitusNode)1 DeschedulingFailure (com.netflix.titus.supplementary.relocation.model.DeschedulingFailure)1