Search in sources :

Example 11 with EntityHolder

use of com.netflix.titus.common.framework.reconciler.EntityHolder in project titus-control-plane by Netflix.

the class CreateOrReplaceBatchTaskActions method createNewTaskModelAction.

private static List<ModelActionHolder> createNewTaskModelAction(TitusChangeAction.Builder changeActionBuilder, BatchJobTask newTask, Optional<BatchJobTask> replacementOf, Retryer nextTaskRetryer) {
    List<ModelActionHolder> actions = new ArrayList<>();
    TitusModelAction.Builder modelBuilder = TitusModelAction.newModelUpdate(changeActionBuilder).summary("Creating new task entity holder");
    EntityHolder taskHolder = EntityHolder.newRoot(newTask.getId(), newTask).addTag(TaskRetryers.ATTR_TASK_RETRY, nextTaskRetryer);
    if (replacementOf.isPresent()) {
        taskHolder = taskHolder.addTag(JobEntityHolders.ATTR_REPLACEMENT_OF, replacementOf.get());
    }
    actions.add(ModelActionHolder.reference(modelBuilder.addTaskHolder(taskHolder)));
    actions.add(ModelActionHolder.store(modelBuilder.taskUpdate(newTask)));
    return actions;
}
Also used : TitusModelAction(com.netflix.titus.master.jobmanager.service.common.action.TitusModelAction) ArrayList(java.util.ArrayList) EntityHolder(com.netflix.titus.common.framework.reconciler.EntityHolder) ModelActionHolder(com.netflix.titus.common.framework.reconciler.ModelActionHolder)

Example 12 with EntityHolder

use of com.netflix.titus.common.framework.reconciler.EntityHolder in project titus-control-plane by Netflix.

the class DifferenceResolverUtils method countActiveNotStartedTasks.

public static int countActiveNotStartedTasks(EntityHolder refJobHolder, EntityHolder runningJobHolder) {
    Set<String> pendingTaskIds = new HashSet<>();
    Consumer<EntityHolder> countingFun = jobHolder -> jobHolder.getChildren().forEach(taskHolder -> {
        TaskState state = ((Task) taskHolder.getEntity()).getStatus().getState();
        if (state != TaskState.Started && state != TaskState.Finished) {
            pendingTaskIds.add(taskHolder.getId());
        }
    });
    countingFun.accept(refJobHolder);
    countingFun.accept(runningJobHolder);
    return pendingTaskIds.size();
}
Also used : JobManagerConstants(com.netflix.titus.api.jobmanager.service.JobManagerConstants) JobServiceRuntime(com.netflix.titus.master.jobmanager.service.JobServiceRuntime) Task(com.netflix.titus.api.jobmanager.model.job.Task) HashMap(java.util.HashMap) Function(java.util.function.Function) TaskTimeoutChangeActions(com.netflix.titus.master.jobmanager.service.common.action.task.TaskTimeoutChangeActions) ArrayList(java.util.ArrayList) EbsVolume(com.netflix.titus.api.jobmanager.model.job.ebs.EbsVolume) TASK_ATTRIBUTES_EBS_VOLUME_ID(com.netflix.titus.api.jobmanager.TaskAttributes.TASK_ATTRIBUTES_EBS_VOLUME_ID) HashSet(java.util.HashSet) Map(java.util.Map) JobState(com.netflix.titus.api.jobmanager.model.job.JobState) BatchJobExt(com.netflix.titus.api.jobmanager.model.job.ext.BatchJobExt) ChangeAction(com.netflix.titus.common.framework.reconciler.ChangeAction) JobManagerConfiguration(com.netflix.titus.master.jobmanager.service.JobManagerConfiguration) JobStore(com.netflix.titus.api.jobmanager.store.JobStore) JobDescriptor(com.netflix.titus.api.jobmanager.model.job.JobDescriptor) Job(com.netflix.titus.api.jobmanager.model.job.Job) ServiceJobExt(com.netflix.titus.api.jobmanager.model.job.ext.ServiceJobExt) TaskStatus(com.netflix.titus.api.jobmanager.model.job.TaskStatus) Set(java.util.Set) JobFunctions(com.netflix.titus.api.jobmanager.model.job.JobFunctions) Collectors(java.util.stream.Collectors) TaskState(com.netflix.titus.api.jobmanager.model.job.TaskState) EntityHolder(com.netflix.titus.common.framework.reconciler.EntityHolder) Consumer(java.util.function.Consumer) List(java.util.List) ExecutableStatus(com.netflix.titus.api.jobmanager.model.job.ExecutableStatus) V3JobOperations(com.netflix.titus.api.jobmanager.service.V3JobOperations) VersionSupplier(com.netflix.titus.master.jobmanager.service.VersionSupplier) ReconciliationEngine(com.netflix.titus.common.framework.reconciler.ReconciliationEngine) Optional(java.util.Optional) BasicTaskActions(com.netflix.titus.master.jobmanager.service.common.action.task.BasicTaskActions) JobManagerReconcilerEvent(com.netflix.titus.master.jobmanager.service.event.JobManagerReconcilerEvent) TitusRuntime(com.netflix.titus.common.runtime.TitusRuntime) TokenBucket(com.netflix.titus.common.util.limiter.tokenbucket.TokenBucket) Clock(com.netflix.titus.common.util.time.Clock) KillInitiatedActions(com.netflix.titus.master.jobmanager.service.common.action.task.KillInitiatedActions) TASK_ATTRIBUTES_IP_ALLOCATION_ID(com.netflix.titus.api.jobmanager.TaskAttributes.TASK_ATTRIBUTES_IP_ALLOCATION_ID) EntityHolder(com.netflix.titus.common.framework.reconciler.EntityHolder) TaskState(com.netflix.titus.api.jobmanager.model.job.TaskState) HashSet(java.util.HashSet)

Example 13 with EntityHolder

use of com.netflix.titus.common.framework.reconciler.EntityHolder in project titus-control-plane by Netflix.

the class ServiceDifferenceResolver method applyRuntime.

private List<ChangeAction> applyRuntime(ReconciliationEngine<JobManagerReconcilerEvent> engine, ServiceJobView refJobView, EntityHolder runningModel, EntityHolder storeModel, AtomicInteger allowedNewTasks, AtomicInteger allowedTaskKills) {
    EntityHolder referenceModel = refJobView.getJobHolder();
    ServiceJobView runningJobView = new ServiceJobView(runningModel);
    if (hasJobState(referenceModel, JobState.KillInitiated)) {
        List<ChangeAction> killInitiatedActions = KillInitiatedActions.reconcilerInitiatedAllTasksKillInitiated(engine, runtime, jobStore, TaskStatus.REASON_TASK_KILLED, "Killing task as its job is in KillInitiated state", allowedTaskKills.get(), versionSupplier, titusRuntime);
        if (killInitiatedActions.isEmpty()) {
            return findTaskStateTimeouts(engine, runningJobView, configuration, runtime, jobStore, versionSupplier, stuckInStateRateLimiter, titusRuntime);
        }
        allowedTaskKills.set(allowedTaskKills.get() - killInitiatedActions.size());
        return killInitiatedActions;
    } else if (hasJobState(referenceModel, JobState.Finished)) {
        return Collections.emptyList();
    }
    List<ChangeAction> actions = new ArrayList<>();
    List<ChangeAction> numberOfTaskAdjustingActions = findJobSizeInconsistencies(engine, refJobView, storeModel, allowedNewTasks, allowedTaskKills);
    actions.addAll(numberOfTaskAdjustingActions);
    if (numberOfTaskAdjustingActions.isEmpty()) {
        actions.addAll(findMissingRunningTasks(engine, refJobView, runningJobView));
    }
    actions.addAll(findTaskStateTimeouts(engine, runningJobView, configuration, runtime, jobStore, versionSupplier, stuckInStateRateLimiter, titusRuntime));
    return actions;
}
Also used : TitusChangeAction(com.netflix.titus.master.jobmanager.service.common.action.TitusChangeAction) ChangeAction(com.netflix.titus.common.framework.reconciler.ChangeAction) ArrayList(java.util.ArrayList) EntityHolder(com.netflix.titus.common.framework.reconciler.EntityHolder)

Example 14 with EntityHolder

use of com.netflix.titus.common.framework.reconciler.EntityHolder in project titus-control-plane by Netflix.

the class TaskTimeoutChangeActions method setTimeout.

public static TitusChangeAction setTimeout(String taskId, TaskState taskState, long timeoutMs, Clock clock) {
    String tagName = STATE_TAGS.get(taskState);
    Preconditions.checkArgument(tagName != null, "Timeout not tracked for state %s", taskState);
    return TitusChangeAction.newAction("setTimeout").id(taskId).trigger(Trigger.Reconciler).summary("Setting timeout for task in state %s: %s", taskState, DateTimeExt.toTimeUnitString(timeoutMs)).callMetadata(JobManagerConstants.RECONCILER_CALLMETADATA.toBuilder().withCallReason("configure timeout").build()).applyModelUpdate(self -> {
        TitusModelAction modelAction = TitusModelAction.newModelUpdate(self).taskMaybeUpdate(jobHolder -> jobHolder.findById(taskId).map(taskHolder -> {
            EntityHolder newTaskHolder = taskHolder.addTag(tagName, clock.wallTime() + timeoutMs);
            if (taskState == TaskState.KillInitiated) {
                newTaskHolder = newTaskHolder.addTag(KILL_INITIATED_ATTEMPT_TAG, 0);
            }
            return Pair.of(jobHolder.addChild(newTaskHolder), newTaskHolder);
        }));
        return ModelActionHolder.running(modelAction);
    });
}
Also used : Trigger(com.netflix.titus.api.jobmanager.service.V3JobOperations.Trigger) DateTimeExt(com.netflix.titus.common.util.DateTimeExt) JobManagerConstants(com.netflix.titus.api.jobmanager.service.JobManagerConstants) ImmutableMap(com.google.common.collect.ImmutableMap) TitusChangeAction(com.netflix.titus.master.jobmanager.service.common.action.TitusChangeAction) Task(com.netflix.titus.api.jobmanager.model.job.Task) JobFunctions(com.netflix.titus.api.jobmanager.model.job.JobFunctions) TitusModelAction(com.netflix.titus.master.jobmanager.service.common.action.TitusModelAction) TaskState(com.netflix.titus.api.jobmanager.model.job.TaskState) EntityHolder(com.netflix.titus.common.framework.reconciler.EntityHolder) ModelActionHolder(com.netflix.titus.common.framework.reconciler.ModelActionHolder) Pair(com.netflix.titus.common.util.tuple.Pair) Map(java.util.Map) JobManagerConfiguration(com.netflix.titus.master.jobmanager.service.JobManagerConfiguration) Preconditions(com.google.common.base.Preconditions) Clock(com.netflix.titus.common.util.time.Clock) TitusModelAction(com.netflix.titus.master.jobmanager.service.common.action.TitusModelAction) EntityHolder(com.netflix.titus.common.framework.reconciler.EntityHolder)

Example 15 with EntityHolder

use of com.netflix.titus.common.framework.reconciler.EntityHolder in project titus-control-plane by Netflix.

the class RetryActionInterceptorTest method expectAboveExecutionLimits.

private EntityHolder expectAboveExecutionLimits(ModelAction updateAction, EntityHolder model) {
    Optional<Pair<EntityHolder, EntityHolder>> pair = updateAction.apply(model);
    assertThat(pair).isPresent();
    EntityHolder modelWithTag = pair.get().getRight();
    assertThat(retryInterceptor.executionLimits(modelWithTag)).isFalse();
    return modelWithTag;
}
Also used : EntityHolder(com.netflix.titus.common.framework.reconciler.EntityHolder) Pair(com.netflix.titus.common.util.tuple.Pair)

Aggregations

EntityHolder (com.netflix.titus.common.framework.reconciler.EntityHolder)31 ArrayList (java.util.ArrayList)17 Task (com.netflix.titus.api.jobmanager.model.job.Task)12 ChangeAction (com.netflix.titus.common.framework.reconciler.ChangeAction)12 ModelActionHolder (com.netflix.titus.common.framework.reconciler.ModelActionHolder)12 TitusChangeAction (com.netflix.titus.master.jobmanager.service.common.action.TitusChangeAction)12 List (java.util.List)10 TaskState (com.netflix.titus.api.jobmanager.model.job.TaskState)7 TitusModelAction (com.netflix.titus.master.jobmanager.service.common.action.TitusModelAction)7 Job (com.netflix.titus.api.jobmanager.model.job.Job)6 TaskStatus (com.netflix.titus.api.jobmanager.model.job.TaskStatus)6 ServiceJobExt (com.netflix.titus.api.jobmanager.model.job.ext.ServiceJobExt)6 ReconciliationEngine (com.netflix.titus.common.framework.reconciler.ReconciliationEngine)6 JobManagerReconcilerEvent (com.netflix.titus.master.jobmanager.service.event.JobManagerReconcilerEvent)6 Test (org.junit.Test)6 JobFunctions (com.netflix.titus.api.jobmanager.model.job.JobFunctions)5 JobStore (com.netflix.titus.api.jobmanager.store.JobStore)5 CallMetadata (com.netflix.titus.api.model.callmetadata.CallMetadata)5 TitusRuntime (com.netflix.titus.common.runtime.TitusRuntime)5 Optional (java.util.Optional)5