Search in sources :

Example 16 with JobStore

use of com.netflix.titus.api.jobmanager.store.JobStore in project titus-control-plane by Netflix.

the class KillInitiatedActions method userInitiateTaskKillAction.

/**
 * Change a task to {@link TaskState#KillInitiated} state, store it, and send the kill command to the compute provider.
 * All models are updated when both operations complete.
 * This method is used for user initiated kill operations, so the store operation happens before response is sent back to the user.
 */
public static ChangeAction userInitiateTaskKillAction(ReconciliationEngine<JobManagerReconcilerEvent> engine, JobServiceRuntime executionContext, JobStore jobStore, VersionSupplier versionSupplier, String taskId, boolean shrink, boolean preventMinSizeUpdate, String reasonCode, String reason, TitusRuntime titusRuntime, CallMetadata callMetadata) {
    return TitusChangeAction.newAction("userInitiateTaskKill").id(taskId).trigger(V3JobOperations.Trigger.API).summary(reason).callMetadata(callMetadata).changeWithModelUpdates(self -> JobEntityHolders.toTaskObservable(engine, taskId, titusRuntime).flatMap(task -> {
        TaskState taskState = task.getStatus().getState();
        if (taskState == TaskState.KillInitiated || taskState == TaskState.Finished) {
            return Observable.just(Collections.<ModelActionHolder>emptyList());
        }
        if (shrink) {
            Job<ServiceJobExt> job = engine.getReferenceView().getEntity();
            Capacity capacity = job.getJobDescriptor().getExtensions().getCapacity();
            if (preventMinSizeUpdate && capacity.getDesired() <= capacity.getMin()) {
                return Observable.<List<ModelActionHolder>>error(JobManagerException.terminateAndShrinkNotAllowed(job, task));
            }
        }
        Task taskWithKillInitiated = VersionSuppliers.nextVersion(JobFunctions.changeTaskStatus(task, TaskState.KillInitiated, reasonCode, reason, titusRuntime.getClock()), versionSupplier);
        Callable<List<ModelActionHolder>> modelUpdateActions = () -> JobEntityHolders.expectTask(engine, task.getId(), titusRuntime).map(current -> {
            List<ModelActionHolder> updateActions = new ArrayList<>();
            TitusModelAction stateUpdateAction = TitusModelAction.newModelUpdate(self).taskUpdate(taskWithKillInitiated);
            updateActions.addAll(ModelActionHolder.allModels(stateUpdateAction));
            if (shrink) {
                TitusModelAction shrinkAction = createShrinkAction(self, versionSupplier);
                updateActions.add(ModelActionHolder.reference(shrinkAction));
            }
            return updateActions;
        }).orElse(Collections.emptyList());
        return jobStore.updateTask(taskWithKillInitiated).andThen(createKillAction(executionContext, task)).andThen(Observable.fromCallable(modelUpdateActions));
    }));
}
Also used : Completable(rx.Completable) JobManagerConstants(com.netflix.titus.api.jobmanager.service.JobManagerConstants) JobServiceRuntime(com.netflix.titus.master.jobmanager.service.JobServiceRuntime) TitusChangeAction(com.netflix.titus.master.jobmanager.service.common.action.TitusChangeAction) Task(com.netflix.titus.api.jobmanager.model.job.Task) Callable(java.util.concurrent.Callable) ReactorExt(com.netflix.titus.common.util.rx.ReactorExt) ArrayList(java.util.ArrayList) Observable(rx.Observable) HashSet(java.util.HashSet) JobStatus(com.netflix.titus.api.jobmanager.model.job.JobStatus) JobState(com.netflix.titus.api.jobmanager.model.job.JobState) ChangeAction(com.netflix.titus.common.framework.reconciler.ChangeAction) JobManagerException(com.netflix.titus.api.jobmanager.service.JobManagerException) JobEntityHolders(com.netflix.titus.master.jobmanager.service.common.action.JobEntityHolders) JobStore(com.netflix.titus.api.jobmanager.store.JobStore) CallMetadata(com.netflix.titus.api.model.callmetadata.CallMetadata) Job(com.netflix.titus.api.jobmanager.model.job.Job) ServiceJobExt(com.netflix.titus.api.jobmanager.model.job.ext.ServiceJobExt) TaskStatus(com.netflix.titus.api.jobmanager.model.job.TaskStatus) Set(java.util.Set) JobFunctions(com.netflix.titus.api.jobmanager.model.job.JobFunctions) TitusModelAction(com.netflix.titus.master.jobmanager.service.common.action.TitusModelAction) TaskState(com.netflix.titus.api.jobmanager.model.job.TaskState) Capacity(com.netflix.titus.api.jobmanager.model.job.Capacity) EntityHolder(com.netflix.titus.common.framework.reconciler.EntityHolder) ModelActionHolder(com.netflix.titus.common.framework.reconciler.ModelActionHolder) List(java.util.List) V3JobOperations(com.netflix.titus.api.jobmanager.service.V3JobOperations) VersionSupplier(com.netflix.titus.master.jobmanager.service.VersionSupplier) ReconciliationEngine(com.netflix.titus.common.framework.reconciler.ReconciliationEngine) VersionSuppliers(com.netflix.titus.master.jobmanager.service.VersionSuppliers) Optional(java.util.Optional) JobManagerReconcilerEvent(com.netflix.titus.master.jobmanager.service.event.JobManagerReconcilerEvent) TitusRuntime(com.netflix.titus.common.runtime.TitusRuntime) Collections(java.util.Collections) TitusModelAction(com.netflix.titus.master.jobmanager.service.common.action.TitusModelAction) Task(com.netflix.titus.api.jobmanager.model.job.Task) Capacity(com.netflix.titus.api.jobmanager.model.job.Capacity) ArrayList(java.util.ArrayList) Job(com.netflix.titus.api.jobmanager.model.job.Job) TaskState(com.netflix.titus.api.jobmanager.model.job.TaskState) Callable(java.util.concurrent.Callable) ModelActionHolder(com.netflix.titus.common.framework.reconciler.ModelActionHolder)

Example 17 with JobStore

use of com.netflix.titus.api.jobmanager.store.JobStore in project titus-control-plane by Netflix.

the class DifferenceResolverUtils method findTaskStateTimeouts.

/**
 * Find all tasks that are stuck in a specific state. The number of {@link ChangeAction changes} will be limited
 * by the {@link TokenBucket stuckInStateRateLimiter}
 */
public static List<ChangeAction> findTaskStateTimeouts(ReconciliationEngine<JobManagerReconcilerEvent> engine, JobView runningJobView, JobManagerConfiguration configuration, JobServiceRuntime runtime, JobStore jobStore, VersionSupplier versionSupplier, TokenBucket stuckInStateRateLimiter, TitusRuntime titusRuntime) {
    Clock clock = titusRuntime.getClock();
    List<ChangeAction> actions = new ArrayList<>();
    runningJobView.getJobHolder().getChildren().forEach(taskHolder -> {
        Task task = taskHolder.getEntity();
        TaskState taskState = task.getStatus().getState();
        if (JobFunctions.isBatchJob(runningJobView.getJob()) && taskState == TaskState.Started) {
            Job<BatchJobExt> batchJob = runningJobView.getJob();
            // We expect runtime limit to be always set, so this is just extra safety measure.
            long runtimeLimitMs = Math.max(BatchJobExt.RUNTIME_LIMIT_MIN, batchJob.getJobDescriptor().getExtensions().getRuntimeLimitMs());
            long deadline = task.getStatus().getTimestamp() + runtimeLimitMs;
            if (deadline < clock.wallTime()) {
                actions.add(KillInitiatedActions.reconcilerInitiatedTaskKillInitiated(engine, task, runtime, jobStore, versionSupplier, TaskStatus.REASON_RUNTIME_LIMIT_EXCEEDED, "Task running too long (runtimeLimit=" + runtimeLimitMs + "ms)", titusRuntime));
            }
            return;
        }
        TaskTimeoutChangeActions.TimeoutStatus timeoutStatus = TaskTimeoutChangeActions.getTimeoutStatus(taskHolder, clock);
        switch(timeoutStatus) {
            case Ignore:
            case Pending:
                break;
            case NotSet:
                long timeoutMs = -1;
                switch(taskState) {
                    case Launched:
                        timeoutMs = configuration.getTaskInLaunchedStateTimeoutMs();
                        break;
                    case StartInitiated:
                        timeoutMs = isBatch(runningJobView.getJob()) ? configuration.getBatchTaskInStartInitiatedStateTimeoutMs() : configuration.getServiceTaskInStartInitiatedStateTimeoutMs();
                        break;
                    case KillInitiated:
                        timeoutMs = configuration.getTaskInKillInitiatedStateTimeoutMs();
                        break;
                }
                if (timeoutMs > 0) {
                    actions.add(TaskTimeoutChangeActions.setTimeout(taskHolder.getId(), task.getStatus().getState(), timeoutMs, clock));
                }
                break;
            case TimedOut:
                if (!stuckInStateRateLimiter.tryTake()) {
                    break;
                }
                if (task.getStatus().getState() == TaskState.KillInitiated) {
                    int attempts = TaskTimeoutChangeActions.getKillInitiatedAttempts(taskHolder) + 1;
                    if (attempts >= configuration.getTaskKillAttempts()) {
                        actions.add(BasicTaskActions.updateTaskInRunningModel(task.getId(), V3JobOperations.Trigger.Reconciler, configuration, engine, taskParam -> Optional.of(taskParam.toBuilder().withStatus(taskParam.getStatus().toBuilder().withState(TaskState.Finished).withReasonCode(TaskStatus.REASON_STUCK_IN_KILLING_STATE).withReasonMessage("stuck in " + taskState + "state").build()).build()), "TimedOut in KillInitiated state", versionSupplier, titusRuntime, JobManagerConstants.RECONCILER_CALLMETADATA.toBuilder().withCallReason("Kill initiated").build()));
                    } else {
                        actions.add(TaskTimeoutChangeActions.incrementTaskKillAttempt(task.getId(), configuration.getTaskInKillInitiatedStateTimeoutMs(), clock));
                        actions.add(KillInitiatedActions.reconcilerInitiatedTaskKillInitiated(engine, task, runtime, jobStore, versionSupplier, TaskStatus.REASON_STUCK_IN_KILLING_STATE, "Another kill attempt (" + (attempts + 1) + ')', titusRuntime));
                    }
                } else {
                    actions.add(KillInitiatedActions.reconcilerInitiatedTaskKillInitiated(engine, task, runtime, jobStore, versionSupplier, TaskStatus.REASON_STUCK_IN_STATE, "Task stuck in " + taskState + " state", titusRuntime));
                }
                break;
        }
    });
    return actions;
}
Also used : JobManagerConstants(com.netflix.titus.api.jobmanager.service.JobManagerConstants) JobServiceRuntime(com.netflix.titus.master.jobmanager.service.JobServiceRuntime) Task(com.netflix.titus.api.jobmanager.model.job.Task) HashMap(java.util.HashMap) Function(java.util.function.Function) TaskTimeoutChangeActions(com.netflix.titus.master.jobmanager.service.common.action.task.TaskTimeoutChangeActions) ArrayList(java.util.ArrayList) EbsVolume(com.netflix.titus.api.jobmanager.model.job.ebs.EbsVolume) TASK_ATTRIBUTES_EBS_VOLUME_ID(com.netflix.titus.api.jobmanager.TaskAttributes.TASK_ATTRIBUTES_EBS_VOLUME_ID) HashSet(java.util.HashSet) Map(java.util.Map) JobState(com.netflix.titus.api.jobmanager.model.job.JobState) BatchJobExt(com.netflix.titus.api.jobmanager.model.job.ext.BatchJobExt) ChangeAction(com.netflix.titus.common.framework.reconciler.ChangeAction) JobManagerConfiguration(com.netflix.titus.master.jobmanager.service.JobManagerConfiguration) JobStore(com.netflix.titus.api.jobmanager.store.JobStore) JobDescriptor(com.netflix.titus.api.jobmanager.model.job.JobDescriptor) Job(com.netflix.titus.api.jobmanager.model.job.Job) ServiceJobExt(com.netflix.titus.api.jobmanager.model.job.ext.ServiceJobExt) TaskStatus(com.netflix.titus.api.jobmanager.model.job.TaskStatus) Set(java.util.Set) JobFunctions(com.netflix.titus.api.jobmanager.model.job.JobFunctions) Collectors(java.util.stream.Collectors) TaskState(com.netflix.titus.api.jobmanager.model.job.TaskState) EntityHolder(com.netflix.titus.common.framework.reconciler.EntityHolder) Consumer(java.util.function.Consumer) List(java.util.List) ExecutableStatus(com.netflix.titus.api.jobmanager.model.job.ExecutableStatus) V3JobOperations(com.netflix.titus.api.jobmanager.service.V3JobOperations) VersionSupplier(com.netflix.titus.master.jobmanager.service.VersionSupplier) ReconciliationEngine(com.netflix.titus.common.framework.reconciler.ReconciliationEngine) Optional(java.util.Optional) BasicTaskActions(com.netflix.titus.master.jobmanager.service.common.action.task.BasicTaskActions) JobManagerReconcilerEvent(com.netflix.titus.master.jobmanager.service.event.JobManagerReconcilerEvent) TitusRuntime(com.netflix.titus.common.runtime.TitusRuntime) TokenBucket(com.netflix.titus.common.util.limiter.tokenbucket.TokenBucket) Clock(com.netflix.titus.common.util.time.Clock) KillInitiatedActions(com.netflix.titus.master.jobmanager.service.common.action.task.KillInitiatedActions) TASK_ATTRIBUTES_IP_ALLOCATION_ID(com.netflix.titus.api.jobmanager.TaskAttributes.TASK_ATTRIBUTES_IP_ALLOCATION_ID) Task(com.netflix.titus.api.jobmanager.model.job.Task) ChangeAction(com.netflix.titus.common.framework.reconciler.ChangeAction) BatchJobExt(com.netflix.titus.api.jobmanager.model.job.ext.BatchJobExt) ArrayList(java.util.ArrayList) TaskTimeoutChangeActions(com.netflix.titus.master.jobmanager.service.common.action.task.TaskTimeoutChangeActions) Clock(com.netflix.titus.common.util.time.Clock) TaskState(com.netflix.titus.api.jobmanager.model.job.TaskState)

Example 18 with JobStore

use of com.netflix.titus.api.jobmanager.store.JobStore in project titus-control-plane by Netflix.

the class EmbeddedTitusMaster method boot.

public EmbeddedTitusMaster boot() {
    Stopwatch timer = Stopwatch.createStarted();
    logger.info("Starting Titus Master");
    Module embeddedKubeModule;
    if (embeddedKubeCluster == null) {
        embeddedKubeModule = new AbstractModule() {

            @Override
            protected void configure() {
            }
        };
    } else {
        embeddedKubeModule = new EmbeddedKubeModule(embeddedKubeCluster);
    }
    injector = InjectorBuilder.fromModules(Modules.override(new TitusRuntimeModule(false)).with(new AbstractModule() {

        @Override
        protected void configure() {
            bind(Archaius2ConfigurationLogger.class).asEagerSingleton();
            bind(Registry.class).toInstance(new DefaultRegistry());
        }
    }), embeddedKubeModule, Modules.override(new TitusMasterModule(enableREST, TitusMasterModule.Mode.EMBEDDED_KUBE)).with(new AbstractModule() {

        @Override
        protected void configure() {
            bind(InstanceCloudConnector.class).toInstance(new NoOpInstanceCloudConnector());
            bind(MasterDescription.class).toInstance(masterDescription);
            bind(MasterMonitor.class).to(LocalMasterMonitor.class);
            bind(AppScalePolicyStore.class).to(InMemoryPolicyStore.class);
            bind(LoadBalancerStore.class).to(InMemoryLoadBalancerStore.class);
            bind(LoadBalancerConnector.class).to(NoOpLoadBalancerConnector.class);
            bind(LoadBalancerJobValidator.class).to(NoOpLoadBalancerJobValidator.class);
        }

        @Provides
        @Singleton
        public JobStore getJobStore(TitusRuntime titusRuntime) {
            if (!cassandraJobStore) {
                return jobStore;
            }
            try {
                JobStore jobStore = EmbeddedCassandraStoreFactory.newBuilder().withTitusRuntime(titusRuntime).build().getJobStore();
                return jobStore;
            } catch (Throwable e) {
                e.printStackTrace();
                return null;
            }
        }
    }), newJettyModule(), new ArchaiusModule() {

        @Override
        protected void configureArchaius() {
            bindApplicationConfigurationOverride().toInstance(config);
        }
    }).createInjector();
    if (grpcPort <= 0) {
        grpcPort = getGrpcPort();
        config.setProperty("titus.master.grpcServer.port", "" + grpcPort);
    }
    injector.getInstance(ContainerEventBus.class).submitInOrder(new ContainerEventBus.ContainerStartedEvent());
    injector.getInstance(LeaderActivator.class).becomeLeader();
    injector.getInstance(AuditLogService.class).auditLogEvents().subscribe(auditLogs::add);
    if (enableREST) {
        // Since jetty API server is run on a separate thread, it may not be ready yet
        // We do not have better way, but call it until it replies.
        getClient().findAllApplicationSLA().retryWhen(attempts -> {
            return attempts.zipWith(Observable.range(1, 5), (n, i) -> i).flatMap(i -> {
                return Observable.timer(i, TimeUnit.SECONDS);
            });
        }).timeout(30, TimeUnit.SECONDS).toBlocking().firstOrDefault(null);
    }
    logger.info("Embedded TitusMaster started in {}ms", timer.elapsed(TimeUnit.MILLISECONDS));
    return this;
}
Also used : Module(com.google.inject.Module) AuditLogEvent(com.netflix.titus.api.audit.model.AuditLogEvent) LoadBalancerConnector(com.netflix.titus.api.connector.cloud.LoadBalancerConnector) LocalMasterMonitor(com.netflix.titus.master.supervisor.service.leader.LocalMasterMonitor) InjectorBuilder(com.netflix.governator.InjectorBuilder) ManagedChannel(io.grpc.ManagedChannel) EmbeddedKubeCluster(com.netflix.titus.testkit.embedded.kube.EmbeddedKubeCluster) Key(com.google.inject.Key) LoggerFactory(org.slf4j.LoggerFactory) NoOpLoadBalancerConnector(com.netflix.titus.api.connector.cloud.noop.NoOpLoadBalancerConnector) MasterDescription(com.netflix.titus.api.supervisor.service.MasterDescription) NoOpLoadBalancerJobValidator(com.netflix.titus.api.loadbalancer.model.sanitizer.NoOpLoadBalancerJobValidator) TitusMaster(com.netflix.titus.master.TitusMaster) TitusRuntimeModule(com.netflix.titus.master.TitusRuntimeModule) JobStore(com.netflix.titus.api.jobmanager.store.JobStore) LoadBalancerStore(com.netflix.titus.api.loadbalancer.store.LoadBalancerStore) AppScalePolicyStore(com.netflix.titus.api.appscale.store.AppScalePolicyStore) InMemoryJobStore(com.netflix.titus.runtime.store.v3.memory.InMemoryJobStore) MasterMonitor(com.netflix.titus.api.supervisor.service.MasterMonitor) ContainerEventBus(com.netflix.titus.common.util.guice.ContainerEventBus) ReactorTitusMasterClient(com.netflix.titus.testkit.client.ReactorTitusMasterClient) SupervisorServiceGrpc(com.netflix.titus.grpc.protogen.SupervisorServiceGrpc) ArchaiusSystemDisruptionBudgetResolver(com.netflix.titus.master.eviction.service.quota.system.ArchaiusSystemDisruptionBudgetResolver) InMemoryPolicyStore(com.netflix.titus.runtime.store.v3.memory.InMemoryPolicyStore) List(java.util.List) MachineServiceGrpc(com.netflix.titus.grpc.protogen.v4.MachineServiceGrpc) LifecycleInjector(com.netflix.governator.LifecycleInjector) SchedulerServiceGrpc(com.netflix.titus.grpc.protogen.SchedulerServiceGrpc) EmbeddedJettyModule(com.netflix.titus.runtime.endpoint.common.rest.EmbeddedJettyModule) EmbeddedCassandraStoreFactory(com.netflix.titus.ext.cassandra.testkit.store.EmbeddedCassandraStoreFactory) SupervisorServiceBlockingStub(com.netflix.titus.grpc.protogen.SupervisorServiceGrpc.SupervisorServiceBlockingStub) JobManagementServiceStub(com.netflix.titus.grpc.protogen.JobManagementServiceGrpc.JobManagementServiceStub) JettyModule(com.netflix.governator.guice.jetty.JettyModule) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) LeaderActivator(com.netflix.titus.api.supervisor.service.LeaderActivator) Stopwatch(com.google.common.base.Stopwatch) Modules(com.google.inject.util.Modules) LoadBalancerJobValidator(com.netflix.titus.api.loadbalancer.model.sanitizer.LoadBalancerJobValidator) HealthStub(com.netflix.titus.grpc.protogen.HealthGrpc.HealthStub) NoOpInstanceCloudConnector(com.netflix.titus.api.connector.cloud.noop.NoOpInstanceCloudConnector) Singleton(javax.inject.Singleton) TitusMasterGrpcServer(com.netflix.titus.master.endpoint.grpc.TitusMasterGrpcServer) Observable(rx.Observable) JobActivityHistoryServiceGrpc(com.netflix.titus.grpc.protogen.JobActivityHistoryServiceGrpc) EvictionServiceGrpc(com.netflix.titus.grpc.protogen.EvictionServiceGrpc) HealthGrpc(com.netflix.titus.grpc.protogen.HealthGrpc) InstanceCloudConnector(com.netflix.titus.api.connector.cloud.InstanceCloudConnector) AutoScalingServiceGrpc(com.netflix.titus.grpc.protogen.AutoScalingServiceGrpc) TitusMasterModule(com.netflix.titus.master.TitusMasterModule) ArchaiusModule(com.netflix.archaius.guice.ArchaiusModule) EmbeddedKubeModule(com.netflix.titus.testkit.embedded.kube.EmbeddedKubeModule) Properties(java.util.Properties) Logger(org.slf4j.Logger) TestKitGrpcClientErrorUtils(com.netflix.titus.testkit.grpc.TestKitGrpcClientErrorUtils) DefaultSettableConfig(com.netflix.archaius.config.DefaultSettableConfig) JobManagementServiceGrpc(com.netflix.titus.grpc.protogen.JobManagementServiceGrpc) TitusMasterClient(com.netflix.titus.testkit.client.TitusMasterClient) Archaius2ConfigurationLogger(com.netflix.titus.common.util.archaius2.Archaius2ConfigurationLogger) TimeUnit(java.util.concurrent.TimeUnit) ManagedChannelBuilder(io.grpc.ManagedChannelBuilder) ObjectMappers(com.netflix.titus.api.json.ObjectMappers) Provides(com.google.inject.Provides) LoadBalancerServiceGrpc(com.netflix.titus.grpc.protogen.LoadBalancerServiceGrpc) DefaultRegistry(com.netflix.spectator.api.DefaultRegistry) JobManagementServiceBlockingStub(com.netflix.titus.grpc.protogen.JobManagementServiceGrpc.JobManagementServiceBlockingStub) InMemoryLoadBalancerStore(com.netflix.titus.runtime.store.v3.memory.InMemoryLoadBalancerStore) Registry(com.netflix.spectator.api.Registry) Preconditions(com.google.common.base.Preconditions) TitusRuntime(com.netflix.titus.common.runtime.TitusRuntime) SystemDisruptionBudgetDescriptor(com.netflix.titus.master.eviction.service.quota.system.SystemDisruptionBudgetDescriptor) AbstractModule(com.google.inject.AbstractModule) AuditLogService(com.netflix.titus.api.audit.service.AuditLogService) AppScalePolicyStore(com.netflix.titus.api.appscale.store.AppScalePolicyStore) MasterDescription(com.netflix.titus.api.supervisor.service.MasterDescription) Stopwatch(com.google.common.base.Stopwatch) TitusRuntime(com.netflix.titus.common.runtime.TitusRuntime) EmbeddedKubeModule(com.netflix.titus.testkit.embedded.kube.EmbeddedKubeModule) NoOpLoadBalancerJobValidator(com.netflix.titus.api.loadbalancer.model.sanitizer.NoOpLoadBalancerJobValidator) LoadBalancerJobValidator(com.netflix.titus.api.loadbalancer.model.sanitizer.LoadBalancerJobValidator) NoOpInstanceCloudConnector(com.netflix.titus.api.connector.cloud.noop.NoOpInstanceCloudConnector) InstanceCloudConnector(com.netflix.titus.api.connector.cloud.InstanceCloudConnector) LeaderActivator(com.netflix.titus.api.supervisor.service.LeaderActivator) LocalMasterMonitor(com.netflix.titus.master.supervisor.service.leader.LocalMasterMonitor) MasterMonitor(com.netflix.titus.api.supervisor.service.MasterMonitor) LoadBalancerConnector(com.netflix.titus.api.connector.cloud.LoadBalancerConnector) NoOpLoadBalancerConnector(com.netflix.titus.api.connector.cloud.noop.NoOpLoadBalancerConnector) NoOpInstanceCloudConnector(com.netflix.titus.api.connector.cloud.noop.NoOpInstanceCloudConnector) Archaius2ConfigurationLogger(com.netflix.titus.common.util.archaius2.Archaius2ConfigurationLogger) ContainerEventBus(com.netflix.titus.common.util.guice.ContainerEventBus) JobStore(com.netflix.titus.api.jobmanager.store.JobStore) InMemoryJobStore(com.netflix.titus.runtime.store.v3.memory.InMemoryJobStore) DefaultRegistry(com.netflix.spectator.api.DefaultRegistry) Registry(com.netflix.spectator.api.Registry) AbstractModule(com.google.inject.AbstractModule) TitusMasterModule(com.netflix.titus.master.TitusMasterModule) ArchaiusModule(com.netflix.archaius.guice.ArchaiusModule) DefaultRegistry(com.netflix.spectator.api.DefaultRegistry) Module(com.google.inject.Module) TitusRuntimeModule(com.netflix.titus.master.TitusRuntimeModule) EmbeddedJettyModule(com.netflix.titus.runtime.endpoint.common.rest.EmbeddedJettyModule) JettyModule(com.netflix.governator.guice.jetty.JettyModule) TitusMasterModule(com.netflix.titus.master.TitusMasterModule) ArchaiusModule(com.netflix.archaius.guice.ArchaiusModule) EmbeddedKubeModule(com.netflix.titus.testkit.embedded.kube.EmbeddedKubeModule) AbstractModule(com.google.inject.AbstractModule) LoadBalancerStore(com.netflix.titus.api.loadbalancer.store.LoadBalancerStore) InMemoryLoadBalancerStore(com.netflix.titus.runtime.store.v3.memory.InMemoryLoadBalancerStore) TitusRuntimeModule(com.netflix.titus.master.TitusRuntimeModule)

Example 19 with JobStore

use of com.netflix.titus.api.jobmanager.store.JobStore in project titus-control-plane by Netflix.

the class CassandraJobStoreTest method testUpdateTask.

@Test
public void testUpdateTask() {
    JobStore store = getJobStore();
    Job<BatchJobExt> job = createBatchJobObject();
    store.init().await();
    store.storeJob(job).await();
    Pair<List<Job<?>>, Integer> jobsAndErrors = store.retrieveJobs().toBlocking().first();
    checkRetrievedJob(job, jobsAndErrors.getLeft().get(0));
    Task task = createTaskObject(job);
    store.storeTask(task).await();
    Task retrievedTask = store.retrieveTask(task.getId()).toBlocking().first();
    checkRetrievedTask(task, retrievedTask);
    BatchJobTask newTask = BatchJobTask.newBuilder((BatchJobTask) task).withStatus(TaskStatus.newBuilder().withState(TaskState.Finished).build()).build();
    store.updateTask(newTask).await();
    Task newRetrievedTask = store.retrieveTask(newTask.getId()).toBlocking().first();
    checkRetrievedTask(newTask, newRetrievedTask);
}
Also used : BatchJobTask(com.netflix.titus.api.jobmanager.model.job.BatchJobTask) Task(com.netflix.titus.api.jobmanager.model.job.Task) ServiceJobTask(com.netflix.titus.api.jobmanager.model.job.ServiceJobTask) BatchJobExt(com.netflix.titus.api.jobmanager.model.job.ext.BatchJobExt) BatchJobTask(com.netflix.titus.api.jobmanager.model.job.BatchJobTask) JobStore(com.netflix.titus.api.jobmanager.store.JobStore) ArrayList(java.util.ArrayList) List(java.util.List) Test(org.junit.Test) IntegrationNotParallelizableTest(com.netflix.titus.testkit.junit.category.IntegrationNotParallelizableTest)

Example 20 with JobStore

use of com.netflix.titus.api.jobmanager.store.JobStore in project titus-control-plane by Netflix.

the class CassandraJobStoreTest method testUpdateJob.

@Test
public void testUpdateJob() {
    JobStore store = getJobStore();
    Job<BatchJobExt> job = createBatchJobObject();
    store.init().await();
    store.storeJob(job).await();
    Pair<List<Job<?>>, Integer> jobsAndErrors = store.retrieveJobs().toBlocking().first();
    checkRetrievedJob(job, jobsAndErrors.getLeft().get(0));
    Job<BatchJobExt> newJob = job.toBuilder().withStatus(JobStatus.newBuilder().withState(JobState.Finished).build()).build();
    store.updateJob(newJob).await();
    Pair<List<Job<?>>, Integer> newJobsAndErrors = store.retrieveJobs().toBlocking().first();
    checkRetrievedJob(newJob, newJobsAndErrors.getLeft().get(0));
}
Also used : BatchJobExt(com.netflix.titus.api.jobmanager.model.job.ext.BatchJobExt) JobStore(com.netflix.titus.api.jobmanager.store.JobStore) ArrayList(java.util.ArrayList) List(java.util.List) Test(org.junit.Test) IntegrationNotParallelizableTest(com.netflix.titus.testkit.junit.category.IntegrationNotParallelizableTest)

Aggregations

JobStore (com.netflix.titus.api.jobmanager.store.JobStore)25 List (java.util.List)24 ArrayList (java.util.ArrayList)23 BatchJobExt (com.netflix.titus.api.jobmanager.model.job.ext.BatchJobExt)18 Task (com.netflix.titus.api.jobmanager.model.job.Task)17 IntegrationNotParallelizableTest (com.netflix.titus.testkit.junit.category.IntegrationNotParallelizableTest)14 Test (org.junit.Test)14 BatchJobTask (com.netflix.titus.api.jobmanager.model.job.BatchJobTask)13 ServiceJobTask (com.netflix.titus.api.jobmanager.model.job.ServiceJobTask)11 Job (com.netflix.titus.api.jobmanager.model.job.Job)8 TitusRuntime (com.netflix.titus.common.runtime.TitusRuntime)6 TaskState (com.netflix.titus.api.jobmanager.model.job.TaskState)5 TaskStatus (com.netflix.titus.api.jobmanager.model.job.TaskStatus)5 EntityHolder (com.netflix.titus.common.framework.reconciler.EntityHolder)5 ReconciliationEngine (com.netflix.titus.common.framework.reconciler.ReconciliationEngine)5 JobServiceRuntime (com.netflix.titus.master.jobmanager.service.JobServiceRuntime)5 VersionSupplier (com.netflix.titus.master.jobmanager.service.VersionSupplier)5 JobFunctions (com.netflix.titus.api.jobmanager.model.job.JobFunctions)4 JobState (com.netflix.titus.api.jobmanager.model.job.JobState)4 ServiceJobExt (com.netflix.titus.api.jobmanager.model.job.ext.ServiceJobExt)4