Search in sources :

Example 1 with Either

use of com.netflix.titus.common.util.tuple.Either in project titus-control-plane by Netflix.

the class AwsIamConnector method canAgentAssume.

@Override
public Mono<Void> canAgentAssume(String iamRoleName) {
    return Mono.defer(() -> {
        long startTime = registry.clock().wallTime();
        // Check cache first
        Either<Boolean, Throwable> lastCheck = canAssumeCache.getIfPresent(iamRoleName);
        if (lastCheck != null) {
            return lastCheck.hasValue() ? Mono.empty() : Mono.error(lastCheck.getError());
        }
        // Must call AWS STS service
        return AwsReactorExt.<AssumeRoleRequest, AssumeRoleResult>toMono(() -> new AssumeRoleRequest().withRoleSessionName("titusIamRoleValidation").withRoleArn(iamRoleName).withDurationSeconds(MIN_ASSUMED_ROLE_DURATION_SEC), stsAgentClient::assumeRoleAsync).flatMap(response -> {
            logger.debug("Assumed into: {}", iamRoleName);
            canAssumeCache.put(iamRoleName, Either.ofValue(true));
            connectorMetrics.success(IamConnectorMetrics.IamMethods.CanAgentAssume, startTime);
            return Mono.<Void>empty();
        }).onErrorMap(error -> {
            logger.debug("Error: {}", error.getMessage());
            connectorMetrics.failure(IamConnectorMetrics.IamMethods.CanAgentAssume, error, startTime);
            String errorCode = ((AWSSecurityTokenServiceException) error).getErrorCode();
            if ("AccessDenied".equals(errorCode)) {
                // STS service returns access denied error with no additional clues. To get more insight we
                // would have to make a call to IAM service, but this would require access to all client accounts.
                IamConnectorException cannotAssumeError = IamConnectorException.iamRoleCannotAssume(iamRoleName, configuration.getDataPlaneAgentRoleArn());
                canAssumeCache.put(iamRoleName, Either.ofError(cannotAssumeError));
                return cannotAssumeError;
            }
            return IamConnectorException.iamRoleUnexpectedError(iamRoleName, error.getMessage());
        });
    });
}
Also used : AWSSecurityTokenServiceAsync(com.amazonaws.services.securitytoken.AWSSecurityTokenServiceAsync) Cache(com.netflix.titus.common.util.cache.Cache) ProxyType(com.netflix.titus.common.util.guice.ProxyType) LoggerFactory(org.slf4j.LoggerFactory) IamRole(com.netflix.titus.api.iam.model.IamRole) Singleton(javax.inject.Singleton) AwsReactorExt(com.netflix.titus.ext.aws.AwsReactorExt) Inject(javax.inject.Inject) PreDestroy(javax.annotation.PreDestroy) AssumeRoleRequest(com.amazonaws.services.securitytoken.model.AssumeRoleRequest) AmazonIdentityManagementAsync(com.amazonaws.services.identitymanagement.AmazonIdentityManagementAsync) AWSSecurityTokenServiceException(com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException) Duration(java.time.Duration) Named(javax.inject.Named) Either(com.netflix.titus.common.util.tuple.Either) Logger(org.slf4j.Logger) IamConnectorException(com.netflix.titus.api.iam.service.IamConnectorException) Mono(reactor.core.publisher.Mono) IamConnectorMetrics(com.netflix.titus.common.util.spectator.IamConnectorMetrics) GetRoleResult(com.amazonaws.services.identitymanagement.model.GetRoleResult) NoSuchEntityException(com.amazonaws.services.identitymanagement.model.NoSuchEntityException) IamConnector(com.netflix.titus.api.connector.cloud.IamConnector) AwsConfiguration(com.netflix.titus.ext.aws.AwsConfiguration) AssumeRoleResult(com.amazonaws.services.securitytoken.model.AssumeRoleResult) Registry(com.netflix.spectator.api.Registry) ProxyConfiguration(com.netflix.titus.common.util.guice.annotation.ProxyConfiguration) GetRoleRequest(com.amazonaws.services.identitymanagement.model.GetRoleRequest) Caches(com.netflix.titus.common.util.cache.Caches) AssumeRoleRequest(com.amazonaws.services.securitytoken.model.AssumeRoleRequest) AWSSecurityTokenServiceException(com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException) IamConnectorException(com.netflix.titus.api.iam.service.IamConnectorException) AssumeRoleResult(com.amazonaws.services.securitytoken.model.AssumeRoleResult)

Example 2 with Either

use of com.netflix.titus.common.util.tuple.Either in project titus-control-plane by Netflix.

the class MapWithStateTransformer method call.

@Override
public Observable<R> call(Observable<T> source) {
    return Observable.unsafeCreate(subscriber -> {
        AtomicReference<S> lastState = new AtomicReference<>(zeroSupplier.get());
        Observable<Either<T, Function<S, Pair<R, S>>>> sourceEither = source.map(Either::ofValue);
        Observable<Either<T, Function<S, Pair<R, S>>>> cleanupEither = cleanupActions.map(Either::ofError);
        Subscription subscription = Observable.merge(sourceEither, cleanupEither).subscribe(next -> {
            Pair<R, S> result;
            if (next.hasValue()) {
                try {
                    result = transformer.apply(next.getValue(), lastState.get());
                } catch (Throwable e) {
                    subscriber.onError(e);
                    return;
                }
            } else {
                try {
                    Function<S, Pair<R, S>> action = next.getError();
                    result = action.apply(lastState.get());
                } catch (Throwable e) {
                    subscriber.onError(e);
                    return;
                }
            }
            lastState.set(result.getRight());
            subscriber.onNext(result.getLeft());
        }, subscriber::onError, subscriber::onCompleted);
        subscriber.add(subscription);
    });
}
Also used : Either(com.netflix.titus.common.util.tuple.Either) AtomicReference(java.util.concurrent.atomic.AtomicReference) Subscription(rx.Subscription) Pair(com.netflix.titus.common.util.tuple.Pair)

Example 3 with Either

use of com.netflix.titus.common.util.tuple.Either in project titus-control-plane by Netflix.

the class CellWebClientConnectorUtil method doGetFromCell.

/**
 * Run GET operation on all cells, but only one is expected to return a result.
 */
public static <T> Either<T, WebApplicationException> doGetFromCell(CellWebClientConnector cellWebClientConnector, String path, ParameterizedTypeReference<T> type, long timeoutMs) {
    List<Either<T, Throwable>> partials;
    try {
        partials = Flux.merge(cellWebClientConnector.getWebClients().values().stream().map(cell -> cell.get().uri(path).retrieve().bodyToMono(type).map(Either::<T, Throwable>ofValue).onErrorResume(e -> Mono.just(Either.ofError(e)))).collect(Collectors.toList())).collectList().block(Duration.ofMillis(timeoutMs));
    } catch (Exception e) {
        logger.error("Unexpected error: path={}, type={}", path, type, e);
        return Either.ofError(new WebApplicationException(e, Response.Status.INTERNAL_SERVER_ERROR));
    }
    if (CollectionsExt.isNullOrEmpty(partials)) {
        logger.error("No result from any cell: path={}, type={}", path, type);
        throw new WebApplicationException(Response.Status.NOT_FOUND);
    }
    Throwable systemError = null;
    for (Either<T, Throwable> partial : partials) {
        if (partial.hasValue()) {
            return Either.ofValue(partial.getValue());
        }
        if (partial.getError() instanceof WebClientResponseException) {
            WebClientResponseException wcError = (WebClientResponseException) partial.getError();
            if (!wcError.getStatusCode().equals(HttpStatus.NOT_FOUND)) {
                systemError = partial.getError();
            }
        } else {
            systemError = partial.getError();
        }
    }
    // the client.
    return Either.ofError(systemError != null ? new WebApplicationException(systemError) : new WebApplicationException(Response.Status.NOT_FOUND));
}
Also used : Logger(org.slf4j.Logger) ParameterizedTypeReference(org.springframework.core.ParameterizedTypeReference) CollectionsExt(com.netflix.titus.common.util.CollectionsExt) LoggerFactory(org.slf4j.LoggerFactory) Mono(reactor.core.publisher.Mono) Collectors(java.util.stream.Collectors) ArrayList(java.util.ArrayList) HttpStatus(org.springframework.http.HttpStatus) Flux(reactor.core.publisher.Flux) List(java.util.List) Response(javax.ws.rs.core.Response) Duration(java.time.Duration) WebApplicationException(javax.ws.rs.WebApplicationException) WebClientResponseException(org.springframework.web.reactive.function.client.WebClientResponseException) Either(com.netflix.titus.common.util.tuple.Either) Collections(java.util.Collections) WebApplicationException(javax.ws.rs.WebApplicationException) WebClientResponseException(org.springframework.web.reactive.function.client.WebClientResponseException) Either(com.netflix.titus.common.util.tuple.Either) WebApplicationException(javax.ws.rs.WebApplicationException) WebClientResponseException(org.springframework.web.reactive.function.client.WebClientResponseException)

Example 4 with Either

use of com.netflix.titus.common.util.tuple.Either in project titus-control-plane by Netflix.

the class KubeNotificationProcessor method handlePodUpdatedEvent.

private Mono<Void> handlePodUpdatedEvent(PodEvent event, Job job, Task task) {
    // This is basic sanity check. If it fails, we have a major problem with pod state.
    if (event.getPod() == null || event.getPod().getStatus() == null || event.getPod().getStatus().getPhase() == null) {
        logger.warn("Pod notification with pod without status or phase set: taskId={}, pod={}", task.getId(), event.getPod());
        metricsNoChangesApplied.increment();
        return Mono.empty();
    }
    PodWrapper podWrapper = new PodWrapper(event.getPod());
    Optional<V1Node> node;
    if (event instanceof PodUpdatedEvent) {
        node = ((PodUpdatedEvent) event).getNode();
    } else if (event instanceof PodDeletedEvent) {
        node = ((PodDeletedEvent) event).getNode();
    } else {
        node = Optional.empty();
    }
    Either<TaskStatus, String> newTaskStatusOrError = new PodToTaskMapper(podWrapper, node, task, event instanceof PodDeletedEvent, containerResultCodeResolver, titusRuntime).getNewTaskStatus();
    if (newTaskStatusOrError.hasError()) {
        logger.info(newTaskStatusOrError.getError());
        metricsNoChangesApplied.increment();
        return Mono.empty();
    }
    TaskStatus newTaskStatus = newTaskStatusOrError.getValue();
    if (TaskStatus.areEquivalent(task.getStatus(), newTaskStatus)) {
        logger.info("Pod change notification does not change task status: taskId={}, status={}, eventSequenceNumber={}", task.getId(), newTaskStatus, event.getSequenceNumber());
    } else {
        logger.info("Pod notification changes task status: taskId={}, fromStatus={}, toStatus={}, eventSequenceNumber={}", task.getId(), task.getStatus(), newTaskStatus, event.getSequenceNumber());
    }
    // against most up to date task version.
    if (!updateTaskStatus(podWrapper, newTaskStatus, node, task, true).isPresent()) {
        return Mono.empty();
    }
    return ReactorExt.toMono(v3JobOperations.updateTask(task.getId(), current -> updateTaskStatus(podWrapper, newTaskStatus, node, current, false), V3JobOperations.Trigger.Kube, "Pod status updated from kubernetes node (k8phase='" + event.getPod().getStatus().getPhase() + "', taskState=" + task.getStatus().getState() + ")", KUBE_CALL_METADATA));
}
Also used : Retry(reactor.util.retry.Retry) Task(com.netflix.titus.api.jobmanager.model.job.Task) CollectionsExt(com.netflix.titus.common.util.CollectionsExt) LoggerFactory(org.slf4j.LoggerFactory) V1PodStatus(io.kubernetes.client.openapi.models.V1PodStatus) ReactorExt(com.netflix.titus.common.util.rx.ReactorExt) KubeUtil(com.netflix.titus.master.kubernetes.KubeUtil) TITUS_NODE_DOMAIN(com.netflix.titus.runtime.kubernetes.KubeConstants.TITUS_NODE_DOMAIN) Duration(java.time.Duration) Map(java.util.Map) DirectKubeApiServerIntegrator(com.netflix.titus.master.kubernetes.client.DirectKubeApiServerIntegrator) Either(com.netflix.titus.common.util.tuple.Either) CallMetadata(com.netflix.titus.api.model.callmetadata.CallMetadata) PodEvent(com.netflix.titus.master.kubernetes.client.model.PodEvent) Job(com.netflix.titus.api.jobmanager.model.job.Job) TaskStatus(com.netflix.titus.api.jobmanager.model.job.TaskStatus) JobFunctions(com.netflix.titus.api.jobmanager.model.job.JobFunctions) TaskState(com.netflix.titus.api.jobmanager.model.job.TaskState) PodNotFoundEvent(com.netflix.titus.master.kubernetes.client.model.PodNotFoundEvent) Timer(com.netflix.spectator.api.Timer) List(java.util.List) Optional(java.util.Optional) PodWrapper(com.netflix.titus.master.kubernetes.client.model.PodWrapper) Gauge(com.netflix.spectator.api.Gauge) Disposable(reactor.core.Disposable) Stopwatch(com.google.common.base.Stopwatch) PodDeletedEvent(com.netflix.titus.master.kubernetes.client.model.PodDeletedEvent) Counter(com.netflix.spectator.api.Counter) HashMap(java.util.HashMap) MetricConstants(com.netflix.titus.master.MetricConstants) V1Node(io.kubernetes.client.openapi.models.V1Node) Singleton(javax.inject.Singleton) Scheduler(reactor.core.scheduler.Scheduler) ArrayList(java.util.ArrayList) Inject(javax.inject.Inject) Pair(com.netflix.titus.common.util.tuple.Pair) ContainerResultCodeResolver(com.netflix.titus.master.kubernetes.ContainerResultCodeResolver) Schedulers(reactor.core.scheduler.Schedulers) Evaluators.acceptNotNull(com.netflix.titus.common.util.Evaluators.acceptNotNull) KubeJobManagementReconciler(com.netflix.titus.master.kubernetes.controller.KubeJobManagementReconciler) ExecutorService(java.util.concurrent.ExecutorService) ExecutorsExt(com.netflix.titus.common.util.ExecutorsExt) Logger(org.slf4j.Logger) PodUpdatedEvent(com.netflix.titus.master.kubernetes.client.model.PodUpdatedEvent) Mono(reactor.core.publisher.Mono) Activator(com.netflix.titus.common.util.guice.annotation.Activator) TimeUnit(java.util.concurrent.TimeUnit) AtomicLong(java.util.concurrent.atomic.AtomicLong) ExecutableStatus(com.netflix.titus.api.jobmanager.model.job.ExecutableStatus) V3JobOperations(com.netflix.titus.api.jobmanager.service.V3JobOperations) TaskAttributes(com.netflix.titus.api.jobmanager.TaskAttributes) PodToTaskMapper(com.netflix.titus.master.kubernetes.PodToTaskMapper) V1ContainerState(io.kubernetes.client.openapi.models.V1ContainerState) VisibleForTesting(com.google.common.annotations.VisibleForTesting) TitusRuntime(com.netflix.titus.common.runtime.TitusRuntime) Comparator(java.util.Comparator) Evaluators(com.netflix.titus.common.util.Evaluators) PodToTaskMapper(com.netflix.titus.master.kubernetes.PodToTaskMapper) PodDeletedEvent(com.netflix.titus.master.kubernetes.client.model.PodDeletedEvent) V1Node(io.kubernetes.client.openapi.models.V1Node) PodWrapper(com.netflix.titus.master.kubernetes.client.model.PodWrapper) PodUpdatedEvent(com.netflix.titus.master.kubernetes.client.model.PodUpdatedEvent) TaskStatus(com.netflix.titus.api.jobmanager.model.job.TaskStatus)

Example 5 with Either

use of com.netflix.titus.common.util.tuple.Either in project titus-control-plane by Netflix.

the class ReactorMapWithStateTransformer method apply.

@Override
public Publisher<R> apply(Flux<T> source) {
    return Flux.create(sink -> {
        AtomicReference<S> lastState = new AtomicReference<>(zeroSupplier.get());
        Flux<Either<T, Function<S, Pair<R, S>>>> sourceEither = source.map(Either::ofValue);
        Flux<Either<T, Function<S, Pair<R, S>>>> cleanupEither = cleanupActions.map(Either::ofError);
        Disposable subscription = Flux.merge(sourceEither, cleanupEither).subscribe(next -> {
            Pair<R, S> result;
            if (next.hasValue()) {
                try {
                    result = transformer.apply(next.getValue(), lastState.get());
                } catch (Throwable e) {
                    sink.error(e);
                    return;
                }
            } else {
                try {
                    Function<S, Pair<R, S>> action = next.getError();
                    result = action.apply(lastState.get());
                } catch (Throwable e) {
                    sink.error(e);
                    return;
                }
            }
            lastState.set(result.getRight());
            sink.next(result.getLeft());
        }, sink::error, sink::complete);
        sink.onCancel(subscription);
    });
}
Also used : Disposable(reactor.core.Disposable) Either(com.netflix.titus.common.util.tuple.Either) AtomicReference(java.util.concurrent.atomic.AtomicReference) Pair(com.netflix.titus.common.util.tuple.Pair)

Aggregations

Either (com.netflix.titus.common.util.tuple.Either)7 Logger (org.slf4j.Logger)5 LoggerFactory (org.slf4j.LoggerFactory)5 Pair (com.netflix.titus.common.util.tuple.Pair)4 ArrayList (java.util.ArrayList)4 List (java.util.List)4 CollectionsExt (com.netflix.titus.common.util.CollectionsExt)3 Duration (java.time.Duration)3 Mono (reactor.core.publisher.Mono)3 Job (com.netflix.titus.api.jobmanager.model.job.Job)2 JobFunctions (com.netflix.titus.api.jobmanager.model.job.JobFunctions)2 Task (com.netflix.titus.api.jobmanager.model.job.Task)2 TaskState (com.netflix.titus.api.jobmanager.model.job.TaskState)2 V3JobOperations (com.netflix.titus.api.jobmanager.service.V3JobOperations)2 TitusRuntime (com.netflix.titus.common.runtime.TitusRuntime)2 Collections (java.util.Collections)2 Collectors (java.util.stream.Collectors)2 Inject (javax.inject.Inject)2 Singleton (javax.inject.Singleton)2 AmazonIdentityManagementAsync (com.amazonaws.services.identitymanagement.AmazonIdentityManagementAsync)1