Search in sources :

Example 1 with ClusterMemberLeadershipState

use of com.netflix.titus.api.clustermembership.model.ClusterMemberLeadershipState in project titus-control-plane by Netflix.

the class DefaultClusterMembershipService method clusterStateEvaluator.

private Mono<Void> clusterStateEvaluator(ExecutionContext context) {
    return Mono.defer(() -> {
        ClusterMember localMember = connector.getLocalClusterMemberRevision().getCurrent();
        ClusterMemberLeadershipState localLeadershipState = connector.getLocalLeadershipRevision().getCurrent().getLeadershipState();
        HealthStatus health = healthIndicator.health();
        // Explicitly disabled
        if (!configuration.isLeaderElectionEnabled() || !localMember.isEnabled()) {
            if (localLeadershipState == ClusterMemberLeadershipState.NonLeader) {
                logger.info("Local member excluded from the leader election. Leaving the leader election process");
                return connector.leaveLeadershipGroup(true).flatMap(success -> success ? connector.register(current -> toInactive(current, "Marked by a user as disabled")).ignoreElement().cast(Void.class) : Mono.empty());
            }
            if (localLeadershipState == ClusterMemberLeadershipState.Disabled && localMember.isActive()) {
                return connector.register(current -> toInactive(current, "Marked by a user as disabled")).ignoreElement().cast(Void.class);
            }
            return Mono.empty();
        }
        // Re-enable if healthy
        if (health.getHealthState() == HealthState.Healthy) {
            if (localLeadershipState == ClusterMemberLeadershipState.Disabled) {
                logger.info("Re-enabling local member which is in the disabled state");
                return connector.joinLeadershipGroup().then(connector.register(this::toActive).ignoreElement().cast(Void.class));
            }
            if (!localMember.isActive()) {
                return connector.register(this::toActive).ignoreElement().cast(Void.class);
            }
            return Mono.empty();
        }
        // Disable if unhealthy (and not the leader)
        if (localLeadershipState != ClusterMemberLeadershipState.Disabled && localLeadershipState != ClusterMemberLeadershipState.Leader) {
            logger.info("Disabling local member as it is unhealthy: {}", health);
            return connector.leaveLeadershipGroup(true).flatMap(success -> success ? connector.register(current -> toInactive(current, "Unhealthy: " + health)).ignoreElement().cast(Void.class) : Mono.empty());
        }
        if (localLeadershipState == ClusterMemberLeadershipState.Disabled && localMember.isActive()) {
            return connector.register(current -> toInactive(current, "Unhealthy: " + health)).ignoreElement().cast(Void.class);
        }
        return Mono.empty();
    }).doOnError(error -> {
        logger.info("Cluster membership health evaluation error: {}", error.getMessage());
        logger.debug("Stack trace", error);
    }).doOnTerminate(() -> {
        metrics.updateLocal(connector.getLocalLeadershipRevision().getCurrent().getLeadershipState(), healthIndicator.health());
        metrics.updateSiblings(connector.getClusterMemberSiblings());
    });
}
Also used : ExecutionContext(com.netflix.titus.common.framework.scheduler.ExecutionContext) Disposable(reactor.core.Disposable) LoggerFactory(org.slf4j.LoggerFactory) Singleton(javax.inject.Singleton) ReactorExt(com.netflix.titus.common.util.rx.ReactorExt) Function(java.util.function.Function) ScheduleReference(com.netflix.titus.common.framework.scheduler.ScheduleReference) Inject(javax.inject.Inject) ClusterMembershipService(com.netflix.titus.api.clustermembership.service.ClusterMembershipService) HealthStatus(com.netflix.titus.api.health.HealthStatus) ClusterMembershipConnector(com.netflix.titus.api.clustermembership.connector.ClusterMembershipConnector) Duration(java.time.Duration) Map(java.util.Map) Schedulers(reactor.core.scheduler.Schedulers) ClusterMember(com.netflix.titus.api.clustermembership.model.ClusterMember) ClusterMembershipRevision(com.netflix.titus.api.clustermembership.model.ClusterMembershipRevision) Logger(org.slf4j.Logger) ClusterMemberLeadershipState(com.netflix.titus.api.clustermembership.model.ClusterMemberLeadershipState) Retryers(com.netflix.titus.common.util.retry.Retryers) ClusterMemberLeadership(com.netflix.titus.api.clustermembership.model.ClusterMemberLeadership) Mono(reactor.core.publisher.Mono) ClusterMembershipServiceException(com.netflix.titus.api.clustermembership.service.ClusterMembershipServiceException) HealthIndicator(com.netflix.titus.api.health.HealthIndicator) Flux(reactor.core.publisher.Flux) ClusterMembershipEvent(com.netflix.titus.api.clustermembership.model.event.ClusterMembershipEvent) HealthState(com.netflix.titus.api.health.HealthState) ScheduleDescriptor(com.netflix.titus.common.framework.scheduler.model.ScheduleDescriptor) Optional(java.util.Optional) TitusRuntime(com.netflix.titus.common.runtime.TitusRuntime) Clock(com.netflix.titus.common.util.time.Clock) ClusterMember(com.netflix.titus.api.clustermembership.model.ClusterMember) HealthStatus(com.netflix.titus.api.health.HealthStatus) ClusterMemberLeadershipState(com.netflix.titus.api.clustermembership.model.ClusterMemberLeadershipState)

Example 2 with ClusterMemberLeadershipState

use of com.netflix.titus.api.clustermembership.model.ClusterMemberLeadershipState in project titus-control-plane by Netflix.

the class ClusterMembershipConnectorStub method leaveLeadershipGroup.

@Override
public Mono<Boolean> leaveLeadershipGroup(boolean onlyNonLeader) {
    return Mono.defer(() -> {
        ClusterMemberLeadershipState state = localLeadershipRevision.getCurrent().getLeadershipState();
        if (state == ClusterMemberLeadershipState.Disabled) {
            return Mono.just(true);
        }
        if (onlyNonLeader && state == ClusterMemberLeadershipState.Leader) {
            return Mono.just(false);
        }
        this.localLeadershipRevision = newLocalLeadershipState(ClusterMemberLeadershipState.Disabled);
        emitEvent(ClusterMembershipEvent.localLeftElection(localLeadershipRevision));
        return Mono.just(true);
    });
}
Also used : ClusterMemberLeadershipState(com.netflix.titus.api.clustermembership.model.ClusterMemberLeadershipState)

Example 3 with ClusterMemberLeadershipState

use of com.netflix.titus.api.clustermembership.model.ClusterMemberLeadershipState in project titus-control-plane by Netflix.

the class ClusterMembershipConnectorStub method joinLeadershipGroup.

@Override
public Mono<Void> joinLeadershipGroup() {
    return Mono.defer(() -> {
        ClusterMemberLeadershipState state = localLeadershipRevision.getCurrent().getLeadershipState();
        if (state == ClusterMemberLeadershipState.Disabled) {
            this.localLeadershipRevision = newLocalLeadershipState(ClusterMemberLeadershipState.NonLeader);
        }
        emitEvent(ClusterMembershipEvent.localJoinedElection(localLeadershipRevision));
        return Mono.empty();
    });
}
Also used : ClusterMemberLeadershipState(com.netflix.titus.api.clustermembership.model.ClusterMemberLeadershipState)

Aggregations

ClusterMemberLeadershipState (com.netflix.titus.api.clustermembership.model.ClusterMemberLeadershipState)3 ClusterMembershipConnector (com.netflix.titus.api.clustermembership.connector.ClusterMembershipConnector)1 ClusterMember (com.netflix.titus.api.clustermembership.model.ClusterMember)1 ClusterMemberLeadership (com.netflix.titus.api.clustermembership.model.ClusterMemberLeadership)1 ClusterMembershipRevision (com.netflix.titus.api.clustermembership.model.ClusterMembershipRevision)1 ClusterMembershipEvent (com.netflix.titus.api.clustermembership.model.event.ClusterMembershipEvent)1 ClusterMembershipService (com.netflix.titus.api.clustermembership.service.ClusterMembershipService)1 ClusterMembershipServiceException (com.netflix.titus.api.clustermembership.service.ClusterMembershipServiceException)1 HealthIndicator (com.netflix.titus.api.health.HealthIndicator)1 HealthState (com.netflix.titus.api.health.HealthState)1 HealthStatus (com.netflix.titus.api.health.HealthStatus)1 ExecutionContext (com.netflix.titus.common.framework.scheduler.ExecutionContext)1 ScheduleReference (com.netflix.titus.common.framework.scheduler.ScheduleReference)1 ScheduleDescriptor (com.netflix.titus.common.framework.scheduler.model.ScheduleDescriptor)1 TitusRuntime (com.netflix.titus.common.runtime.TitusRuntime)1 Retryers (com.netflix.titus.common.util.retry.Retryers)1 ReactorExt (com.netflix.titus.common.util.rx.ReactorExt)1 Clock (com.netflix.titus.common.util.time.Clock)1 Duration (java.time.Duration)1 Map (java.util.Map)1