Search in sources :

Example 11 with SingularityDeployKey

use of com.hubspot.singularity.SingularityDeployKey in project Singularity by HubSpot.

the class HistoryResource method getInactiveDeployTasksWithMetadata.

@GET
@Path("/request/{requestId}/deploy/{deployId}/tasks/inactive/withmetadata")
@ApiOperation("Retrieve the task history for a specific deploy.")
public SingularityPaginatedResponse<SingularityTaskIdHistory> getInactiveDeployTasksWithMetadata(@Auth SingularityUser user, @ApiParam("Request ID for deploy") @PathParam("requestId") String requestId, @ApiParam("Deploy ID") @PathParam("deployId") String deployId, @ApiParam("Maximum number of items to return") @QueryParam("count") Integer count, @ApiParam("Which page of items to view") @QueryParam("page") Integer page) {
    authorizationHelper.checkForAuthorizationByRequestId(requestId, user, SingularityAuthorizationScope.READ);
    SingularityDeployKey key = new SingularityDeployKey(requestId, deployId);
    Optional<Integer> dataCount = deployTaskHistoryHelper.getBlendedHistoryCount(key);
    final Integer limitCount = getLimitCount(count);
    final Integer limitStart = getLimitStart(limitCount, page);
    final List<SingularityTaskIdHistory> data = deployTaskHistoryHelper.getBlendedHistory(key, limitStart, limitCount);
    Optional<Integer> pageCount = getPageCount(dataCount, limitCount);
    return new SingularityPaginatedResponse<>(dataCount, pageCount, Optional.fromNullable(page), data);
}
Also used : SingularityDeployKey(com.hubspot.singularity.SingularityDeployKey) SingularityTaskIdHistory(com.hubspot.singularity.SingularityTaskIdHistory) SingularityPaginatedResponse(com.hubspot.singularity.SingularityPaginatedResponse) Path(javax.ws.rs.Path) GET(javax.ws.rs.GET) ApiOperation(com.wordnik.swagger.annotations.ApiOperation)

Example 12 with SingularityDeployKey

use of com.hubspot.singularity.SingularityDeployKey in project Singularity by HubSpot.

the class HistoryResource method getInactiveDeployTasks.

@GET
@Path("/request/{requestId}/deploy/{deployId}/tasks/inactive")
@ApiOperation("Retrieve the task history for a specific deploy.")
public List<SingularityTaskIdHistory> getInactiveDeployTasks(@Auth SingularityUser user, @ApiParam("Request ID for deploy") @PathParam("requestId") String requestId, @ApiParam("Deploy ID") @PathParam("deployId") String deployId, @ApiParam("Maximum number of items to return") @QueryParam("count") Integer count, @ApiParam("Which page of items to view") @QueryParam("page") Integer page) {
    authorizationHelper.checkForAuthorizationByRequestId(requestId, user, SingularityAuthorizationScope.READ);
    final Integer limitCount = getLimitCount(count);
    final Integer limitStart = getLimitStart(limitCount, page);
    SingularityDeployKey key = new SingularityDeployKey(requestId, deployId);
    return deployTaskHistoryHelper.getBlendedHistory(key, limitStart, limitCount);
}
Also used : SingularityDeployKey(com.hubspot.singularity.SingularityDeployKey) Path(javax.ws.rs.Path) GET(javax.ws.rs.GET) ApiOperation(com.wordnik.swagger.annotations.ApiOperation)

Example 13 with SingularityDeployKey

use of com.hubspot.singularity.SingularityDeployKey in project Singularity by HubSpot.

the class SingularityCleaner method shouldKillTask.

private boolean shouldKillTask(SingularityTaskCleanup taskCleanup, List<SingularityTaskId> activeTaskIds, Set<SingularityTaskId> cleaningTasks, Multiset<SingularityDeployKey> incrementalCleaningTasks) {
    final Optional<SingularityRequestWithState> requestWithState = requestManager.getRequest(taskCleanup.getTaskId().getRequestId());
    if (!requestWithState.isPresent()) {
        LOG.debug("Killing a task {} immediately because the request was missing", taskCleanup);
        return true;
    }
    final SingularityRequest request = requestWithState.get().getRequest();
    if (taskCleanup.getRunBeforeKillId().isPresent()) {
        List<SingularityTaskShellCommandUpdate> shellCommandUpdates = taskManager.getTaskShellCommandUpdates(taskCleanup.getRunBeforeKillId().get());
        boolean finished = false;
        for (SingularityTaskShellCommandUpdate update : shellCommandUpdates) {
            if (update.getUpdateType().isFinished()) {
                finished = true;
                break;
            }
        }
        if (!finished) {
            LOG.debug("Waiting for pre-kill shell command {} to finish before killing task", taskCleanup.getRunBeforeKillId());
            return false;
        }
    }
    if (taskCleanup.getCleanupType().shouldKillTaskInstantly(request)) {
        LOG.debug("Killing a task {} immediately because of its cleanup type", taskCleanup);
        return true;
    }
    // If pausing, must be a long-running task to kill here
    if (requestWithState.get().getState() == RequestState.PAUSED && (!(taskCleanup.getCleanupType() == TaskCleanupType.PAUSING) || request.isLongRunning())) {
        LOG.debug("Killing a task {} immediately because the request was paused", taskCleanup);
        return true;
    }
    if (!request.isLongRunning()) {
        final long timeSinceCleanup = System.currentTimeMillis() - taskCleanup.getTimestamp();
        final long maxWaitTime = request.getKillOldNonLongRunningTasksAfterMillis().or(killNonLongRunningTasksInCleanupAfterMillis);
        final boolean tooOld = (maxWaitTime < 1) || (timeSinceCleanup > maxWaitTime);
        if (!tooOld) {
            LOG.trace("Not killing a non-longRunning task {}, running time since cleanup {} (max wait time is {})", taskCleanup, timeSinceCleanup, maxWaitTime);
        } else {
            LOG.debug("Killing a non-longRunning task {} - running time since cleanup {} exceeded max wait time {}", taskCleanup, timeSinceCleanup, maxWaitTime);
        }
        return tooOld;
    }
    final String requestId = request.getId();
    final Optional<SingularityRequestDeployState> deployState = deployManager.getRequestDeployState(requestId);
    if (taskCleanup.getCleanupType() == TaskCleanupType.DECOMISSIONING && deployState.get().getPendingDeploy().isPresent() && deployState.get().getPendingDeploy().get().getDeployId().equals(taskCleanup.getTaskId().getDeployId())) {
        final long timeSinceCleanup = System.currentTimeMillis() - taskCleanup.getTimestamp();
        final long maxWaitTime = configuration.getPendingDeployHoldTaskDuringDecommissionMillis();
        final boolean tooOld = (maxWaitTime < 1) || (timeSinceCleanup > maxWaitTime);
        if (!tooOld) {
            LOG.trace("Not killing {} - part of pending deploy - running time since cleanup {} (max wait time is {})", taskCleanup, timeSinceCleanup, maxWaitTime);
            return false;
        } else {
            LOG.debug("Killing {} - part of pending deploy but running time since cleanup {} exceeded max wait time {}", taskCleanup, timeSinceCleanup, maxWaitTime);
            return true;
        }
    }
    if (!deployState.isPresent() || !deployState.get().getActiveDeploy().isPresent()) {
        LOG.debug("Killing a task {} immediately because there is no active deploy state {}", taskCleanup, deployState);
        return true;
    }
    final String activeDeployId = deployState.get().getActiveDeploy().get().getDeployId();
    final String matchingTasksDeployId = taskCleanup.getCleanupType() == TaskCleanupType.INCREMENTAL_DEPLOY_CANCELLED || taskCleanup.getCleanupType() == TaskCleanupType.INCREMENTAL_DEPLOY_FAILED ? activeDeployId : taskCleanup.getTaskId().getDeployId();
    // check to see if there are enough active tasks out there that have been active for long enough that we can safely shut this task down.
    final List<SingularityTaskId> matchingTasks = new ArrayList<>();
    for (SingularityTaskId taskId : activeTaskIds) {
        if (!taskId.getRequestId().equals(requestId) || !taskId.getDeployId().equals(matchingTasksDeployId)) {
            continue;
        }
        if (cleaningTasks.contains(taskId)) {
            continue;
        }
        matchingTasks.add(taskId);
    }
    // For an incremental bounce or incremental deploy cleanup, shut down old tasks as new ones are started
    final SingularityDeployKey key = SingularityDeployKey.fromTaskId(taskCleanup.getTaskId());
    if (taskCleanup.getCleanupType() == TaskCleanupType.INCREMENTAL_BOUNCE) {
        return shouldKillIncrementalBounceTask(request, taskCleanup, matchingTasksDeployId, matchingTasks, key, incrementalCleaningTasks);
    } else if (isIncrementalDeployCleanup(taskCleanup)) {
        return shouldKillIncrementalDeployCleanupTask(request, taskCleanup, matchingTasksDeployId, matchingTasks, key, incrementalCleaningTasks);
    } else {
        if (matchingTasks.size() < request.getInstancesSafe()) {
            LOG.trace("Not killing a task {} yet, only {} matching out of a required {}", taskCleanup, matchingTasks.size(), request.getInstancesSafe());
            return false;
        }
    }
    final Optional<SingularityDeploy> deploy = deployManager.getDeploy(requestId, activeDeployId);
    final DeployHealth deployHealth = deployHealthHelper.getDeployHealth(requestWithState.get().getRequest(), deploy, matchingTasks, false);
    switch(deployHealth) {
        case HEALTHY:
            for (SingularityTaskId taskId : matchingTasks) {
                DeployHealth lbHealth = getLbHealth(request, taskId);
                if (lbHealth != DeployHealth.HEALTHY) {
                    LOG.trace("Not killing a task {}, waiting for new replacement tasks to be added to LB (current state: {})", taskCleanup, lbHealth);
                    return false;
                }
            }
            LOG.debug("Killing a task {}, all replacement tasks are healthy", taskCleanup);
            return true;
        case WAITING:
        case UNHEALTHY:
        default:
            LOG.trace("Not killing a task {}, waiting for new replacement tasks to be healthy (current state: {})", taskCleanup, deployHealth);
            return false;
    }
}
Also used : SingularityRequest(com.hubspot.singularity.SingularityRequest) ArrayList(java.util.ArrayList) SingularityRequestDeployState(com.hubspot.singularity.SingularityRequestDeployState) SingularityDeploy(com.hubspot.singularity.SingularityDeploy) SingularityDeployKey(com.hubspot.singularity.SingularityDeployKey) SingularityRequestWithState(com.hubspot.singularity.SingularityRequestWithState) SingularityTaskShellCommandUpdate(com.hubspot.singularity.SingularityTaskShellCommandUpdate) DeployHealth(com.hubspot.singularity.scheduler.SingularityDeployHealthHelper.DeployHealth) SingularityTaskId(com.hubspot.singularity.SingularityTaskId)

Example 14 with SingularityDeployKey

use of com.hubspot.singularity.SingularityDeployKey in project Singularity by HubSpot.

the class SingularityCooldownChecker method shouldExitCooldown.

private boolean shouldExitCooldown(SingularityRequestWithState cooldownRequest) {
    Optional<SingularityRequestDeployState> maybeDeployState = deployManager.getRequestDeployState(cooldownRequest.getRequest().getId());
    if (!maybeDeployState.isPresent() || !maybeDeployState.get().getActiveDeploy().isPresent()) {
        LOG.trace("{} had no deployState / activeDeploy {}, exiting cooldown", cooldownRequest.getRequest().getId(), maybeDeployState);
        return true;
    }
    Optional<SingularityDeployStatistics> maybeDeployStatistics = deployManager.getDeployStatistics(cooldownRequest.getRequest().getId(), maybeDeployState.get().getActiveDeploy().get().getDeployId());
    if (!maybeDeployStatistics.isPresent()) {
        LOG.trace("{} had no deploy statistics, exiting cooldown", new SingularityDeployKey(cooldownRequest.getRequest().getId(), maybeDeployState.get().getActiveDeploy().get().getDeployId()));
        return true;
    }
    Optional<Long> lastFinishAt = maybeDeployStatistics.get().getLastFinishAt();
    if (!lastFinishAt.isPresent()) {
        LOG.trace("{} had no last finish, exiting cooldown", new SingularityDeployKey(cooldownRequest.getRequest().getId(), maybeDeployState.get().getActiveDeploy().get().getDeployId()));
        return true;
    }
    if (cooldown.hasCooldownExpired(cooldownRequest.getRequest(), maybeDeployStatistics.get(), Optional.<Integer>absent(), Optional.<Long>absent())) {
        return true;
    }
    return false;
}
Also used : SingularityDeployKey(com.hubspot.singularity.SingularityDeployKey) SingularityRequestDeployState(com.hubspot.singularity.SingularityRequestDeployState) SingularityDeployStatistics(com.hubspot.singularity.SingularityDeployStatistics)

Example 15 with SingularityDeployKey

use of com.hubspot.singularity.SingularityDeployKey in project Singularity by HubSpot.

the class SingularityClient method cancelPendingDeployForSingularityRequest.

public SingularityRequestParent cancelPendingDeployForSingularityRequest(String requestId, String deployId) {
    final Function<String, String> requestUri = (host) -> String.format(DELETE_DEPLOY_FORMAT, getApiBase(host), deployId, requestId);
    SingularityRequestParent singularityRequestParent = delete(requestUri, "pending deploy", new SingularityDeployKey(requestId, deployId).getId(), Optional.absent(), Optional.of(SingularityRequestParent.class)).get();
    return getAndLogRequestAndDeployStatus(singularityRequestParent);
}
Also used : SingularityTask(com.hubspot.singularity.SingularityTask) Arrays(java.util.Arrays) Method(com.hubspot.horizon.HttpRequest.Method) ExtendedTaskState(com.hubspot.singularity.ExtendedTaskState) SingularityTaskCleanupResult(com.hubspot.singularity.SingularityTaskCleanupResult) SingularityPaginatedResponse(com.hubspot.singularity.SingularityPaginatedResponse) SingularityWebhook(com.hubspot.singularity.SingularityWebhook) Provider(javax.inject.Provider) Retryer(com.github.rholder.retry.Retryer) RetryerBuilder(com.github.rholder.retry.RetryerBuilder) SingularityDeployRequest(com.hubspot.singularity.api.SingularityDeployRequest) RetryException(com.github.rholder.retry.RetryException) Inject(com.google.inject.Inject) SingularityRequestCleanup(com.hubspot.singularity.SingularityRequestCleanup) SingularityState(com.hubspot.singularity.SingularityState) LoggerFactory(org.slf4j.LoggerFactory) Random(java.util.Random) SingularityTaskReconciliationStatistics(com.hubspot.singularity.SingularityTaskReconciliationStatistics) SingularityRequestGroup(com.hubspot.singularity.SingularityRequestGroup) SingularityDisasterType(com.hubspot.singularity.SingularityDisasterType) SingularityBounceRequest(com.hubspot.singularity.api.SingularityBounceRequest) SingularityPauseRequest(com.hubspot.singularity.api.SingularityPauseRequest) SingularityUpdateGroupsRequest(com.hubspot.singularity.api.SingularityUpdateGroupsRequest) Optional(com.google.common.base.Optional) Map(java.util.Map) HttpRequest(com.hubspot.horizon.HttpRequest) HttpClient(com.hubspot.horizon.HttpClient) TypeReference(com.fasterxml.jackson.core.type.TypeReference) SingularityPriorityFreeze(com.hubspot.singularity.api.SingularityPriorityFreeze) SingularityDisabledAction(com.hubspot.singularity.SingularityDisabledAction) SingularityPendingTaskId(com.hubspot.singularity.SingularityPendingTaskId) SingularityTaskId(com.hubspot.singularity.SingularityTaskId) SingularityUnpauseRequest(com.hubspot.singularity.api.SingularityUnpauseRequest) SingularityMachineChangeRequest(com.hubspot.singularity.api.SingularityMachineChangeRequest) SingularityDisastersData(com.hubspot.singularity.SingularityDisastersData) ImmutableMap(com.google.common.collect.ImmutableMap) SingularityPriorityFreezeParent(com.hubspot.singularity.SingularityPriorityFreezeParent) Predicate(java.util.function.Predicate) Collection(java.util.Collection) SingularityUpdatePendingDeployRequest(com.hubspot.singularity.SingularityUpdatePendingDeployRequest) List(java.util.List) SingularityTaskShellCommandRequest(com.hubspot.singularity.SingularityTaskShellCommandRequest) SingularityTaskShellCommandUpdate(com.hubspot.singularity.SingularityTaskShellCommandUpdate) StopStrategies(com.github.rholder.retry.StopStrategies) SingularityClusterUtilization(com.hubspot.singularity.SingularityClusterUtilization) OrderDirection(com.hubspot.singularity.OrderDirection) SingularityAuthorizationScope(com.hubspot.singularity.SingularityAuthorizationScope) Entry(java.util.Map.Entry) SingularitySandbox(com.hubspot.singularity.SingularitySandbox) SingularityS3Log(com.hubspot.singularity.SingularityS3Log) SingularityDisabledActionRequest(com.hubspot.singularity.api.SingularityDisabledActionRequest) SingularityCreateResult(com.hubspot.singularity.SingularityCreateResult) SingularityRequest(com.hubspot.singularity.SingularityRequest) SingularityTaskHistoryUpdate(com.hubspot.singularity.SingularityTaskHistoryUpdate) SingularityShellCommand(com.hubspot.singularity.SingularityShellCommand) SingularityRequestHistory(com.hubspot.singularity.SingularityRequestHistory) SingularityTaskIdHistory(com.hubspot.singularity.SingularityTaskIdHistory) SingularityPendingRequest(com.hubspot.singularity.SingularityPendingRequest) SingularityTaskShellCommandHistory(com.hubspot.singularity.SingularityTaskShellCommandHistory) HashMap(java.util.HashMap) SingularityClientCredentials(com.hubspot.singularity.SingularityClientCredentials) SingularityDeployUpdate(com.hubspot.singularity.SingularityDeployUpdate) Function(java.util.function.Function) SingularityTaskIdsByStatus(com.hubspot.singularity.SingularityTaskIdsByStatus) SingularityTaskState(com.hubspot.singularity.SingularityTaskState) SingularityExitCooldownRequest(com.hubspot.singularity.api.SingularityExitCooldownRequest) ArrayList(java.util.ArrayList) Builder(com.google.common.collect.ImmutableMap.Builder) ImmutableList(com.google.common.collect.ImmutableList) SingularitySlave(com.hubspot.singularity.SingularitySlave) SingularityDeleteResult(com.hubspot.singularity.SingularityDeleteResult) SingularityDeployKey(com.hubspot.singularity.SingularityDeployKey) RetryStrategy(com.hubspot.horizon.RetryStrategy) HttpResponse(com.hubspot.horizon.HttpResponse) SingularityPendingRequestParent(com.hubspot.singularity.SingularityPendingRequestParent) WaitStrategies(com.github.rholder.retry.WaitStrategies) SingularityAction(com.hubspot.singularity.SingularityAction) SingularityDeployHistory(com.hubspot.singularity.SingularityDeployHistory) Logger(org.slf4j.Logger) SingularityTaskHistory(com.hubspot.singularity.SingularityTaskHistory) Preconditions.checkNotNull(com.google.common.base.Preconditions.checkNotNull) SingularityDeploy(com.hubspot.singularity.SingularityDeploy) SingularityRack(com.hubspot.singularity.SingularityRack) ExecutionException(java.util.concurrent.ExecutionException) SingularityKillTaskRequest(com.hubspot.singularity.api.SingularityKillTaskRequest) SingularityTaskRequest(com.hubspot.singularity.SingularityTaskRequest) MachineState(com.hubspot.singularity.MachineState) SingularityRequestParent(com.hubspot.singularity.SingularityRequestParent) SingularityDeleteRequestRequest(com.hubspot.singularity.api.SingularityDeleteRequestRequest) SingularityRunNowRequest(com.hubspot.singularity.api.SingularityRunNowRequest) Named(com.google.inject.name.Named) SingularityScaleRequest(com.hubspot.singularity.api.SingularityScaleRequest) MesosFileChunkObject(com.hubspot.mesos.json.MesosFileChunkObject) Collections(java.util.Collections) SingularityDeployKey(com.hubspot.singularity.SingularityDeployKey) SingularityRequestParent(com.hubspot.singularity.SingularityRequestParent)

Aggregations

SingularityDeployKey (com.hubspot.singularity.SingularityDeployKey)19 SingularityRequest (com.hubspot.singularity.SingularityRequest)7 SingularityTaskId (com.hubspot.singularity.SingularityTaskId)7 SingularityDeploy (com.hubspot.singularity.SingularityDeploy)6 SingularityPendingTaskId (com.hubspot.singularity.SingularityPendingTaskId)6 SingularityRequestWithState (com.hubspot.singularity.SingularityRequestWithState)6 ArrayList (java.util.ArrayList)6 SingularityPendingRequest (com.hubspot.singularity.SingularityPendingRequest)5 SingularityRequestDeployState (com.hubspot.singularity.SingularityRequestDeployState)5 Optional (com.google.common.base.Optional)3 ImmutableList (com.google.common.collect.ImmutableList)3 Inject (com.google.inject.Inject)3 HttpResponse (com.hubspot.horizon.HttpResponse)3 ExtendedTaskState (com.hubspot.singularity.ExtendedTaskState)3 MachineState (com.hubspot.singularity.MachineState)3 SingularityTask (com.hubspot.singularity.SingularityTask)3 SingularityTaskRequest (com.hubspot.singularity.SingularityTaskRequest)3 SingularityTaskShellCommandUpdate (com.hubspot.singularity.SingularityTaskShellCommandUpdate)3 List (java.util.List)3 Map (java.util.Map)3