Search in sources :

Example 36 with SingularityTask

use of com.hubspot.singularity.SingularityTask in project Singularity by HubSpot.

the class SingularityHealthchecksTest method testStartupTimeout.

@Test
public void testStartupTimeout() {
    initRequest();
    final long hourAgo = System.currentTimeMillis() - TimeUnit.HOURS.toMillis(1);
    final String deployId = "startup_timeout_test";
    HealthcheckOptions options = new HealthcheckOptionsBuilder("http://uri").setMaxRetries(Optional.of(2)).setStartupTimeoutSeconds(Optional.of((int) TimeUnit.MINUTES.toSeconds(30))).build();
    SingularityDeployBuilder db = new SingularityDeployBuilder(requestId, deployId).setHealthcheck(Optional.of(options));
    db.setDeployHealthTimeoutSeconds(Optional.of(TimeUnit.DAYS.toMillis(1)));
    SingularityDeploy deploy = initDeploy(db, hourAgo);
    deployChecker.checkDeploys();
    Assert.assertTrue(!deployManager.getDeployResult(requestId, deployId).isPresent());
    SingularityTask task = launchTask(request, deploy, hourAgo, hourAgo + 1, 1, TaskState.TASK_RUNNING);
    deployChecker.checkDeploys();
    Assert.assertTrue(!deployManager.getDeployResult(requestId, deployId).isPresent());
    taskManager.saveHealthcheckResult(new SingularityTaskHealthcheckResult(Optional.<Integer>absent(), Optional.of(1000L), hourAgo + 1, Optional.<String>absent(), Optional.of("ERROR"), task.getTaskId(), Optional.of(true)));
    deployChecker.checkDeploys();
    Assert.assertEquals(DeployState.FAILED, deployManager.getDeployResult(requestId, deployId).get().getDeployState());
}
Also used : SingularityTaskHealthcheckResult(com.hubspot.singularity.SingularityTaskHealthcheckResult) SingularityTask(com.hubspot.singularity.SingularityTask) HealthcheckOptions(com.hubspot.deploy.HealthcheckOptions) SingularityDeployBuilder(com.hubspot.singularity.SingularityDeployBuilder) HealthcheckOptionsBuilder(com.hubspot.deploy.HealthcheckOptionsBuilder) SingularityDeploy(com.hubspot.singularity.SingularityDeploy) Test(org.junit.Test)

Example 37 with SingularityTask

use of com.hubspot.singularity.SingularityTask in project Singularity by HubSpot.

the class SingularityHealthchecksTest method testSkipHealthchecksEdgeCases.

@Test
public void testSkipHealthchecksEdgeCases() {
    try {
        setConfigurationForNoDelay();
        configuration.setKillAfterTasksDoNotRunDefaultSeconds(100);
        configuration.setCheckNewTasksEverySeconds(1);
        initRequest();
        initHCDeploy();
        requestResource.skipHealthchecks(requestId, new SingularitySkipHealthchecksRequest(Optional.of(Boolean.TRUE), Optional.absent(), Optional.absent(), Optional.absent()), singularityUser);
        SingularityTask firstTask = startTask(firstDeploy, 1);
        Assert.assertTrue(!taskManager.getLastHealthcheck(firstTask.getTaskId()).isPresent());
        finishHealthchecks();
        finishNewTaskChecksAndCleanup();
        Assert.assertEquals(1, taskManager.getNumActiveTasks());
        requestResource.skipHealthchecks(requestId, new SingularitySkipHealthchecksRequest(Optional.of(Boolean.FALSE), Optional.absent(), Optional.absent(), Optional.absent()), singularityUser);
        // run new task check ONLY.
        newTaskChecker.enqueueNewTaskCheck(firstTask, requestManager.getRequest(requestId), healthchecker);
        finishNewTaskChecks();
        finishHealthchecks();
        finishNewTaskChecksAndCleanup();
        // healthcheck will fail
        Assert.assertTrue(taskManager.getLastHealthcheck(firstTask.getTaskId()).isPresent());
        Assert.assertEquals(0, taskManager.getNumActiveTasks());
    } finally {
        unsetConfigurationForNoDelay();
        configuration.setCheckNewTasksEverySeconds(5);
    }
}
Also used : SingularityTask(com.hubspot.singularity.SingularityTask) SingularitySkipHealthchecksRequest(com.hubspot.singularity.api.SingularitySkipHealthchecksRequest) Test(org.junit.Test)

Example 38 with SingularityTask

use of com.hubspot.singularity.SingularityTask in project Singularity by HubSpot.

the class SingularityHealthchecksTest method testPortIndices.

@Test
public void testPortIndices() {
    try {
        setConfigurationForNoDelay();
        initRequest();
        HealthcheckOptions options = new HealthcheckOptionsBuilder("http://uri").setPortIndex(Optional.of(1)).setStartupDelaySeconds(Optional.of(0)).build();
        firstDeploy = initAndFinishDeploy(request, new SingularityDeployBuilder(request.getId(), firstDeployId).setCommand(Optional.of("sleep 100")).setHealthcheck(Optional.of(options)), Optional.of(new Resources(1, 64, 3, 0)));
        requestResource.postRequest(request.toBuilder().setInstances(Optional.of(2)).build(), singularityUser);
        scheduler.drainPendingQueue();
        String[] portRange = { "80:82" };
        sms.resourceOffers(Arrays.asList(createOffer(20, 20000, 50000, "slave1", "host1", Optional.<String>absent(), Collections.<String, String>emptyMap(), portRange)));
        SingularityTaskId firstTaskId = taskManager.getActiveTaskIdsForRequest(requestId).get(0);
        SingularityTask firstTask = taskManager.getTask(firstTaskId).get();
        statusUpdate(firstTask, TaskState.TASK_RUNNING);
        newTaskChecker.enqueueNewTaskCheck(firstTask, requestManager.getRequest(requestId), healthchecker);
        Awaitility.await("healthcheck present").atMost(5, TimeUnit.SECONDS).until(() -> taskManager.getLastHealthcheck(firstTask.getTaskId()).isPresent());
        Assert.assertTrue(taskManager.getLastHealthcheck(firstTask.getTaskId()).get().toString().contains("host1:81"));
    } finally {
        unsetConfigurationForNoDelay();
    }
}
Also used : SingularityTask(com.hubspot.singularity.SingularityTask) HealthcheckOptions(com.hubspot.deploy.HealthcheckOptions) SingularityDeployBuilder(com.hubspot.singularity.SingularityDeployBuilder) Resources(com.hubspot.mesos.Resources) HealthcheckOptionsBuilder(com.hubspot.deploy.HealthcheckOptionsBuilder) SingularityTaskId(com.hubspot.singularity.SingularityTaskId) Test(org.junit.Test)

Example 39 with SingularityTask

use of com.hubspot.singularity.SingularityTask in project Singularity by HubSpot.

the class SingularityHealthchecksTest method testNewTaskCheckerRespectsDeployHealthcheckRetries.

@Test
public void testNewTaskCheckerRespectsDeployHealthcheckRetries() {
    initRequest();
    final String deployId = "new_task_healthcheck";
    HealthcheckOptions options = new HealthcheckOptionsBuilder("http://uri").setMaxRetries(Optional.of(1)).build();
    SingularityDeployBuilder db = new SingularityDeployBuilder(requestId, deployId).setHealthcheck(Optional.of(options));
    SingularityDeploy deploy = initAndFinishDeploy(request, db, Optional.absent());
    SingularityTask task = launchTask(request, deploy, System.currentTimeMillis(), 1, TaskState.TASK_RUNNING);
    Assert.assertEquals(CheckTaskState.CHECK_IF_HEALTHCHECK_OVERDUE, newTaskChecker.getTaskState(task, requestManager.getRequest(requestId), healthchecker));
    Assert.assertTrue(taskManager.getCleanupTaskIds().isEmpty());
    taskManager.saveHealthcheckResult(new SingularityTaskHealthcheckResult(Optional.of(503), Optional.of(1000L), System.currentTimeMillis() + 1, Optional.<String>absent(), Optional.<String>absent(), task.getTaskId(), Optional.<Boolean>absent()));
    Assert.assertEquals(CheckTaskState.CHECK_IF_HEALTHCHECK_OVERDUE, newTaskChecker.getTaskState(task, requestManager.getRequest(requestId), healthchecker));
    taskManager.saveHealthcheckResult(new SingularityTaskHealthcheckResult(Optional.of(503), Optional.of(1000L), System.currentTimeMillis() + 1, Optional.<String>absent(), Optional.<String>absent(), task.getTaskId(), Optional.<Boolean>absent()));
    Assert.assertEquals(CheckTaskState.UNHEALTHY_KILL_TASK, newTaskChecker.getTaskState(task, requestManager.getRequest(requestId), healthchecker));
}
Also used : SingularityTaskHealthcheckResult(com.hubspot.singularity.SingularityTaskHealthcheckResult) SingularityTask(com.hubspot.singularity.SingularityTask) HealthcheckOptions(com.hubspot.deploy.HealthcheckOptions) SingularityDeployBuilder(com.hubspot.singularity.SingularityDeployBuilder) HealthcheckOptionsBuilder(com.hubspot.deploy.HealthcheckOptionsBuilder) SingularityDeploy(com.hubspot.singularity.SingularityDeploy) Test(org.junit.Test)

Example 40 with SingularityTask

use of com.hubspot.singularity.SingularityTask in project Singularity by HubSpot.

the class SingularityHealthchecksTest method testHealthchecksTimeout.

@Test
public void testHealthchecksTimeout() {
    initRequest();
    final long hourAgo = System.currentTimeMillis() - TimeUnit.HOURS.toMillis(1);
    final String deployId = "timeout_test";
    HealthcheckOptions options = new HealthcheckOptionsBuilder("http://uri").setMaxRetries(Optional.of(2)).build();
    SingularityDeployBuilder db = new SingularityDeployBuilder(requestId, deployId).setHealthcheck(Optional.of(options));
    db.setDeployHealthTimeoutSeconds(Optional.of(TimeUnit.DAYS.toMillis(1)));
    SingularityDeploy deploy = initDeploy(db, hourAgo);
    deployChecker.checkDeploys();
    Assert.assertTrue(!deployManager.getDeployResult(requestId, deployId).isPresent());
    SingularityTask task = launchTask(request, deploy, hourAgo, hourAgo + 1, 1, TaskState.TASK_RUNNING);
    deployChecker.checkDeploys();
    Assert.assertTrue(!deployManager.getDeployResult(requestId, deployId).isPresent());
    taskManager.saveHealthcheckResult(new SingularityTaskHealthcheckResult(Optional.of(503), Optional.of(1000L), hourAgo + 1, Optional.<String>absent(), Optional.<String>absent(), task.getTaskId(), Optional.<Boolean>absent()));
    deployChecker.checkDeploys();
    Assert.assertEquals(DeployState.FAILED, deployManager.getDeployResult(requestId, deployId).get().getDeployState());
}
Also used : SingularityTaskHealthcheckResult(com.hubspot.singularity.SingularityTaskHealthcheckResult) SingularityTask(com.hubspot.singularity.SingularityTask) HealthcheckOptions(com.hubspot.deploy.HealthcheckOptions) SingularityDeployBuilder(com.hubspot.singularity.SingularityDeployBuilder) HealthcheckOptionsBuilder(com.hubspot.deploy.HealthcheckOptionsBuilder) SingularityDeploy(com.hubspot.singularity.SingularityDeploy) Test(org.junit.Test)

Aggregations

SingularityTask (com.hubspot.singularity.SingularityTask)94 Test (org.junit.Test)66 SingularityTaskId (com.hubspot.singularity.SingularityTaskId)33 SingularityDeployBuilder (com.hubspot.singularity.SingularityDeployBuilder)15 SingularityRequest (com.hubspot.singularity.SingularityRequest)15 SingularityDeploy (com.hubspot.singularity.SingularityDeploy)12 SingularityTaskHealthcheckResult (com.hubspot.singularity.SingularityTaskHealthcheckResult)11 HealthcheckOptions (com.hubspot.deploy.HealthcheckOptions)9 HealthcheckOptionsBuilder (com.hubspot.deploy.HealthcheckOptionsBuilder)9 Resources (com.hubspot.mesos.Resources)7 ArrayList (java.util.ArrayList)7 SingularityTaskHistoryUpdate (com.hubspot.singularity.SingularityTaskHistoryUpdate)6 List (java.util.List)6 SingularityDeployProgress (com.hubspot.singularity.SingularityDeployProgress)5 SingularityRunNowRequestBuilder (com.hubspot.singularity.SingularityRunNowRequestBuilder)5 SingularityLoadBalancerUpdate (com.hubspot.singularity.SingularityLoadBalancerUpdate)4 SingularityPendingTask (com.hubspot.singularity.SingularityPendingTask)4 SingularityPendingTaskId (com.hubspot.singularity.SingularityPendingTaskId)4 SingularityRunNowRequest (com.hubspot.singularity.api.SingularityRunNowRequest)4 SingularityPendingRequest (com.hubspot.singularity.SingularityPendingRequest)3