Search in sources :

Example 11 with WorkerDone

use of org.apache.kafka.trogdor.rest.WorkerDone in project kafka by apache.

the class AgentTest method testWorkerCompletions.

@Test
public void testWorkerCompletions() throws Exception {
    MockTime time = new MockTime(0, 0, 0);
    MockScheduler scheduler = new MockScheduler(time);
    Agent agent = createAgent(scheduler);
    AgentClient client = new AgentClient.Builder().maxTries(10).target("localhost", agent.port()).build();
    new ExpectedTasks().waitFor(client);
    SampleTaskSpec fooSpec = new SampleTaskSpec(0, 900000, Collections.singletonMap("node01", 1L), "");
    client.createWorker(new CreateWorkerRequest(0, "foo", fooSpec));
    new ExpectedTasks().addTask(new ExpectedTaskBuilder("foo").workerState(new WorkerRunning("foo", fooSpec, 0, new TextNode("active"))).build()).waitFor(client);
    SampleTaskSpec barSpec = new SampleTaskSpec(0, 900000, Collections.singletonMap("node01", 2L), "baz");
    client.createWorker(new CreateWorkerRequest(1, "bar", barSpec));
    time.sleep(1);
    new ExpectedTasks().addTask(new ExpectedTaskBuilder("foo").workerState(new WorkerDone("foo", fooSpec, 0, 1, new TextNode("halted"), "")).build()).addTask(new ExpectedTaskBuilder("bar").workerState(new WorkerRunning("bar", barSpec, 0, new TextNode("active"))).build()).waitFor(client);
    time.sleep(1);
    new ExpectedTasks().addTask(new ExpectedTaskBuilder("foo").workerState(new WorkerDone("foo", fooSpec, 0, 1, new TextNode("halted"), "")).build()).addTask(new ExpectedTaskBuilder("bar").workerState(new WorkerDone("bar", barSpec, 0, 2, new TextNode("halted"), "baz")).build()).waitFor(client);
}
Also used : MockScheduler(org.apache.kafka.common.utils.MockScheduler) ExpectedTasks(org.apache.kafka.trogdor.common.ExpectedTasks) WorkerDone(org.apache.kafka.trogdor.rest.WorkerDone) WorkerRunning(org.apache.kafka.trogdor.rest.WorkerRunning) SampleTaskSpec(org.apache.kafka.trogdor.task.SampleTaskSpec) TextNode(com.fasterxml.jackson.databind.node.TextNode) CreateWorkerRequest(org.apache.kafka.trogdor.rest.CreateWorkerRequest) ExpectedTaskBuilder(org.apache.kafka.trogdor.common.ExpectedTasks.ExpectedTaskBuilder) MockTime(org.apache.kafka.common.utils.MockTime) Test(org.junit.jupiter.api.Test)

Example 12 with WorkerDone

use of org.apache.kafka.trogdor.rest.WorkerDone in project kafka by apache.

the class AgentTest method testCreateExpiredWorkerIsNotScheduled.

@Test
public void testCreateExpiredWorkerIsNotScheduled() throws Exception {
    long initialTimeMs = 100;
    long tickMs = 15;
    final boolean[] toSleep = { true };
    MockTime time = new MockTime(tickMs, initialTimeMs, 0) {

        /**
         * Modify sleep() to call super.sleep() every second call
         * in order to avoid the endless loop in the tick() calls to the MockScheduler listener
         */
        @Override
        public void sleep(long ms) {
            toSleep[0] = !toSleep[0];
            if (toSleep[0])
                super.sleep(ms);
        }
    };
    MockScheduler scheduler = new MockScheduler(time);
    Agent agent = createAgent(scheduler);
    AgentClient client = new AgentClient.Builder().maxTries(10).target("localhost", agent.port()).build();
    AgentStatusResponse status = client.status();
    assertEquals(Collections.emptyMap(), status.workers());
    new ExpectedTasks().waitFor(client);
    final NoOpTaskSpec fooSpec = new NoOpTaskSpec(10, 10);
    client.createWorker(new CreateWorkerRequest(0, "foo", fooSpec));
    long actualStartTimeMs = initialTimeMs + tickMs;
    long doneMs = actualStartTimeMs + 2 * tickMs;
    new ExpectedTasks().addTask(new ExpectedTaskBuilder("foo").workerState(new WorkerDone("foo", fooSpec, actualStartTimeMs, doneMs, null, "worker expired")).taskState(new TaskDone(fooSpec, actualStartTimeMs, doneMs, "worker expired", false, null)).build()).waitFor(client);
}
Also used : MockScheduler(org.apache.kafka.common.utils.MockScheduler) WorkerDone(org.apache.kafka.trogdor.rest.WorkerDone) TaskDone(org.apache.kafka.trogdor.rest.TaskDone) ExpectedTaskBuilder(org.apache.kafka.trogdor.common.ExpectedTasks.ExpectedTaskBuilder) NoOpTaskSpec(org.apache.kafka.trogdor.task.NoOpTaskSpec) AgentStatusResponse(org.apache.kafka.trogdor.rest.AgentStatusResponse) ExpectedTasks(org.apache.kafka.trogdor.common.ExpectedTasks) CreateWorkerRequest(org.apache.kafka.trogdor.rest.CreateWorkerRequest) MockTime(org.apache.kafka.common.utils.MockTime) Test(org.junit.jupiter.api.Test)

Example 13 with WorkerDone

use of org.apache.kafka.trogdor.rest.WorkerDone in project kafka by apache.

the class AgentTest method testDestroyWorkers.

@Test
public void testDestroyWorkers() throws Exception {
    MockTime time = new MockTime(0, 0, 0);
    MockScheduler scheduler = new MockScheduler(time);
    Agent agent = createAgent(scheduler);
    AgentClient client = new AgentClient.Builder().maxTries(10).target("localhost", agent.port()).build();
    new ExpectedTasks().waitFor(client);
    final NoOpTaskSpec fooSpec = new NoOpTaskSpec(0, 5);
    client.createWorker(new CreateWorkerRequest(0, "foo", fooSpec));
    new ExpectedTasks().addTask(new ExpectedTaskBuilder("foo").workerState(new WorkerRunning("foo", fooSpec, 0, new TextNode("active"))).build()).waitFor(client);
    time.sleep(1);
    client.destroyWorker(new DestroyWorkerRequest(0));
    client.destroyWorker(new DestroyWorkerRequest(0));
    client.destroyWorker(new DestroyWorkerRequest(1));
    new ExpectedTasks().waitFor(client);
    time.sleep(1);
    final NoOpTaskSpec fooSpec2 = new NoOpTaskSpec(2, 1);
    client.createWorker(new CreateWorkerRequest(1, "foo", fooSpec2));
    new ExpectedTasks().addTask(new ExpectedTaskBuilder("foo").workerState(new WorkerRunning("foo", fooSpec2, 2, new TextNode("active"))).build()).waitFor(client);
    time.sleep(2);
    new ExpectedTasks().addTask(new ExpectedTaskBuilder("foo").workerState(new WorkerDone("foo", fooSpec2, 2, 4, new TextNode("done"), "")).build()).waitFor(client);
    time.sleep(1);
    client.destroyWorker(new DestroyWorkerRequest(1));
    new ExpectedTasks().waitFor(client);
    agent.beginShutdown();
    agent.waitForShutdown();
}
Also used : MockScheduler(org.apache.kafka.common.utils.MockScheduler) WorkerDone(org.apache.kafka.trogdor.rest.WorkerDone) WorkerRunning(org.apache.kafka.trogdor.rest.WorkerRunning) TextNode(com.fasterxml.jackson.databind.node.TextNode) ExpectedTaskBuilder(org.apache.kafka.trogdor.common.ExpectedTasks.ExpectedTaskBuilder) NoOpTaskSpec(org.apache.kafka.trogdor.task.NoOpTaskSpec) DestroyWorkerRequest(org.apache.kafka.trogdor.rest.DestroyWorkerRequest) ExpectedTasks(org.apache.kafka.trogdor.common.ExpectedTasks) CreateWorkerRequest(org.apache.kafka.trogdor.rest.CreateWorkerRequest) MockTime(org.apache.kafka.common.utils.MockTime) Test(org.junit.jupiter.api.Test)

Example 14 with WorkerDone

use of org.apache.kafka.trogdor.rest.WorkerDone in project kafka by apache.

the class JsonSerializationTest method testDeserializationDoesNotProduceNulls.

@Test
public void testDeserializationDoesNotProduceNulls() throws Exception {
    verify(new FilesUnreadableFaultSpec(0, 0, null, null, null, 0));
    verify(new Kibosh.KiboshControlFile(null));
    verify(new NetworkPartitionFaultSpec(0, 0, null));
    verify(new ProcessStopFaultSpec(0, 0, null, null));
    verify(new AgentStatusResponse(0, null));
    verify(new TasksResponse(null));
    verify(new WorkerDone(null, null, 0, 0, null, null));
    verify(new WorkerRunning(null, null, 0, null));
    verify(new WorkerStopping(null, null, 0, null));
    verify(new ProduceBenchSpec(0, 0, null, null, 0, 0, null, null, Optional.empty(), null, null, null, null, null, false, false));
    verify(new RoundTripWorkloadSpec(0, 0, null, null, null, null, null, null, 0, null, null, 0));
    verify(new TopicsSpec());
    verify(new PartitionsSpec(0, (short) 0, null, null));
    Map<Integer, List<Integer>> partitionAssignments = new HashMap<Integer, List<Integer>>();
    partitionAssignments.put(0, Arrays.asList(1, 2, 3));
    partitionAssignments.put(1, Arrays.asList(1, 2, 3));
    verify(new PartitionsSpec(0, (short) 0, partitionAssignments, null));
    verify(new PartitionsSpec(0, (short) 0, null, null));
}
Also used : WorkerDone(org.apache.kafka.trogdor.rest.WorkerDone) WorkerStopping(org.apache.kafka.trogdor.rest.WorkerStopping) TasksResponse(org.apache.kafka.trogdor.rest.TasksResponse) WorkerRunning(org.apache.kafka.trogdor.rest.WorkerRunning) HashMap(java.util.HashMap) NetworkPartitionFaultSpec(org.apache.kafka.trogdor.fault.NetworkPartitionFaultSpec) RoundTripWorkloadSpec(org.apache.kafka.trogdor.workload.RoundTripWorkloadSpec) Kibosh(org.apache.kafka.trogdor.fault.Kibosh) AgentStatusResponse(org.apache.kafka.trogdor.rest.AgentStatusResponse) ProcessStopFaultSpec(org.apache.kafka.trogdor.fault.ProcessStopFaultSpec) ProduceBenchSpec(org.apache.kafka.trogdor.workload.ProduceBenchSpec) PartitionsSpec(org.apache.kafka.trogdor.workload.PartitionsSpec) FilesUnreadableFaultSpec(org.apache.kafka.trogdor.fault.FilesUnreadableFaultSpec) List(java.util.List) TopicsSpec(org.apache.kafka.trogdor.workload.TopicsSpec) Test(org.junit.jupiter.api.Test)

Example 15 with WorkerDone

use of org.apache.kafka.trogdor.rest.WorkerDone in project kafka by apache.

the class CoordinatorTest method testAgentFailureAndTaskExpiry.

/**
 * If an agent fails in the middle of a task and comes back up when the task is considered expired,
 * we want the task to be marked as DONE and not re-sent should a second failure happen.
 */
@Test
public void testAgentFailureAndTaskExpiry() throws Exception {
    MockTime time = new MockTime(0, 0, 0);
    Scheduler scheduler = new MockScheduler(time);
    try (MiniTrogdorCluster cluster = new MiniTrogdorCluster.Builder().addCoordinator("node01").addAgent("node02").scheduler(scheduler).build()) {
        CoordinatorClient coordinatorClient = cluster.coordinatorClient();
        NoOpTaskSpec fooSpec = new NoOpTaskSpec(1, 500);
        coordinatorClient.createTask(new CreateTaskRequest("foo", fooSpec));
        TaskState expectedState = new ExpectedTaskBuilder("foo").taskState(new TaskPending(fooSpec)).build().taskState();
        TaskState resp = coordinatorClient.task(new TaskRequest("foo"));
        assertEquals(expectedState, resp);
        time.sleep(2);
        new ExpectedTasks().addTask(new ExpectedTaskBuilder("foo").taskState(new TaskRunning(fooSpec, 2, new TextNode("active"))).workerState(new WorkerRunning("foo", fooSpec, 2, new TextNode("active"))).build()).waitFor(coordinatorClient).waitFor(cluster.agentClient("node02"));
        cluster.restartAgent("node02");
        time.sleep(550);
        // coordinator heartbeat sees that the agent is back up, re-schedules the task but the agent expires it
        new ExpectedTasks().addTask(new ExpectedTaskBuilder("foo").taskState(new TaskDone(fooSpec, 2, 552, "worker expired", false, null)).workerState(new WorkerDone("foo", fooSpec, 552, 552, null, "worker expired")).build()).waitFor(coordinatorClient).waitFor(cluster.agentClient("node02"));
        cluster.restartAgent("node02");
        // coordinator heartbeat sees that the agent is back up but does not re-schedule the task as it is DONE
        new ExpectedTasks().addTask(new ExpectedTaskBuilder("foo").taskState(new TaskDone(fooSpec, 2, 552, "worker expired", false, null)).build()).waitFor(coordinatorClient).waitFor(cluster.agentClient("node02"));
    }
}
Also used : MockScheduler(org.apache.kafka.common.utils.MockScheduler) WorkerDone(org.apache.kafka.trogdor.rest.WorkerDone) WorkerRunning(org.apache.kafka.trogdor.rest.WorkerRunning) MockScheduler(org.apache.kafka.common.utils.MockScheduler) Scheduler(org.apache.kafka.common.utils.Scheduler) ExpectedTaskBuilder(org.apache.kafka.trogdor.common.ExpectedTasks.ExpectedTaskBuilder) TaskDone(org.apache.kafka.trogdor.rest.TaskDone) TextNode(com.fasterxml.jackson.databind.node.TextNode) DestroyTaskRequest(org.apache.kafka.trogdor.rest.DestroyTaskRequest) StopTaskRequest(org.apache.kafka.trogdor.rest.StopTaskRequest) CreateTaskRequest(org.apache.kafka.trogdor.rest.CreateTaskRequest) TaskRequest(org.apache.kafka.trogdor.rest.TaskRequest) ExpectedTaskBuilder(org.apache.kafka.trogdor.common.ExpectedTasks.ExpectedTaskBuilder) NoOpTaskSpec(org.apache.kafka.trogdor.task.NoOpTaskSpec) TaskPending(org.apache.kafka.trogdor.rest.TaskPending) ExpectedTasks(org.apache.kafka.trogdor.common.ExpectedTasks) CreateTaskRequest(org.apache.kafka.trogdor.rest.CreateTaskRequest) MiniTrogdorCluster(org.apache.kafka.trogdor.common.MiniTrogdorCluster) TaskRunning(org.apache.kafka.trogdor.rest.TaskRunning) TaskState(org.apache.kafka.trogdor.rest.TaskState) MockTime(org.apache.kafka.common.utils.MockTime) Test(org.junit.jupiter.api.Test)

Aggregations

WorkerDone (org.apache.kafka.trogdor.rest.WorkerDone)16 WorkerRunning (org.apache.kafka.trogdor.rest.WorkerRunning)15 MockScheduler (org.apache.kafka.common.utils.MockScheduler)14 MockTime (org.apache.kafka.common.utils.MockTime)14 ExpectedTasks (org.apache.kafka.trogdor.common.ExpectedTasks)14 ExpectedTaskBuilder (org.apache.kafka.trogdor.common.ExpectedTasks.ExpectedTaskBuilder)14 Test (org.junit.jupiter.api.Test)10 NoOpTaskSpec (org.apache.kafka.trogdor.task.NoOpTaskSpec)9 TextNode (com.fasterxml.jackson.databind.node.TextNode)8 CreateWorkerRequest (org.apache.kafka.trogdor.rest.CreateWorkerRequest)8 TaskDone (org.apache.kafka.trogdor.rest.TaskDone)7 Scheduler (org.apache.kafka.common.utils.Scheduler)6 MiniTrogdorCluster (org.apache.kafka.trogdor.common.MiniTrogdorCluster)6 CreateTaskRequest (org.apache.kafka.trogdor.rest.CreateTaskRequest)6 TaskPending (org.apache.kafka.trogdor.rest.TaskPending)6 TaskRunning (org.apache.kafka.trogdor.rest.TaskRunning)6 Test (org.junit.Test)6 AgentClient (org.apache.kafka.trogdor.agent.AgentClient)4 FilesUnreadableFaultSpec (org.apache.kafka.trogdor.fault.FilesUnreadableFaultSpec)4 StopWorkerRequest (org.apache.kafka.trogdor.rest.StopWorkerRequest)4