Search in sources :

Example 16 with JetInstance

use of com.hazelcast.jet.JetInstance in project hazelcast-jet by hazelcast.

the class SplitBrainTest method when_newMemberJoinsToCluster_then_jobQuorumSizeIsUpdated.

@Test
public void when_newMemberJoinsToCluster_then_jobQuorumSizeIsUpdated() {
    int clusterSize = 3;
    JetConfig jetConfig = new JetConfig();
    JetInstance[] instances = new JetInstance[clusterSize];
    for (int i = 0; i < clusterSize; i++) {
        instances[i] = createJetMember(jetConfig);
    }
    StuckProcessor.executionStarted = new CountDownLatch(clusterSize * PARALLELISM);
    MockPS processorSupplier = new MockPS(StuckProcessor::new, clusterSize);
    DAG dag = new DAG().vertex(new Vertex("test", processorSupplier));
    Job job = instances[0].newJob(dag, new JobConfig().setSplitBrainProtection(true));
    assertOpenEventually(StuckProcessor.executionStarted);
    createJetMember(jetConfig);
    assertTrueEventually(() -> {
        JobRepository jobRepository = getJetService(instances[0]).getJobRepository();
        JobRecord jobRecord = jobRepository.getJobRecord(job.getId());
        assertEquals(3, jobRecord.getQuorumSize());
    });
    StuckProcessor.proceedLatch.countDown();
}
Also used : MockPS(com.hazelcast.jet.core.TestProcessors.MockPS) JetInstance(com.hazelcast.jet.JetInstance) CountDownLatch(java.util.concurrent.CountDownLatch) JobRepository(com.hazelcast.jet.impl.JobRepository) JetConfig(com.hazelcast.jet.config.JetConfig) JobConfig(com.hazelcast.jet.config.JobConfig) JobRecord(com.hazelcast.jet.impl.JobRecord) StuckProcessor(com.hazelcast.jet.core.TestProcessors.StuckProcessor) Job(com.hazelcast.jet.Job) Test(org.junit.Test)

Example 17 with JetInstance

use of com.hazelcast.jet.JetInstance in project hazelcast-jet by hazelcast.

the class SplitBrainTest method when_newMemberIsAddedAfterClusterSizeFallsBelowQuorumSize_then_jobRestartDoesNotSucceed.

@Test
public void when_newMemberIsAddedAfterClusterSizeFallsBelowQuorumSize_then_jobRestartDoesNotSucceed() {
    int clusterSize = 5;
    JetConfig jetConfig = new JetConfig();
    JetInstance[] instances = new JetInstance[clusterSize];
    for (int i = 0; i < clusterSize; i++) {
        instances[i] = createJetMember(jetConfig);
    }
    StuckProcessor.executionStarted = new CountDownLatch(clusterSize * PARALLELISM);
    MockPS processorSupplier = new MockPS(StuckProcessor::new, clusterSize);
    DAG dag = new DAG().vertex(new Vertex("test", processorSupplier));
    Job job = instances[0].newJob(dag, new JobConfig().setSplitBrainProtection(true));
    assertOpenEventually(StuckProcessor.executionStarted);
    for (int i = 1; i < clusterSize; i++) {
        instances[i].getHazelcastInstance().getLifecycleService().terminate();
    }
    StuckProcessor.proceedLatch.countDown();
    assertTrueEventually(() -> assertEquals(RESTARTING, job.getStatus()));
    createJetMember(jetConfig);
    assertTrueAllTheTime(() -> assertEquals(RESTARTING, job.getStatus()), 5);
}
Also used : MockPS(com.hazelcast.jet.core.TestProcessors.MockPS) JetInstance(com.hazelcast.jet.JetInstance) StuckProcessor(com.hazelcast.jet.core.TestProcessors.StuckProcessor) CountDownLatch(java.util.concurrent.CountDownLatch) Job(com.hazelcast.jet.Job) JetConfig(com.hazelcast.jet.config.JetConfig) JobConfig(com.hazelcast.jet.config.JobConfig) Test(org.junit.Test)

Example 18 with JetInstance

use of com.hazelcast.jet.JetInstance in project hazelcast-jet by hazelcast.

the class TopologyChangeTest method when_addAndRemoveNodeDuringExecution_then_completeSuccessfully.

@Test
public void when_addAndRemoveNodeDuringExecution_then_completeSuccessfully() throws Throwable {
    // Given
    DAG dag = new DAG().vertex(new Vertex("test", new MockPS(StuckProcessor::new, nodeCount)));
    // When
    Job job = instances[0].newJob(dag);
    StuckProcessor.executionStarted.await();
    JetInstance instance = createJetMember();
    instance.shutdown();
    StuckProcessor.proceedLatch.countDown();
    job.join();
    // Then
    assertEquals(nodeCount, MockPS.initCount.get());
    assertTrueEventually(() -> {
        assertEquals(nodeCount, MockPS.closeCount.get());
        assertThat(MockPS.receivedCloseErrors, empty());
    });
}
Also used : MockPS(com.hazelcast.jet.core.TestProcessors.MockPS) JetInstance(com.hazelcast.jet.JetInstance) StuckProcessor(com.hazelcast.jet.core.TestProcessors.StuckProcessor) Job(com.hazelcast.jet.Job) Test(org.junit.Test)

Example 19 with JetInstance

use of com.hazelcast.jet.JetInstance in project hazelcast-jet by hazelcast.

the class TopologyChangeTest method when_coordinatorLeavesDuringExecution_then_clientStillGetsJobResult.

@Test
public void when_coordinatorLeavesDuringExecution_then_clientStillGetsJobResult() throws Throwable {
    // Given
    JetInstance client = createJetClient();
    DAG dag = new DAG().vertex(new Vertex("test", new MockPS(StuckProcessor::new, nodeCount)));
    // When
    Job job = client.newJob(dag);
    StuckProcessor.executionStarted.await();
    instances[0].getHazelcastInstance().getLifecycleService().terminate();
    StuckProcessor.proceedLatch.countDown();
    // Then
    job.join();
}
Also used : MockPS(com.hazelcast.jet.core.TestProcessors.MockPS) JetInstance(com.hazelcast.jet.JetInstance) StuckProcessor(com.hazelcast.jet.core.TestProcessors.StuckProcessor) Job(com.hazelcast.jet.Job) Test(org.junit.Test)

Example 20 with JetInstance

use of com.hazelcast.jet.JetInstance in project hazelcast-jet by hazelcast.

the class TopologyChangeTest method when_nodeIsShuttingDownAfterInit_then_jobRestarts.

@Test
public void when_nodeIsShuttingDownAfterInit_then_jobRestarts() {
    // Given that the second node has not received ExecuteOperation yet
    for (JetInstance instance : instances) {
        warmUpPartitions(instance.getHazelcastInstance());
    }
    rejectOperationsBetween(instances[0].getHazelcastInstance(), instances[2].getHazelcastInstance(), JetInitDataSerializerHook.FACTORY_ID, singletonList(START_EXECUTION_OP));
    DAG dag = new DAG().vertex(new Vertex("test", new MockPS(TestProcessors.Identity::new, nodeCount - 1)));
    Job job = instances[0].newJob(dag);
    assertTrueEventually(() -> assertEquals(RUNNING, job.getStatus()));
    // When a participant shuts down during execution
    instances[2].shutdown();
    // Then, the job restarts and successfully completes
    job.join();
}
Also used : MockPS(com.hazelcast.jet.core.TestProcessors.MockPS) JetInstance(com.hazelcast.jet.JetInstance) Job(com.hazelcast.jet.Job) Test(org.junit.Test)

Aggregations

JetInstance (com.hazelcast.jet.JetInstance)84 Test (org.junit.Test)45 Job (com.hazelcast.jet.Job)32 JetConfig (com.hazelcast.jet.config.JetConfig)22 MockPS (com.hazelcast.jet.core.TestProcessors.MockPS)14 Pipeline (com.hazelcast.jet.pipeline.Pipeline)14 JobConfig (com.hazelcast.jet.config.JobConfig)13 StuckProcessor (com.hazelcast.jet.core.TestProcessors.StuckProcessor)12 DAG (com.hazelcast.jet.core.DAG)10 ExpectedException (org.junit.rules.ExpectedException)10 Jet (com.hazelcast.jet.Jet)9 Assert.assertEquals (org.junit.Assert.assertEquals)9 JetService (com.hazelcast.jet.impl.JetService)8 Map (java.util.Map)8 CountDownLatch (java.util.concurrent.CountDownLatch)8 JobRepository (com.hazelcast.jet.impl.JobRepository)7 Sources (com.hazelcast.jet.pipeline.Sources)7 CancellationException (java.util.concurrent.CancellationException)7 Assert.assertNotNull (org.junit.Assert.assertNotNull)7 Before (org.junit.Before)7