Search in sources :

Example 91 with CancellationException

use of java.util.concurrent.CancellationException in project hazelcast-jet by hazelcast.

the class StartExecutionOperation method doRun.

@Override
protected void doRun() throws Exception {
    ExecutionContext execCtx = getExecutionCtx();
    Address coordinator = getCallerAddress();
    getLogger().info("Start execution of " + jobAndExecutionId(jobId(), executionId) + " from coordinator " + coordinator);
    execCtx.beginExecution().whenComplete(withTryCatch(getLogger(), (i, e) -> {
        if (e instanceof CancellationException) {
            getLogger().fine("Execution of " + jobAndExecutionId(jobId(), executionId) + " was cancelled");
        } else if (e != null) {
            getLogger().fine("Execution of " + jobAndExecutionId(jobId(), executionId) + " completed with failure", e);
        } else {
            getLogger().fine("Execution of " + jobAndExecutionId(jobId(), executionId) + " completed");
        }
        doSendResponse(e);
    }));
}
Also used : ExecutionContext(com.hazelcast.jet.impl.execution.ExecutionContext) JetService(com.hazelcast.jet.impl.JetService) CancellationException(java.util.concurrent.CancellationException) ExceptionUtil.withTryCatch(com.hazelcast.jet.impl.util.ExceptionUtil.withTryCatch) Address(com.hazelcast.nio.Address) ObjectDataOutput(com.hazelcast.nio.ObjectDataOutput) IOException(java.io.IOException) Util.jobAndExecutionId(com.hazelcast.jet.impl.util.Util.jobAndExecutionId) ObjectDataInput(com.hazelcast.nio.ObjectDataInput) JetInitDataSerializerHook(com.hazelcast.jet.impl.execution.init.JetInitDataSerializerHook) ExecutionContext(com.hazelcast.jet.impl.execution.ExecutionContext) Address(com.hazelcast.nio.Address) CancellationException(java.util.concurrent.CancellationException)

Example 92 with CancellationException

use of java.util.concurrent.CancellationException in project hazelcast-jet by hazelcast.

the class MasterContext method getExecuteResult.

/**
 * <ul>
 * <li>Returns null if there is no failure.
 * <li>Returns CancellationException if the job is cancelled.
 * <li>Returns JobRestartRequestedException if the current execution is cancelled
 * <li>If there is at least one non-restartable failure, such as an exception in user code, then returns that failure.
 * <li>Otherwise, the failure is because a job participant has left the cluster.
 *   In that case, {@code TopologyChangeException} is returned so that the job will be restarted.
 * </ul>
 */
private Throwable getExecuteResult(Map<MemberInfo, Object> responses, boolean isRestartRequested) {
    if (cancellationToken.isCompleted()) {
        logger.fine(jobIdString() + " to be cancelled after execute");
        return new CancellationException();
    } else if (isRestartRequested) {
        return new JobRestartRequestedException();
    }
    Map<Boolean, List<Entry<MemberInfo, Object>>> grouped = groupResponses(responses);
    Collection<MemberInfo> successfulMembers = grouped.get(false).stream().map(Entry::getKey).collect(toList());
    if (successfulMembers.size() == executionPlanMap.size()) {
        logger.fine("Execute of " + jobIdString() + " is successful.");
        return null;
    }
    List<Entry<MemberInfo, Object>> failures = grouped.get(true);
    logger.fine("Execute of " + jobIdString() + " has failures: " + failures);
    // In that case, all remaining participants return a CancellationException.
    return failures.stream().map(e -> (Throwable) e.getValue()).filter(t -> !(t instanceof CancellationException || isTopologicalFailure(t))).findFirst().map(ExceptionUtil::peel).orElse(new TopologyChangedException());
}
Also used : NO_SNAPSHOT(com.hazelcast.jet.impl.execution.SnapshotContext.NO_SNAPSHOT) SnapshotRepository.snapshotDataMapName(com.hazelcast.jet.impl.SnapshotRepository.snapshotDataMapName) NonCompletableFuture(com.hazelcast.jet.impl.util.NonCompletableFuture) Address(com.hazelcast.nio.Address) Util.jobAndExecutionId(com.hazelcast.jet.impl.util.Util.jobAndExecutionId) Processors.mapP(com.hazelcast.jet.core.processor.Processors.mapP) SourceProcessors.readMapP(com.hazelcast.jet.core.processor.SourceProcessors.readMapP) CompletionToken(com.hazelcast.jet.impl.util.CompletionToken) Util.idToString(com.hazelcast.jet.impl.util.Util.idToString) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) MemberInfo(com.hazelcast.internal.cluster.MemberInfo) Map(java.util.Map) STARTING(com.hazelcast.jet.core.JobStatus.STARTING) DAG(com.hazelcast.jet.core.DAG) JobStatus(com.hazelcast.jet.core.JobStatus) ExceptionUtil(com.hazelcast.jet.impl.util.ExceptionUtil) ExecutionService(com.hazelcast.spi.ExecutionService) Operation(com.hazelcast.spi.Operation) CancellationException(java.util.concurrent.CancellationException) Collections.emptyList(java.util.Collections.emptyList) Collection(java.util.Collection) Util.getJetInstance(com.hazelcast.jet.impl.util.Util.getJetInstance) JobConfig(com.hazelcast.jet.config.JobConfig) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) Set(java.util.Set) Collectors(java.util.stream.Collectors) BroadcastKey(com.hazelcast.jet.core.BroadcastKey) List(java.util.List) ExecutionCallback(com.hazelcast.core.ExecutionCallback) ExecutionPlan(com.hazelcast.jet.impl.execution.init.ExecutionPlan) Entry(java.util.Map.Entry) CancelExecutionOperation(com.hazelcast.jet.impl.operation.CancelExecutionOperation) TopologyChangedException(com.hazelcast.jet.core.TopologyChangedException) COMPLETED(com.hazelcast.jet.core.JobStatus.COMPLETED) InternalCompletableFuture(com.hazelcast.spi.InternalCompletableFuture) ExecutionPlanBuilder.createExecutionPlans(com.hazelcast.jet.impl.execution.init.ExecutionPlanBuilder.createExecutionPlans) Collectors.partitioningBy(java.util.stream.Collectors.partitioningBy) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) HashMap(java.util.HashMap) CompletableFuture(java.util.concurrent.CompletableFuture) StartExecutionOperation(com.hazelcast.jet.impl.operation.StartExecutionOperation) AtomicReference(java.util.concurrent.atomic.AtomicReference) Function(java.util.function.Function) HashSet(java.util.HashSet) InitExecutionOperation(com.hazelcast.jet.impl.operation.InitExecutionOperation) ILogger(com.hazelcast.logging.ILogger) ExceptionUtil.withTryCatch(com.hazelcast.jet.impl.util.ExceptionUtil.withTryCatch) NOT_STARTED(com.hazelcast.jet.core.JobStatus.NOT_STARTED) MembersView(com.hazelcast.internal.cluster.impl.MembersView) DistributedFunction(com.hazelcast.jet.function.DistributedFunction) Edge(com.hazelcast.jet.core.Edge) ClusterServiceImpl(com.hazelcast.internal.cluster.impl.ClusterServiceImpl) Nullable(javax.annotation.Nullable) NodeEngineImpl(com.hazelcast.spi.impl.NodeEngineImpl) RESTARTING(com.hazelcast.jet.core.JobStatus.RESTARTING) BroadcastEntry(com.hazelcast.jet.impl.execution.BroadcastEntry) ExceptionUtil.isTopologicalFailure(com.hazelcast.jet.impl.util.ExceptionUtil.isTopologicalFailure) DistributedFunctions.entryKey(com.hazelcast.jet.function.DistributedFunctions.entryKey) Consumer(java.util.function.Consumer) Vertex(com.hazelcast.jet.core.Vertex) Collectors.toList(java.util.stream.Collectors.toList) CustomClassLoadedObject.deserializeWithCustomClassLoader(com.hazelcast.jet.impl.execution.init.CustomClassLoadedObject.deserializeWithCustomClassLoader) CompleteExecutionOperation(com.hazelcast.jet.impl.operation.CompleteExecutionOperation) ExceptionUtil.peel(com.hazelcast.jet.impl.util.ExceptionUtil.peel) FAILED(com.hazelcast.jet.core.JobStatus.FAILED) RUNNING(com.hazelcast.jet.core.JobStatus.RUNNING) ProcessingGuarantee(com.hazelcast.jet.config.ProcessingGuarantee) JobRestartRequestedException(com.hazelcast.jet.impl.exception.JobRestartRequestedException) SnapshotOperation(com.hazelcast.jet.impl.operation.SnapshotOperation) Edge.between(com.hazelcast.jet.core.Edge.between) Entry(java.util.Map.Entry) BroadcastEntry(com.hazelcast.jet.impl.execution.BroadcastEntry) MemberInfo(com.hazelcast.internal.cluster.MemberInfo) CancellationException(java.util.concurrent.CancellationException) JobRestartRequestedException(com.hazelcast.jet.impl.exception.JobRestartRequestedException) Collections.emptyList(java.util.Collections.emptyList) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) TopologyChangedException(com.hazelcast.jet.core.TopologyChangedException)

Example 93 with CancellationException

use of java.util.concurrent.CancellationException in project hazelcast-jet by hazelcast.

the class ExecutionLifecycleTest method when_executionCancelled_then_jobCompletedWithCancellationException.

@Test
public void when_executionCancelled_then_jobCompletedWithCancellationException() throws Throwable {
    // Given
    DAG dag = new DAG().vertex(new Vertex("test", new MockPMS(() -> new MockPS(StuckProcessor::new, NODE_COUNT))));
    // When
    Job job = instance.newJob(dag);
    try {
        StuckProcessor.executionStarted.await();
        job.cancel();
        job.join();
        fail("Job execution should fail");
    } catch (CancellationException ignored) {
    }
    assertTrueEventually(() -> {
        assertJobFailed(job, new CancellationException());
        assertPsClosedWithError(new CancellationException());
        assertPmsClosedWithError(new CancellationException());
    });
}
Also used : MockPMS(com.hazelcast.jet.core.TestProcessors.MockPMS) MockPS(com.hazelcast.jet.core.TestProcessors.MockPS) CancellationException(java.util.concurrent.CancellationException) Job(com.hazelcast.jet.Job) Test(org.junit.Test)

Example 94 with CancellationException

use of java.util.concurrent.CancellationException in project hazelcast-jet by hazelcast.

the class ExecutionLifecycleTest method assertJobFailed.

private void assertJobFailed(Job job, Throwable e) {
    JobResult jobResult = getJobResult(job);
    assertFalse("jobResult.isSuccessful", jobResult.isSuccessful());
    assertExceptionInCauses(e, jobResult.getFailure());
    JobStatus expectedStatus = e instanceof CancellationException ? JobStatus.COMPLETED : JobStatus.FAILED;
    assertEquals("jobStatus", expectedStatus, job.getStatus());
}
Also used : JobResult(com.hazelcast.jet.impl.JobResult) CancellationException(java.util.concurrent.CancellationException)

Example 95 with CancellationException

use of java.util.concurrent.CancellationException in project hazelcast-jet by hazelcast.

the class SplitBrainTest method when_quorumIsLostOnMinority_then_jobRestartsUntilMerge.

@Test
public void when_quorumIsLostOnMinority_then_jobRestartsUntilMerge() {
    int firstSubClusterSize = 3;
    int secondSubClusterSize = 2;
    int clusterSize = firstSubClusterSize + secondSubClusterSize;
    StuckProcessor.executionStarted = new CountDownLatch(clusterSize * PARALLELISM);
    Job[] jobRef = new Job[1];
    Consumer<JetInstance[]> beforeSplit = instances -> {
        MockPS processorSupplier = new MockPS(StuckProcessor::new, clusterSize);
        DAG dag = new DAG().vertex(new Vertex("test", processorSupplier));
        jobRef[0] = instances[0].newJob(dag, new JobConfig().setSplitBrainProtection(true));
        assertOpenEventually(StuckProcessor.executionStarted);
    };
    Future[] minorityJobFutureRef = new Future[1];
    BiConsumer<JetInstance[], JetInstance[]> onSplit = (firstSubCluster, secondSubCluster) -> {
        StuckProcessor.proceedLatch.countDown();
        assertTrueEventually(() -> assertEquals(clusterSize + firstSubClusterSize, MockPS.initCount.get()));
        long jobId = jobRef[0].getId();
        assertTrueEventually(() -> {
            JetService service = getJetService(firstSubCluster[0]);
            assertEquals(COMPLETED, service.getJobCoordinationService().getJobStatus(jobId));
        });
        JetService service2 = getJetService(secondSubCluster[0]);
        assertTrueEventually(() -> {
            assertEquals(STARTING, service2.getJobCoordinationService().getJobStatus(jobId));
        });
        MasterContext masterContext = service2.getJobCoordinationService().getMasterContext(jobId);
        assertNotNull(masterContext);
        minorityJobFutureRef[0] = masterContext.completionFuture();
        assertTrueAllTheTime(() -> {
            assertEquals(STARTING, service2.getJobCoordinationService().getJobStatus(jobId));
        }, 20);
    };
    Consumer<JetInstance[]> afterMerge = instances -> {
        assertTrueEventually(() -> {
            assertEquals(clusterSize + firstSubClusterSize, MockPS.initCount.get());
            assertEquals(clusterSize + firstSubClusterSize, MockPS.closeCount.get());
        });
        assertEquals(clusterSize, MockPS.receivedCloseErrors.size());
        MockPS.receivedCloseErrors.forEach(t -> assertTrue(t instanceof TopologyChangedException));
        try {
            minorityJobFutureRef[0].get();
            fail();
        } catch (CancellationException ignored) {
        } catch (Exception e) {
            throw new AssertionError(e);
        }
    };
    testSplitBrain(firstSubClusterSize, secondSubClusterSize, beforeSplit, onSplit, afterMerge);
}
Also used : MasterContext(com.hazelcast.jet.impl.MasterContext) JetInstance(com.hazelcast.jet.JetInstance) RunWith(org.junit.runner.RunWith) HazelcastSerialClassRunner(com.hazelcast.test.HazelcastSerialClassRunner) Future(java.util.concurrent.Future) STARTING(com.hazelcast.jet.core.JobStatus.STARTING) BiConsumer(java.util.function.BiConsumer) Assert.fail(org.junit.Assert.fail) ExpectedException(org.junit.rules.ExpectedException) Job(com.hazelcast.jet.Job) JobRepository(com.hazelcast.jet.impl.JobRepository) JetConfig(com.hazelcast.jet.config.JetConfig) MAX_BACKUP_COUNT(com.hazelcast.spi.partition.IPartition.MAX_BACKUP_COUNT) MockPS(com.hazelcast.jet.core.TestProcessors.MockPS) CancellationException(java.util.concurrent.CancellationException) RESTARTING(com.hazelcast.jet.core.JobStatus.RESTARTING) Assert.assertNotNull(org.junit.Assert.assertNotNull) JobConfig(com.hazelcast.jet.config.JobConfig) StuckProcessor(com.hazelcast.jet.core.TestProcessors.StuckProcessor) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) TimeUnit(java.util.concurrent.TimeUnit) Consumer(java.util.function.Consumer) CountDownLatch(java.util.concurrent.CountDownLatch) Rule(org.junit.Rule) JetService(com.hazelcast.jet.impl.JetService) COMPLETED(com.hazelcast.jet.core.JobStatus.COMPLETED) JobRecord(com.hazelcast.jet.impl.JobRecord) Assert.assertEquals(org.junit.Assert.assertEquals) MockPS(com.hazelcast.jet.core.TestProcessors.MockPS) JetService(com.hazelcast.jet.impl.JetService) CountDownLatch(java.util.concurrent.CountDownLatch) JobConfig(com.hazelcast.jet.config.JobConfig) ExpectedException(org.junit.rules.ExpectedException) CancellationException(java.util.concurrent.CancellationException) CancellationException(java.util.concurrent.CancellationException) Future(java.util.concurrent.Future) Job(com.hazelcast.jet.Job) MasterContext(com.hazelcast.jet.impl.MasterContext) Test(org.junit.Test)

Aggregations

CancellationException (java.util.concurrent.CancellationException)505 ExecutionException (java.util.concurrent.ExecutionException)150 Test (org.junit.Test)91 TimeoutException (java.util.concurrent.TimeoutException)76 ArrayList (java.util.ArrayList)47 CountDownLatch (java.util.concurrent.CountDownLatch)46 Future (java.util.concurrent.Future)46 IOException (java.io.IOException)42 ExecutorService (java.util.concurrent.ExecutorService)30 RejectedExecutionException (java.util.concurrent.RejectedExecutionException)29 CompletableFuture (java.util.concurrent.CompletableFuture)28 Callable (java.util.concurrent.Callable)27 AtomicBoolean (java.util.concurrent.atomic.AtomicBoolean)27 List (java.util.List)24 Map (java.util.Map)23 HashMap (java.util.HashMap)22 AtomicInteger (java.util.concurrent.atomic.AtomicInteger)22 TimeUnit (java.util.concurrent.TimeUnit)20 CharStream (org.antlr.v4.runtime.CharStream)20 CommonTokenStream (org.antlr.v4.runtime.CommonTokenStream)20