Search in sources :

Example 6 with MockSplitEnumeratorContext

use of org.apache.flink.api.connector.source.mocks.MockSplitEnumeratorContext in project flink by apache.

the class SourceCoordinatorTest method testBlockOnClose.

@Test
public void testBlockOnClose() throws Exception {
    // It is possible that the split enumerator submits some heavy-duty work to the
    // coordinator executor which blocks the coordinator closure.
    final CountDownLatch latch = new CountDownLatch(1);
    try (final MockSplitEnumeratorContext<MockSourceSplit> enumeratorContext = new MockSplitEnumeratorContext<>(1);
        final MockSplitEnumerator splitEnumerator = new MockSplitEnumerator(1, enumeratorContext) {

            @Override
            public void handleSourceEvent(int subtaskId, SourceEvent sourceEvent) {
                context.callAsync(() -> 1L, (ignored, t) -> {
                    latch.countDown();
                    // Submit a callable that will never return.
                    try {
                        Thread.sleep(Long.MAX_VALUE);
                    } catch (InterruptedException e) {
                        throw new RuntimeException(e);
                    }
                });
            }
        };
        final SourceCoordinator<?, ?> coordinator = new SourceCoordinator<>(OPERATOR_NAME, new EnumeratorCreatingSource<>(() -> splitEnumerator), context, new CoordinatorStoreImpl())) {
        coordinator.start();
        coordinator.handleEventFromOperator(1, new SourceEventWrapper(new SourceEvent() {
        }));
        // Wait until the coordinator executor blocks.
        latch.await();
        CompletableFuture<?> future = ComponentClosingUtils.closeAsyncWithTimeout("testBlockOnClose", (ThrowingRunnable<Exception>) coordinator::close, Duration.ofMillis(1));
        future.exceptionally(e -> {
            assertTrue(e instanceof TimeoutException);
            return null;
        }).get();
        waitUtil(splitEnumerator::closed, Duration.ofSeconds(5), "Split enumerator was not closed in 5 seconds.");
    }
}
Also used : MockSplitEnumeratorCheckpointSerializer(org.apache.flink.api.connector.source.mocks.MockSplitEnumeratorCheckpointSerializer) CoordinatorTestUtils.verifyException(org.apache.flink.runtime.source.coordinator.CoordinatorTestUtils.verifyException) Arrays(java.util.Arrays) CoordinatorTestUtils.verifyAssignment(org.apache.flink.runtime.source.coordinator.CoordinatorTestUtils.verifyAssignment) SourceEventWrapper(org.apache.flink.runtime.source.event.SourceEventWrapper) ThrowingRunnable(org.apache.flink.util.function.ThrowingRunnable) URL(java.net.URL) TimeoutException(java.util.concurrent.TimeoutException) MockSplitEnumerator(org.apache.flink.api.connector.source.mocks.MockSplitEnumerator) CompletableFuture(java.util.concurrent.CompletableFuture) Supplier(java.util.function.Supplier) HashSet(java.util.HashSet) Assert.assertSame(org.junit.Assert.assertSame) MockSourceSplit(org.apache.flink.api.connector.source.mocks.MockSourceSplit) ComponentClosingUtils(org.apache.flink.runtime.operators.coordination.ComponentClosingUtils) URLClassLoader(java.net.URLClassLoader) SplitEnumerator(org.apache.flink.api.connector.source.SplitEnumerator) Duration(java.time.Duration) Map(java.util.Map) DataOutputSerializer(org.apache.flink.core.memory.DataOutputSerializer) Assert.fail(org.junit.Assert.fail) Nullable(javax.annotation.Nullable) SourceReaderContext(org.apache.flink.api.connector.source.SourceReaderContext) WatermarkAlignmentParams(org.apache.flink.api.common.eventtime.WatermarkAlignmentParams) SourceReader(org.apache.flink.api.connector.source.SourceReader) SourceEvent(org.apache.flink.api.connector.source.SourceEvent) SplitEnumeratorContext(org.apache.flink.api.connector.source.SplitEnumeratorContext) MockSplitEnumeratorContext(org.apache.flink.api.connector.source.mocks.MockSplitEnumeratorContext) Set(java.util.Set) CommonTestUtils.waitUtil(org.apache.flink.core.testutils.CommonTestUtils.waitUtil) Assert.assertTrue(org.junit.Assert.assertTrue) MockSourceSplitSerializer(org.apache.flink.api.connector.source.mocks.MockSourceSplitSerializer) Test(org.junit.Test) MockOperatorCoordinatorContext(org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext) CountDownLatch(java.util.concurrent.CountDownLatch) List(java.util.List) SimpleVersionedSerializer(org.apache.flink.core.io.SimpleVersionedSerializer) OperatorCoordinator(org.apache.flink.runtime.operators.coordination.OperatorCoordinator) Assert.assertFalse(org.junit.Assert.assertFalse) OperatorID(org.apache.flink.runtime.jobgraph.OperatorID) Source(org.apache.flink.api.connector.source.Source) CoordinatorStoreImpl(org.apache.flink.runtime.operators.coordination.CoordinatorStoreImpl) Collections(java.util.Collections) Boundedness(org.apache.flink.api.connector.source.Boundedness) Assert.assertEquals(org.junit.Assert.assertEquals) CoordinatorStoreImpl(org.apache.flink.runtime.operators.coordination.CoordinatorStoreImpl) MockSplitEnumeratorContext(org.apache.flink.api.connector.source.mocks.MockSplitEnumeratorContext) SourceEventWrapper(org.apache.flink.runtime.source.event.SourceEventWrapper) CountDownLatch(java.util.concurrent.CountDownLatch) CoordinatorTestUtils.verifyException(org.apache.flink.runtime.source.coordinator.CoordinatorTestUtils.verifyException) TimeoutException(java.util.concurrent.TimeoutException) SourceEvent(org.apache.flink.api.connector.source.SourceEvent) MockSourceSplit(org.apache.flink.api.connector.source.mocks.MockSourceSplit) MockSplitEnumerator(org.apache.flink.api.connector.source.mocks.MockSplitEnumerator) TimeoutException(java.util.concurrent.TimeoutException) Test(org.junit.Test)

Example 7 with MockSplitEnumeratorContext

use of org.apache.flink.api.connector.source.mocks.MockSplitEnumeratorContext in project flink by apache.

the class SourceCoordinatorTest method testFailJobWhenExceptionThrownFromStart.

@Test
public void testFailJobWhenExceptionThrownFromStart() throws Exception {
    final RuntimeException failureReason = new RuntimeException("Artificial Exception");
    try (final MockSplitEnumeratorContext<MockSourceSplit> enumeratorContext = new MockSplitEnumeratorContext<>(1);
        final SplitEnumerator<MockSourceSplit, Set<MockSourceSplit>> splitEnumerator = new MockSplitEnumerator(1, enumeratorContext) {

            @Override
            public void start() {
                throw failureReason;
            }
        };
        final SourceCoordinator<?, ?> coordinator = new SourceCoordinator<>(OPERATOR_NAME, new EnumeratorCreatingSource<>(() -> splitEnumerator), context, new CoordinatorStoreImpl(), WatermarkAlignmentParams.WATERMARK_ALIGNMENT_DISABLED)) {
        coordinator.start();
        waitUtil(() -> operatorCoordinatorContext.isJobFailed(), Duration.ofSeconds(10), "The job should have failed due to the artificial exception.");
        assertEquals(failureReason, operatorCoordinatorContext.getJobFailureReason());
    }
}
Also used : CoordinatorStoreImpl(org.apache.flink.runtime.operators.coordination.CoordinatorStoreImpl) MockSplitEnumeratorContext(org.apache.flink.api.connector.source.mocks.MockSplitEnumeratorContext) HashSet(java.util.HashSet) Set(java.util.Set) MockSourceSplit(org.apache.flink.api.connector.source.mocks.MockSourceSplit) MockSplitEnumerator(org.apache.flink.api.connector.source.mocks.MockSplitEnumerator) Test(org.junit.Test)

Aggregations

MockSplitEnumeratorContext (org.apache.flink.api.connector.source.mocks.MockSplitEnumeratorContext)7 Test (org.junit.Test)7 HashSet (java.util.HashSet)4 KafkaPartitionSplit (org.apache.flink.connector.kafka.source.split.KafkaPartitionSplit)4 Set (java.util.Set)3 MockSourceSplit (org.apache.flink.api.connector.source.mocks.MockSourceSplit)3 MockSplitEnumerator (org.apache.flink.api.connector.source.mocks.MockSplitEnumerator)3 CoordinatorStoreImpl (org.apache.flink.runtime.operators.coordination.CoordinatorStoreImpl)3 Properties (java.util.Properties)2 SourceEvent (org.apache.flink.api.connector.source.SourceEvent)2 SourceEventWrapper (org.apache.flink.runtime.source.event.SourceEventWrapper)2 TopicPartition (org.apache.kafka.common.TopicPartition)2 URL (java.net.URL)1 URLClassLoader (java.net.URLClassLoader)1 Duration (java.time.Duration)1 Arrays (java.util.Arrays)1 Collections (java.util.Collections)1 List (java.util.List)1 Map (java.util.Map)1 CompletableFuture (java.util.concurrent.CompletableFuture)1