Search in sources :

Example 1 with SourceEvent

use of org.apache.flink.api.connector.source.SourceEvent in project flink by apache.

the class SourceCoordinator method handleEventFromOperator.

@Override
public void handleEventFromOperator(int subtask, OperatorEvent event) {
    runInEventLoop(() -> {
        if (event instanceof RequestSplitEvent) {
            LOG.info("Source {} received split request from parallel task {}", operatorName, subtask);
            enumerator.handleSplitRequest(subtask, ((RequestSplitEvent) event).hostName());
        } else if (event instanceof SourceEventWrapper) {
            final SourceEvent sourceEvent = ((SourceEventWrapper) event).getSourceEvent();
            LOG.debug("Source {} received custom event from parallel task {}: {}", operatorName, subtask, sourceEvent);
            enumerator.handleSourceEvent(subtask, sourceEvent);
        } else if (event instanceof ReaderRegistrationEvent) {
            final ReaderRegistrationEvent registrationEvent = (ReaderRegistrationEvent) event;
            LOG.info("Source {} registering reader for parallel task {} @ {}", operatorName, subtask, registrationEvent.location());
            handleReaderRegistrationEvent(registrationEvent);
        } else if (event instanceof ReportedWatermarkEvent) {
            handleReportedWatermark(subtask, new Watermark(((ReportedWatermarkEvent) event).getWatermark()));
        } else {
            throw new FlinkException("Unrecognized Operator Event: " + event);
        }
    }, "handling operator event %s from subtask %d", event, subtask);
}
Also used : RequestSplitEvent(org.apache.flink.runtime.source.event.RequestSplitEvent) SourceEventWrapper(org.apache.flink.runtime.source.event.SourceEventWrapper) ReaderRegistrationEvent(org.apache.flink.runtime.source.event.ReaderRegistrationEvent) Watermark(org.apache.flink.api.common.eventtime.Watermark) FlinkException(org.apache.flink.util.FlinkException) SourceEvent(org.apache.flink.api.connector.source.SourceEvent) ReportedWatermarkEvent(org.apache.flink.runtime.source.event.ReportedWatermarkEvent)

Example 2 with SourceEvent

use of org.apache.flink.api.connector.source.SourceEvent in project flink by apache.

the class SourceCoordinatorTest method testHandleSourceEvent.

@Test
public void testHandleSourceEvent() throws Exception {
    sourceReady();
    SourceEvent sourceEvent = new SourceEvent() {
    };
    sourceCoordinator.handleEventFromOperator(0, new SourceEventWrapper(sourceEvent));
    waitForCoordinatorToProcessActions();
    assertEquals(1, getEnumerator().getHandledSourceEvent().size());
    assertEquals(sourceEvent, getEnumerator().getHandledSourceEvent().get(0));
}
Also used : SourceEventWrapper(org.apache.flink.runtime.source.event.SourceEventWrapper) SourceEvent(org.apache.flink.api.connector.source.SourceEvent) Test(org.junit.Test)

Example 3 with SourceEvent

use of org.apache.flink.api.connector.source.SourceEvent in project flink by apache.

the class SourceCoordinatorTest method testErrorThrownFromSplitEnumerator.

@Test
public void testErrorThrownFromSplitEnumerator() throws Exception {
    final Error error = new Error("Test Error");
    try (final MockSplitEnumeratorContext<MockSourceSplit> enumeratorContext = new MockSplitEnumeratorContext<>(1);
        final SplitEnumerator<MockSourceSplit, Set<MockSourceSplit>> splitEnumerator = new MockSplitEnumerator(1, enumeratorContext) {

            @Override
            public void handleSourceEvent(int subtaskId, SourceEvent sourceEvent) {
                throw error;
            }
        };
        final SourceCoordinator<?, ?> coordinator = new SourceCoordinator<>(OPERATOR_NAME, new EnumeratorCreatingSource<>(() -> splitEnumerator), context, new CoordinatorStoreImpl(), WatermarkAlignmentParams.WATERMARK_ALIGNMENT_DISABLED)) {
        coordinator.start();
        coordinator.handleEventFromOperator(1, new SourceEventWrapper(new SourceEvent() {
        }));
        waitUtil(() -> operatorCoordinatorContext.isJobFailed(), Duration.ofSeconds(10), "The job should have failed due to the artificial exception.");
        assertEquals(error, operatorCoordinatorContext.getJobFailureReason());
    }
}
Also used : CoordinatorStoreImpl(org.apache.flink.runtime.operators.coordination.CoordinatorStoreImpl) MockSplitEnumeratorContext(org.apache.flink.api.connector.source.mocks.MockSplitEnumeratorContext) HashSet(java.util.HashSet) Set(java.util.Set) SourceEventWrapper(org.apache.flink.runtime.source.event.SourceEventWrapper) SourceEvent(org.apache.flink.api.connector.source.SourceEvent) MockSourceSplit(org.apache.flink.api.connector.source.mocks.MockSourceSplit) MockSplitEnumerator(org.apache.flink.api.connector.source.mocks.MockSplitEnumerator) Test(org.junit.Test)

Example 4 with SourceEvent

use of org.apache.flink.api.connector.source.SourceEvent in project flink by apache.

the class SourceCoordinatorTest method testBlockOnClose.

@Test
public void testBlockOnClose() throws Exception {
    // It is possible that the split enumerator submits some heavy-duty work to the
    // coordinator executor which blocks the coordinator closure.
    final CountDownLatch latch = new CountDownLatch(1);
    try (final MockSplitEnumeratorContext<MockSourceSplit> enumeratorContext = new MockSplitEnumeratorContext<>(1);
        final MockSplitEnumerator splitEnumerator = new MockSplitEnumerator(1, enumeratorContext) {

            @Override
            public void handleSourceEvent(int subtaskId, SourceEvent sourceEvent) {
                context.callAsync(() -> 1L, (ignored, t) -> {
                    latch.countDown();
                    // Submit a callable that will never return.
                    try {
                        Thread.sleep(Long.MAX_VALUE);
                    } catch (InterruptedException e) {
                        throw new RuntimeException(e);
                    }
                });
            }
        };
        final SourceCoordinator<?, ?> coordinator = new SourceCoordinator<>(OPERATOR_NAME, new EnumeratorCreatingSource<>(() -> splitEnumerator), context, new CoordinatorStoreImpl())) {
        coordinator.start();
        coordinator.handleEventFromOperator(1, new SourceEventWrapper(new SourceEvent() {
        }));
        // Wait until the coordinator executor blocks.
        latch.await();
        CompletableFuture<?> future = ComponentClosingUtils.closeAsyncWithTimeout("testBlockOnClose", (ThrowingRunnable<Exception>) coordinator::close, Duration.ofMillis(1));
        future.exceptionally(e -> {
            assertTrue(e instanceof TimeoutException);
            return null;
        }).get();
        waitUtil(splitEnumerator::closed, Duration.ofSeconds(5), "Split enumerator was not closed in 5 seconds.");
    }
}
Also used : MockSplitEnumeratorCheckpointSerializer(org.apache.flink.api.connector.source.mocks.MockSplitEnumeratorCheckpointSerializer) CoordinatorTestUtils.verifyException(org.apache.flink.runtime.source.coordinator.CoordinatorTestUtils.verifyException) Arrays(java.util.Arrays) CoordinatorTestUtils.verifyAssignment(org.apache.flink.runtime.source.coordinator.CoordinatorTestUtils.verifyAssignment) SourceEventWrapper(org.apache.flink.runtime.source.event.SourceEventWrapper) ThrowingRunnable(org.apache.flink.util.function.ThrowingRunnable) URL(java.net.URL) TimeoutException(java.util.concurrent.TimeoutException) MockSplitEnumerator(org.apache.flink.api.connector.source.mocks.MockSplitEnumerator) CompletableFuture(java.util.concurrent.CompletableFuture) Supplier(java.util.function.Supplier) HashSet(java.util.HashSet) Assert.assertSame(org.junit.Assert.assertSame) MockSourceSplit(org.apache.flink.api.connector.source.mocks.MockSourceSplit) ComponentClosingUtils(org.apache.flink.runtime.operators.coordination.ComponentClosingUtils) URLClassLoader(java.net.URLClassLoader) SplitEnumerator(org.apache.flink.api.connector.source.SplitEnumerator) Duration(java.time.Duration) Map(java.util.Map) DataOutputSerializer(org.apache.flink.core.memory.DataOutputSerializer) Assert.fail(org.junit.Assert.fail) Nullable(javax.annotation.Nullable) SourceReaderContext(org.apache.flink.api.connector.source.SourceReaderContext) WatermarkAlignmentParams(org.apache.flink.api.common.eventtime.WatermarkAlignmentParams) SourceReader(org.apache.flink.api.connector.source.SourceReader) SourceEvent(org.apache.flink.api.connector.source.SourceEvent) SplitEnumeratorContext(org.apache.flink.api.connector.source.SplitEnumeratorContext) MockSplitEnumeratorContext(org.apache.flink.api.connector.source.mocks.MockSplitEnumeratorContext) Set(java.util.Set) CommonTestUtils.waitUtil(org.apache.flink.core.testutils.CommonTestUtils.waitUtil) Assert.assertTrue(org.junit.Assert.assertTrue) MockSourceSplitSerializer(org.apache.flink.api.connector.source.mocks.MockSourceSplitSerializer) Test(org.junit.Test) MockOperatorCoordinatorContext(org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext) CountDownLatch(java.util.concurrent.CountDownLatch) List(java.util.List) SimpleVersionedSerializer(org.apache.flink.core.io.SimpleVersionedSerializer) OperatorCoordinator(org.apache.flink.runtime.operators.coordination.OperatorCoordinator) Assert.assertFalse(org.junit.Assert.assertFalse) OperatorID(org.apache.flink.runtime.jobgraph.OperatorID) Source(org.apache.flink.api.connector.source.Source) CoordinatorStoreImpl(org.apache.flink.runtime.operators.coordination.CoordinatorStoreImpl) Collections(java.util.Collections) Boundedness(org.apache.flink.api.connector.source.Boundedness) Assert.assertEquals(org.junit.Assert.assertEquals) CoordinatorStoreImpl(org.apache.flink.runtime.operators.coordination.CoordinatorStoreImpl) MockSplitEnumeratorContext(org.apache.flink.api.connector.source.mocks.MockSplitEnumeratorContext) SourceEventWrapper(org.apache.flink.runtime.source.event.SourceEventWrapper) CountDownLatch(java.util.concurrent.CountDownLatch) CoordinatorTestUtils.verifyException(org.apache.flink.runtime.source.coordinator.CoordinatorTestUtils.verifyException) TimeoutException(java.util.concurrent.TimeoutException) SourceEvent(org.apache.flink.api.connector.source.SourceEvent) MockSourceSplit(org.apache.flink.api.connector.source.mocks.MockSourceSplit) MockSplitEnumerator(org.apache.flink.api.connector.source.mocks.MockSplitEnumerator) TimeoutException(java.util.concurrent.TimeoutException) Test(org.junit.Test)

Example 5 with SourceEvent

use of org.apache.flink.api.connector.source.SourceEvent in project flink by apache.

the class SourceOperator method initReader.

/**
 * Initializes the reader. The code from this method should ideally happen in the constructor or
 * in the operator factory even. It has to happen here at a slightly later stage, because of the
 * lazy metric initialization.
 *
 * <p>Calling this method explicitly is an optional way to have the reader initialization a bit
 * earlier than in open(), as needed by the {@link
 * org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask}
 *
 * <p>This code should move to the constructor once the metric groups are available at task
 * setup time.
 */
public void initReader() throws Exception {
    if (sourceReader != null) {
        return;
    }
    final int subtaskIndex = getRuntimeContext().getIndexOfThisSubtask();
    final SourceReaderContext context = new SourceReaderContext() {

        @Override
        public SourceReaderMetricGroup metricGroup() {
            return sourceMetricGroup;
        }

        @Override
        public Configuration getConfiguration() {
            return configuration;
        }

        @Override
        public String getLocalHostName() {
            return localHostname;
        }

        @Override
        public int getIndexOfSubtask() {
            return subtaskIndex;
        }

        @Override
        public void sendSplitRequest() {
            operatorEventGateway.sendEventToCoordinator(new RequestSplitEvent(getLocalHostName()));
        }

        @Override
        public void sendSourceEventToCoordinator(SourceEvent event) {
            operatorEventGateway.sendEventToCoordinator(new SourceEventWrapper(event));
        }

        @Override
        public UserCodeClassLoader getUserCodeClassLoader() {
            return new UserCodeClassLoader() {

                @Override
                public ClassLoader asClassLoader() {
                    return getRuntimeContext().getUserCodeClassLoader();
                }

                @Override
                public void registerReleaseHookIfAbsent(String releaseHookName, Runnable releaseHook) {
                    getRuntimeContext().registerUserCodeClassLoaderReleaseHookIfAbsent(releaseHookName, releaseHook);
                }
            };
        }
    };
    sourceReader = readerFactory.apply(context);
}
Also used : RequestSplitEvent(org.apache.flink.runtime.source.event.RequestSplitEvent) UserCodeClassLoader(org.apache.flink.util.UserCodeClassLoader) SourceEventWrapper(org.apache.flink.runtime.source.event.SourceEventWrapper) SourceReaderContext(org.apache.flink.api.connector.source.SourceReaderContext) SourceEvent(org.apache.flink.api.connector.source.SourceEvent)

Aggregations

SourceEvent (org.apache.flink.api.connector.source.SourceEvent)6 SourceEventWrapper (org.apache.flink.runtime.source.event.SourceEventWrapper)6 Test (org.junit.Test)4 HashSet (java.util.HashSet)2 Set (java.util.Set)2 SourceReaderContext (org.apache.flink.api.connector.source.SourceReaderContext)2 MockSourceSplit (org.apache.flink.api.connector.source.mocks.MockSourceSplit)2 MockSplitEnumerator (org.apache.flink.api.connector.source.mocks.MockSplitEnumerator)2 MockSplitEnumeratorContext (org.apache.flink.api.connector.source.mocks.MockSplitEnumeratorContext)2 CoordinatorStoreImpl (org.apache.flink.runtime.operators.coordination.CoordinatorStoreImpl)2 RequestSplitEvent (org.apache.flink.runtime.source.event.RequestSplitEvent)2 URL (java.net.URL)1 URLClassLoader (java.net.URLClassLoader)1 Duration (java.time.Duration)1 Arrays (java.util.Arrays)1 Collections (java.util.Collections)1 List (java.util.List)1 Map (java.util.Map)1 CompletableFuture (java.util.concurrent.CompletableFuture)1 CountDownLatch (java.util.concurrent.CountDownLatch)1