Examples with TestingFileEnumerator org.apache.flink.connector.file.src.testutils.TestingFileEnumerator used on opensource projects

Example 1 with TestingFileEnumerator

use of org.apache.flink.connector.file.src.testutils.TestingFileEnumerator in project flink by apache.

the class ContinuousFileSplitEnumeratorTest method testDiscoverSplitWhenNoReaderRegistered.

@Test
public void testDiscoverSplitWhenNoReaderRegistered() throws Exception {
    final TestingFileEnumerator fileEnumerator = new TestingFileEnumerator();
    final TestingSplitEnumeratorContext<FileSourceSplit> context = new TestingSplitEnumeratorContext<>(4);
    final ContinuousFileSplitEnumerator enumerator = createEnumerator(fileEnumerator, context);
    // make one split available and trigger the periodic discovery
    final FileSourceSplit split = createRandomSplit();
    fileEnumerator.addSplits(split);
    context.triggerAllActions();
    assertThat(enumerator.snapshotState(1L).getSplits(), contains(split));
}

Example 2 with TestingFileEnumerator

use of org.apache.flink.connector.file.src.testutils.TestingFileEnumerator in project flink by apache.

the class ContinuousFileSplitEnumeratorTest method testDiscoverWhenReaderRegistered.

@Test
public void testDiscoverWhenReaderRegistered() throws Exception {
    final TestingFileEnumerator fileEnumerator = new TestingFileEnumerator();
    final TestingSplitEnumeratorContext<FileSourceSplit> context = new TestingSplitEnumeratorContext<>(4);
    final ContinuousFileSplitEnumerator enumerator = createEnumerator(fileEnumerator, context);
    // register one reader, and let it request a split
    context.registerReader(2, "localhost");
    enumerator.addReader(2);
    enumerator.handleSplitRequest(2, "localhost");
    // make one split available and trigger the periodic discovery
    final FileSourceSplit split = createRandomSplit();
    fileEnumerator.addSplits(split);
    context.triggerAllActions();
    assertThat(enumerator.snapshotState(1L).getSplits(), empty());
    assertThat(context.getSplitAssignments().get(2).getAssignedSplits(), contains(split));
}

Example 3 with TestingFileEnumerator

use of org.apache.flink.connector.file.src.testutils.TestingFileEnumerator in project flink by apache.

the class ContinuousFileSplitEnumeratorTest method testRequestingReaderUnavailableWhenSplitDiscovered.

@Test
public void testRequestingReaderUnavailableWhenSplitDiscovered() throws Exception {
    final TestingFileEnumerator fileEnumerator = new TestingFileEnumerator();
    final TestingSplitEnumeratorContext<FileSourceSplit> context = new TestingSplitEnumeratorContext<>(4);
    final ContinuousFileSplitEnumerator enumerator = createEnumerator(fileEnumerator, context);
    // register one reader, and let it request a split
    context.registerReader(2, "localhost");
    enumerator.addReader(2);
    enumerator.handleSplitRequest(2, "localhost");
    // remove the reader (like in a failure)
    context.registeredReaders().remove(2);
    // make one split available and trigger the periodic discovery
    final FileSourceSplit split = createRandomSplit();
    fileEnumerator.addSplits(split);
    context.triggerAllActions();
    assertFalse(context.getSplitAssignments().containsKey(2));
    assertThat(enumerator.snapshotState(1L).getSplits(), contains(split));
}