Examples with CheckpointedPosition org.apache.flink.connector.file.src.util.CheckpointedPosition used on opensource projects

Example 6 with CheckpointedPosition

use of org.apache.flink.connector.file.src.util.CheckpointedPosition in project flink by apache.

the class FileSourceSplitStateTest method testStateStartsWithSplitValues.

@Test
public void testStateStartsWithSplitValues() {
    final FileSourceSplit split = getTestSplit(new CheckpointedPosition(123L, 456L));
    final FileSourceSplitState state = new FileSourceSplitState(split);
    assertEquals(123L, state.getOffset());
    assertEquals(456L, state.getRecordsToSkipAfterOffset());
}

Example 7 with CheckpointedPosition

use of org.apache.flink.connector.file.src.util.CheckpointedPosition in project flink by apache.

the class AdapterTestBase method testClosesStreamIfReaderRestoreFails.

@Test
public void testClosesStreamIfReaderRestoreFails() throws Exception {
    // setup
    final Path testPath = new Path("testFs:///testpath-1");
    final CloseTestingInputStream in = new CloseTestingInputStream();
    final TestingFileSystem testFs = TestingFileSystem.createForFileStatus("testFs", TestingFileSystem.TestFileStatus.forFileWithStream(testPath, 1024, in));
    testFs.register();
    // test
    final BulkFormat<Integer, FileSourceSplit> adapter = wrapWithAdapter(createFormatFailingInInstantiation());
    final FileSourceSplit split = new FileSourceSplit("id", testPath, 0, 1024, 0, 1024, new String[0], new CheckpointedPosition(0L, 5L));
    try {
        adapter.restoreReader(new Configuration(), split);
    } catch (IOException ignored) {
    }
    // assertions
    assertTrue(in.closed);
    // cleanup
    testFs.unregister();
}

Example 8 with CheckpointedPosition

use of org.apache.flink.connector.file.src.util.CheckpointedPosition in project flink by apache.

the class ParquetVectorizedInputFormat method restoreReader.

@Override
public ParquetReader restoreReader(final Configuration config, final SplitT split) throws IOException {
    assert split.getReaderPosition().isPresent();
    final CheckpointedPosition checkpointedPosition = split.getReaderPosition().get();
    Preconditions.checkArgument(checkpointedPosition.getOffset() == CheckpointedPosition.NO_OFFSET, "The offset of CheckpointedPosition should always be NO_OFFSET");
    ParquetReader reader = createReader(config, split);
    reader.seek(checkpointedPosition.getRecordsAfterOffset());
    return reader;
}

Example 9 with CheckpointedPosition

use of org.apache.flink.connector.file.src.util.CheckpointedPosition in project flink by apache.

the class FileSourceSplitStateTest method testNewSplitTakesModifiedOffsetAndCount.

@Test
public void testNewSplitTakesModifiedOffsetAndCount() {
    final FileSourceSplit split = getTestSplit();
    final FileSourceSplitState state = new FileSourceSplitState(split);
    state.setPosition(1234L, 7566L);
    final Optional<CheckpointedPosition> position = state.toFileSourceSplit().getReaderPosition();
    assertTrue(position.isPresent());
    assertEquals(new CheckpointedPosition(1234L, 7566L), position.get());
}

Example 10 with CheckpointedPosition

use of org.apache.flink.connector.file.src.util.CheckpointedPosition in project flink by apache.

the class FileSourceSplitSerializerTest method serializeSplitWithReaderPosition.

@Test
public void serializeSplitWithReaderPosition() throws Exception {
    final FileSourceSplit split = new FileSourceSplit("random-id", new Path("hdfs://namenode:14565/some/path/to/a/file"), 100_000_000, 64_000_000, System.currentTimeMillis(), 200_000_000, new String[] { "host1", "host2", "host3" }, new CheckpointedPosition(7665391L, 100L));
    final FileSourceSplit deSerialized = serializeAndDeserialize(split);
    assertSplitsEqual(split, deSerialized);
}