Examples with FileEnumerator org.apache.flink.connector.file.src.enumerate.FileEnumerator used on opensource projects

Example 1 with FileEnumerator

use of org.apache.flink.connector.file.src.enumerate.FileEnumerator in project flink by apache.

the class AbstractFileSource method createEnumerator.

@Override
public SplitEnumerator<SplitT, PendingSplitsCheckpoint<SplitT>> createEnumerator(SplitEnumeratorContext<SplitT> enumContext) {
    final FileEnumerator enumerator = enumeratorFactory.create();
    // read the initial set of splits (which is also the total set of splits for bounded
    // sources)
    final Collection<FileSourceSplit> splits;
    try {
        // TODO - in the next cleanup pass, we should try to remove the need to "wrap unchecked"
        // here
        splits = enumerator.enumerateSplits(inputPaths, enumContext.currentParallelism());
    } catch (IOException e) {
        throw new FlinkRuntimeException("Could not enumerate file splits", e);
    }
    return createSplitEnumerator(enumContext, enumerator, splits, null);
}

Example 2 with FileEnumerator

use of org.apache.flink.connector.file.src.enumerate.FileEnumerator in project flink by apache.

the class AbstractFileSource method restoreEnumerator.

@Override
public SplitEnumerator<SplitT, PendingSplitsCheckpoint<SplitT>> restoreEnumerator(SplitEnumeratorContext<SplitT> enumContext, PendingSplitsCheckpoint<SplitT> checkpoint) {
    final FileEnumerator enumerator = enumeratorFactory.create();
    // cast this to a collection of FileSourceSplit because the enumerator code work
    // non-generically just on that base split type
    @SuppressWarnings("unchecked") final Collection<FileSourceSplit> splits = (Collection<FileSourceSplit>) checkpoint.getSplits();
    return createSplitEnumerator(enumContext, enumerator, splits, checkpoint.getAlreadyProcessedPaths());
}