Examples with FileSourceSplit org.apache.flink.connector.file.src.FileSourceSplit used on opensource projects

Example 46 with FileSourceSplit

use of org.apache.flink.connector.file.src.FileSourceSplit in project flink by apache.

the class LocalityAwareSplitAssigner method addSplits.

@Override
public void addSplits(Collection<FileSourceSplit> splits) {
    for (FileSourceSplit split : splits) {
        SplitWithInfo sc = new SplitWithInfo(split);
        remoteSplitChooser.addInputSplit(sc);
        unassigned.add(sc);
    }
}

Example 47 with FileSourceSplit

use of org.apache.flink.connector.file.src.FileSourceSplit in project flink by apache.

the class NonSplittingRecursiveEnumerator method convertToSourceSplits.

protected void convertToSourceSplits(final FileStatus file, final FileSystem fs, final List<FileSourceSplit> target) throws IOException {
    final String[] hosts = getHostsFromBlockLocations(fs.getFileBlockLocations(file, 0L, file.getLen()));
    target.add(new FileSourceSplit(getNextId(), file.getPath(), 0, file.getLen(), file.getModificationTime(), file.getLen(), hosts));
}

Example 48 with FileSourceSplit

use of org.apache.flink.connector.file.src.FileSourceSplit in project flink by apache.

the class NonSplittingRecursiveEnumerator method enumerateSplits.

// ------------------------------------------------------------------------
@Override
public Collection<FileSourceSplit> enumerateSplits(Path[] paths, int minDesiredSplits) throws IOException {
    final ArrayList<FileSourceSplit> splits = new ArrayList<>();
    for (Path path : paths) {
        final FileSystem fs = path.getFileSystem();
        final FileStatus status = fs.getFileStatus(path);
        addSplitsForPath(status, fs, splits);
    }
    return splits;
}

Example 49 with FileSourceSplit

use of org.apache.flink.connector.file.src.FileSourceSplit in project flink by apache.

the class ContinuousFileSplitEnumerator method assignSplits.

private void assignSplits() {
    final Iterator<Map.Entry<Integer, String>> awaitingReader = readersAwaitingSplit.entrySet().iterator();
    while (awaitingReader.hasNext()) {
        final Map.Entry<Integer, String> nextAwaiting = awaitingReader.next();
        // it from the list of waiting readers
        if (!context.registeredReaders().containsKey(nextAwaiting.getKey())) {
            awaitingReader.remove();
            continue;
        }
        final String hostname = nextAwaiting.getValue();
        final int awaitingSubtask = nextAwaiting.getKey();
        final Optional<FileSourceSplit> nextSplit = splitAssigner.getNext(hostname);
        if (nextSplit.isPresent()) {
            context.assignSplit(nextSplit.get(), awaitingSubtask);
            awaitingReader.remove();
        } else {
            break;
        }
    }
}

Example 50 with FileSourceSplit

use of org.apache.flink.connector.file.src.FileSourceSplit in project flink by apache.

the class StaticFileSplitEnumerator method handleSplitRequest.

@Override
public void handleSplitRequest(int subtask, @Nullable String hostname) {
    if (!context.registeredReaders().containsKey(subtask)) {
        // reader failed between sending the request and now. skip this request.
        return;
    }
    if (LOG.isInfoEnabled()) {
        final String hostInfo = hostname == null ? "(no host locality info)" : "(on host '" + hostname + "')";
        LOG.info("Subtask {} {} is requesting a file source split", subtask, hostInfo);
    }
    final Optional<FileSourceSplit> nextSplit = splitAssigner.getNext(hostname);
    if (nextSplit.isPresent()) {
        final FileSourceSplit split = nextSplit.get();
        context.assignSplit(split, subtask);
        LOG.info("Assigned split to subtask {} : {}", subtask, split);
    } else {
        context.signalNoMoreSplits(subtask);
        LOG.info("No more splits available for subtask {}", subtask);
    }
}