Search in sources :

Example 6 with PendingFileRecoverable

use of org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable in project flink by apache.

the class CompactService method compact.

@SuppressWarnings({ "rawtypes", "unchecked" })
private Iterable<FileSinkCommittable> compact(CompactorRequest request) throws Exception {
    List<FileSinkCommittable> results = new ArrayList<>(request.getCommittableToPassthrough());
    List<Path> compactingFiles = getCompactingPath(request);
    if (compactingFiles.isEmpty()) {
        return results;
    }
    Path targetPath = assembleCompactedFilePath(compactingFiles.get(0));
    CompactingFileWriter compactingFileWriter = bucketWriter.openNewCompactingFile(compactingWriterType, request.getBucketId(), targetPath, System.currentTimeMillis());
    if (compactingWriterType == Type.RECORD_WISE) {
        ((RecordWiseFileCompactor) fileCompactor).compact(compactingFiles, ((RecordWiseCompactingFileWriter) compactingFileWriter)::write);
    } else if (compactingWriterType == CompactingFileWriter.Type.OUTPUT_STREAM) {
        ((OutputStreamBasedFileCompactor) fileCompactor).compact(compactingFiles, ((OutputStreamBasedCompactingFileWriter) compactingFileWriter).asOutputStream());
    }
    PendingFileRecoverable compactedPendingFile = compactingFileWriter.closeForCommit();
    FileSinkCommittable compacted = new FileSinkCommittable(request.getBucketId(), compactedPendingFile);
    results.add(compacted);
    for (Path f : compactingFiles) {
        // cleanup compacted files
        results.add(new FileSinkCommittable(request.getBucketId(), f));
    }
    return results;
}
Also used : Path(org.apache.flink.core.fs.Path) OutputStreamBasedCompactingFileWriter(org.apache.flink.streaming.api.functions.sink.filesystem.OutputStreamBasedCompactingFileWriter) RecordWiseFileCompactor(org.apache.flink.connector.file.sink.compactor.RecordWiseFileCompactor) ArrayList(java.util.ArrayList) PendingFileRecoverable(org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable) FileSinkCommittable(org.apache.flink.connector.file.sink.FileSinkCommittable) CompactingFileWriter(org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter) OutputStreamBasedCompactingFileWriter(org.apache.flink.streaming.api.functions.sink.filesystem.OutputStreamBasedCompactingFileWriter) RecordWiseCompactingFileWriter(org.apache.flink.streaming.api.functions.sink.filesystem.RecordWiseCompactingFileWriter)

Aggregations

PendingFileRecoverable (org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable)6 Path (org.apache.flink.core.fs.Path)4 ArrayList (java.util.ArrayList)3 FileSinkCommittable (org.apache.flink.connector.file.sink.FileSinkCommittable)3 CompactingFileWriter (org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter)2 InProgressFileWriter (org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter)2 InProgressFileRecoverable (org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.InProgressFileRecoverable)2 OutputStreamBasedCompactingFileWriter (org.apache.flink.streaming.api.functions.sink.filesystem.OutputStreamBasedCompactingFileWriter)2 BufferedWriter (java.io.BufferedWriter)1 FileOutputStream (java.io.FileOutputStream)1 FileWriter (java.io.FileWriter)1 HashMap (java.util.HashMap)1 List (java.util.List)1 RecordWiseFileCompactor (org.apache.flink.connector.file.sink.compactor.RecordWiseFileCompactor)1 TestInProgressFileRecoverable (org.apache.flink.connector.file.sink.utils.FileSinkTestUtils.TestInProgressFileRecoverable)1 TestPendingFileRecoverable (org.apache.flink.connector.file.sink.utils.FileSinkTestUtils.TestPendingFileRecoverable)1 RecoverableWriter (org.apache.flink.core.fs.RecoverableWriter)1 CommittableMessage (org.apache.flink.streaming.api.connector.sink2.CommittableMessage)1 CommittableSummary (org.apache.flink.streaming.api.connector.sink2.CommittableSummary)1 CommittableWithLineage (org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage)1