Examples with RefCountedFileWithStream org.apache.flink.fs.s3.common.utils.RefCountedFileWithStream used on opensource projects

Example 1 with RefCountedFileWithStream

use of org.apache.flink.fs.s3.common.utils.RefCountedFileWithStream in project flink by apache.

the class S3RecoverableFsDataOutputStream method newStream.

// ------------------------------------------------------------------------
// factory methods
// ------------------------------------------------------------------------
public static S3RecoverableFsDataOutputStream newStream(final RecoverableMultiPartUpload upload, final FunctionWithException<File, RefCountedFileWithStream, IOException> tmpFileCreator, final long userDefinedMinPartSize) throws IOException {
    checkArgument(userDefinedMinPartSize >= S3_MULTIPART_MIN_PART_SIZE);
    final RefCountedBufferingFileStream fileStream = boundedBufferingFileStream(tmpFileCreator, Optional.empty());
    return new S3RecoverableFsDataOutputStream(upload, tmpFileCreator, fileStream, userDefinedMinPartSize, 0L);
}

Example 2 with RefCountedFileWithStream

use of org.apache.flink.fs.s3.common.utils.RefCountedFileWithStream in project flink by apache.

the class S3RecoverableFsDataOutputStream method recoverStream.

public static S3RecoverableFsDataOutputStream recoverStream(final RecoverableMultiPartUpload upload, final FunctionWithException<File, RefCountedFileWithStream, IOException> tmpFileCreator, final long userDefinedMinPartSize, final long bytesBeforeCurrentPart) throws IOException {
    checkArgument(userDefinedMinPartSize >= S3_MULTIPART_MIN_PART_SIZE);
    final RefCountedBufferingFileStream fileStream = boundedBufferingFileStream(tmpFileCreator, upload.getIncompletePart());
    return new S3RecoverableFsDataOutputStream(upload, tmpFileCreator, fileStream, userDefinedMinPartSize, bytesBeforeCurrentPart);
}

Example 3 with RefCountedFileWithStream

use of org.apache.flink.fs.s3.common.utils.RefCountedFileWithStream in project flink by apache.

the class S3RecoverableMultipartUploadFactory method recoverInProgressPart.

private Optional<File> recoverInProgressPart(S3Recoverable recoverable) throws IOException {
    final String objectKey = recoverable.incompleteObjectName();
    if (objectKey == null) {
        return Optional.empty();
    }
    // download the file (simple way)
    final RefCountedFileWithStream refCountedFile = tmpFileSupplier.apply(null);
    final File file = refCountedFile.getFile();
    final long numBytes = s3AccessHelper.getObject(objectKey, file);
    if (numBytes != recoverable.incompleteObjectLength()) {
        throw new IOException(String.format("Error recovering writer: " + "Downloading the last data chunk file gives incorrect length." + "File length is %d bytes, RecoveryData indicates %d bytes", numBytes, recoverable.incompleteObjectLength()));
    }
    return Optional.of(file);
}