Examples with RefCountedBufferingFileStream org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream used on opensource projects

Example 1 with RefCountedBufferingFileStream

use of org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream in project flink by apache.

the class RecoverableMultiPartUploadImplTest method uploadPart.

private void uploadPart(final byte[] content) throws IOException {
    RefCountedBufferingFileStream partFile = writeContent(content);
    // as in the production code, we assume that a file containing
    // a completed part is closed before being passed to the uploader.
    partFile.close();
    multiPartUploadUnderTest.uploadPart(partFile);
}

Example 2 with RefCountedBufferingFileStream

use of org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream in project flink by apache.

the class S3RecoverableFsDataOutputStreamTest method reopenStreamUnderTestAfterRecovery.

// ------------------------------------------------------------------------------------------------------------
// Utils
// ------------------------------------------------------------------------------------------------------------
private S3RecoverableFsDataOutputStream reopenStreamUnderTestAfterRecovery() throws IOException {
    final long bytesBeforeCurrentPart = multipartUploadUnderTest.numBytes;
    final Optional<File> incompletePart = multipartUploadUnderTest.getIncompletePart();
    RefCountedBufferingFileStream fileStream = RefCountedBufferingFileStream.restore(fileProvider, incompletePart.get());
    multipartUploadUnderTest.discardUnpersistedData();
    return new S3RecoverableFsDataOutputStream(multipartUploadUnderTest, fileProvider, fileStream, USER_DEFINED_MIN_PART_SIZE, bytesBeforeCurrentPart);
}

Example 3 with RefCountedBufferingFileStream

use of org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream in project flink by apache.

the class S3RecoverableFsDataOutputStreamTest method beforeTest.

@Before
public void beforeTest() throws IOException {
    fileProvider = new TestFileProvider(TEMP_FOLDER);
    multipartUploadUnderTest = new TestMultipartUpload(fileProvider);
    RefCountedBufferingFileStream fileStream = RefCountedBufferingFileStream.openNew(fileProvider);
    streamUnderTest = new S3RecoverableFsDataOutputStream(multipartUploadUnderTest, fileProvider, fileStream, USER_DEFINED_MIN_PART_SIZE, 0L);
}

Example 4 with RefCountedBufferingFileStream

use of org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream in project flink by apache.

the class S3RecoverableFsDataOutputStream method newStream.

// ------------------------------------------------------------------------
// factory methods
// ------------------------------------------------------------------------
public static S3RecoverableFsDataOutputStream newStream(final RecoverableMultiPartUpload upload, final FunctionWithException<File, RefCountedFileWithStream, IOException> tmpFileCreator, final long userDefinedMinPartSize) throws IOException {
    checkArgument(userDefinedMinPartSize >= S3_MULTIPART_MIN_PART_SIZE);
    final RefCountedBufferingFileStream fileStream = boundedBufferingFileStream(tmpFileCreator, Optional.empty());
    return new S3RecoverableFsDataOutputStream(upload, tmpFileCreator, fileStream, userDefinedMinPartSize, 0L);
}

Example 5 with RefCountedBufferingFileStream

use of org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream in project flink by apache.

the class S3RecoverableFsDataOutputStream method recoverStream.

public static S3RecoverableFsDataOutputStream recoverStream(final RecoverableMultiPartUpload upload, final FunctionWithException<File, RefCountedFileWithStream, IOException> tmpFileCreator, final long userDefinedMinPartSize, final long bytesBeforeCurrentPart) throws IOException {
    checkArgument(userDefinedMinPartSize >= S3_MULTIPART_MIN_PART_SIZE);
    final RefCountedBufferingFileStream fileStream = boundedBufferingFileStream(tmpFileCreator, upload.getIncompletePart());
    return new S3RecoverableFsDataOutputStream(upload, tmpFileCreator, fileStream, userDefinedMinPartSize, bytesBeforeCurrentPart);
}