Search in sources :

Example 26 with MultipartUpload

use of com.amazonaws.services.s3.model.MultipartUpload in project bender by Nextdoor.

the class S3TransporterTest method testMultipleUploads.

@Test
public void testMultipleUploads() throws TransportException, IllegalStateException, IOException {
    /*
     * Create mock client, requets, and replies
     */
    AmazonS3Client mockClient = getMockClient();
    /*
     * Fill buffer with mock data
     */
    S3TransportBuffer buffer1 = new S3TransportBuffer(1000, false, new S3TransportSerializer());
    S3TransportBuffer buffer2 = new S3TransportBuffer(1000, false, new S3TransportSerializer());
    InternalEvent mockIevent = mock(InternalEvent.class);
    doReturn("foo").doReturn("bar1").when(mockIevent).getSerialized();
    buffer1.add(mockIevent);
    buffer2.add(mockIevent);
    /*
     * Create transport
     */
    Map<String, MultiPartUpload> multiPartUploads = new HashMap<String, MultiPartUpload>(0);
    S3Transport transport = new S3Transport(mockClient, "bucket", "basepath", false, multiPartUploads);
    /*
     * Do actual test
     */
    LinkedHashMap<String, String> partitions = new LinkedHashMap<String, String>();
    partitions.put(S3Transport.FILENAME_KEY, "a_filename");
    ArgumentCaptor<UploadPartRequest> argument = ArgumentCaptor.forClass(UploadPartRequest.class);
    transport.sendBatch(buffer1, partitions, new TestContext());
    transport.sendBatch(buffer2, partitions, new TestContext());
    verify(mockClient, times(2)).uploadPart(argument.capture());
    List<UploadPartRequest> arguments = argument.getAllValues();
    assertEquals(1, arguments.get(0).getPartNumber());
    // foo\n
    assertEquals(4, arguments.get(0).getPartSize());
    assertEquals("123", arguments.get(0).getUploadId());
    assertEquals(2, arguments.get(1).getPartNumber());
    // bar1\n
    assertEquals(5, arguments.get(1).getPartSize());
    assertEquals("123", arguments.get(1).getUploadId());
}
Also used : HashMap(java.util.HashMap) LinkedHashMap(java.util.LinkedHashMap) TestContext(com.nextdoor.bender.aws.TestContext) UploadPartRequest(com.amazonaws.services.s3.model.UploadPartRequest) InternalEvent(com.nextdoor.bender.InternalEvent) LinkedHashMap(java.util.LinkedHashMap) AmazonS3Client(com.amazonaws.services.s3.AmazonS3Client) Test(org.junit.Test)

Aggregations

Test (org.junit.Test)15 AmazonS3Client (com.amazonaws.services.s3.AmazonS3Client)13 MultipartUpload (com.amazonaws.services.s3.model.MultipartUpload)11 UploadPartRequest (com.amazonaws.services.s3.model.UploadPartRequest)11 MultipartUploadListing (com.amazonaws.services.s3.model.MultipartUploadListing)10 HashMap (java.util.HashMap)10 InternalEvent (com.nextdoor.bender.InternalEvent)9 TestContext (com.nextdoor.bender.aws.TestContext)9 LinkedHashMap (java.util.LinkedHashMap)9 ArrayList (java.util.ArrayList)8 AmazonClientException (com.amazonaws.AmazonClientException)7 AmazonS3Exception (com.amazonaws.services.s3.model.AmazonS3Exception)6 ListMultipartUploadsRequest (com.amazonaws.services.s3.model.ListMultipartUploadsRequest)5 AbortMultipartUploadRequest (com.amazonaws.services.s3.model.AbortMultipartUploadRequest)4 ObjectMetadata (com.amazonaws.services.s3.model.ObjectMetadata)4 TestRunner (org.apache.nifi.util.TestRunner)4 InvocationOnMock (org.mockito.invocation.InvocationOnMock)4 Answer (org.mockito.stubbing.Answer)4 InitiateMultipartUploadRequest (com.amazonaws.services.s3.model.InitiateMultipartUploadRequest)3 InitiateMultipartUploadResult (com.amazonaws.services.s3.model.InitiateMultipartUploadResult)3