Search in sources :

Example 41 with S3FileTransferRequestParamsDto

use of org.finra.herd.model.dto.S3FileTransferRequestParamsDto in project herd by FINRAOS.

the class S3DaoTest method testPrepareMetadataAssertSetKmsHeaders.

@Test
public void testPrepareMetadataAssertSetKmsHeaders() {
    S3Operations originalS3Operations = (S3Operations) ReflectionTestUtils.getField(s3Dao, "s3Operations");
    S3Operations mockS3Operations = mock(S3Operations.class);
    ReflectionTestUtils.setField(s3Dao, "s3Operations", mockS3Operations);
    try {
        String s3BucketName = "s3BucketName";
        String s3KeyPrefix = "s3KeyPrefix";
        String kmsKeyId = "kmsKeyId";
        S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto = new S3FileTransferRequestParamsDto();
        s3FileTransferRequestParamsDto.setS3BucketName(s3BucketName);
        s3FileTransferRequestParamsDto.setS3KeyPrefix(s3KeyPrefix);
        s3FileTransferRequestParamsDto.setKmsKeyId(kmsKeyId);
        when(mockS3Operations.putObject(any(), any())).then(new Answer<PutObjectResult>() {

            @Override
            public PutObjectResult answer(InvocationOnMock invocation) throws Throwable {
                PutObjectRequest putObjectRequest = invocation.getArgument(0);
                ObjectMetadata metadata = putObjectRequest.getMetadata();
                assertEquals("aws:kms", metadata.getSSEAlgorithm());
                assertEquals(kmsKeyId, metadata.getRawMetadata().get(Headers.SERVER_SIDE_ENCRYPTION_AWS_KMS_KEYID));
                return new PutObjectResult();
            }
        });
        s3Dao.createDirectory(s3FileTransferRequestParamsDto);
    } finally {
        ReflectionTestUtils.setField(s3Dao, "s3Operations", originalS3Operations);
    }
}
Also used : S3FileTransferRequestParamsDto(org.finra.herd.model.dto.S3FileTransferRequestParamsDto) PutObjectResult(com.amazonaws.services.s3.model.PutObjectResult) InvocationOnMock(org.mockito.invocation.InvocationOnMock) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest) Test(org.junit.Test)

Example 42 with S3FileTransferRequestParamsDto

use of org.finra.herd.model.dto.S3FileTransferRequestParamsDto in project herd by FINRAOS.

the class S3DaoTest method testDeleteFileListAssertHandleGenericExceptions.

@Test
public void testDeleteFileListAssertHandleGenericExceptions() {
    S3Operations originalS3Operations = (S3Operations) ReflectionTestUtils.getField(s3Dao, "s3Operations");
    S3Operations mockS3Operations = mock(S3Operations.class);
    ReflectionTestUtils.setField(s3Dao, "s3Operations", mockS3Operations);
    try {
        S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto = new S3FileTransferRequestParamsDto();
        s3FileTransferRequestParamsDto.setS3BucketName("s3BucketName");
        s3FileTransferRequestParamsDto.setFiles(Arrays.asList(new File("file")));
        when(mockS3Operations.deleteObjects(any(), any())).thenThrow(new AmazonServiceException("testException"));
        try {
            s3Dao.deleteFileList(s3FileTransferRequestParamsDto);
            fail();
        } catch (Exception e) {
            assertEquals(IllegalStateException.class, e.getClass());
            assertEquals("Failed to delete a list of keys from bucket \"s3BucketName\". Reason: testException (Service: null; Status Code: 0; Error Code: null; " + "Request ID: null)", e.getMessage());
        }
    } finally {
        ReflectionTestUtils.setField(s3Dao, "s3Operations", originalS3Operations);
    }
}
Also used : S3FileTransferRequestParamsDto(org.finra.herd.model.dto.S3FileTransferRequestParamsDto) AmazonServiceException(com.amazonaws.AmazonServiceException) File(java.io.File) MultiObjectDeleteException(com.amazonaws.services.s3.model.MultiObjectDeleteException) ObjectNotFoundException(org.finra.herd.model.ObjectNotFoundException) AmazonServiceException(com.amazonaws.AmazonServiceException) AmazonClientException(com.amazonaws.AmazonClientException) AmazonS3Exception(com.amazonaws.services.s3.model.AmazonS3Exception) IOException(java.io.IOException) Test(org.junit.Test)

Example 43 with S3FileTransferRequestParamsDto

use of org.finra.herd.model.dto.S3FileTransferRequestParamsDto in project herd by FINRAOS.

the class S3DaoTest method testCreateDirectoryAssertAddTrailingSlashOnlyIfNotPresent.

@Test
public void testCreateDirectoryAssertAddTrailingSlashOnlyIfNotPresent() {
    S3Operations originalS3Operations = (S3Operations) ReflectionTestUtils.getField(s3Dao, "s3Operations");
    S3Operations mockS3Operations = mock(S3Operations.class);
    ReflectionTestUtils.setField(s3Dao, "s3Operations", mockS3Operations);
    try {
        String s3BucketName = "s3BucketName";
        String s3KeyPrefix = "s3KeyPrefix/";
        S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto = new S3FileTransferRequestParamsDto();
        s3FileTransferRequestParamsDto.setS3BucketName(s3BucketName);
        s3FileTransferRequestParamsDto.setS3KeyPrefix(s3KeyPrefix);
        when(mockS3Operations.putObject(any(), any())).then(new Answer<PutObjectResult>() {

            @Override
            public PutObjectResult answer(InvocationOnMock invocation) throws Throwable {
                PutObjectRequest putObjectRequest = invocation.getArgument(0);
                assertEquals(s3BucketName, putObjectRequest.getBucketName());
                assertEquals(s3KeyPrefix, putObjectRequest.getKey());
                PutObjectResult putObjectResult = new PutObjectResult();
                return putObjectResult;
            }
        });
        s3Dao.createDirectory(s3FileTransferRequestParamsDto);
    } finally {
        ReflectionTestUtils.setField(s3Dao, "s3Operations", originalS3Operations);
    }
}
Also used : S3FileTransferRequestParamsDto(org.finra.herd.model.dto.S3FileTransferRequestParamsDto) PutObjectResult(com.amazonaws.services.s3.model.PutObjectResult) InvocationOnMock(org.mockito.invocation.InvocationOnMock) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest) Test(org.junit.Test)

Example 44 with S3FileTransferRequestParamsDto

use of org.finra.herd.model.dto.S3FileTransferRequestParamsDto in project herd by FINRAOS.

the class S3DaoTest method testGetAmazonS3AssertS3EndpointIsSet.

@Test
public void testGetAmazonS3AssertS3EndpointIsSet() {
    S3Operations originalS3Operations = (S3Operations) ReflectionTestUtils.getField(s3Dao, "s3Operations");
    S3Operations mockS3Operations = mock(S3Operations.class);
    ReflectionTestUtils.setField(s3Dao, "s3Operations", mockS3Operations);
    try {
        String s3BucketName = "s3BucketName";
        String s3KeyPrefix = "s3KeyPrefix";
        String s3Endpoint = "s3Endpoint";
        S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto = new S3FileTransferRequestParamsDto();
        s3FileTransferRequestParamsDto.setS3BucketName(s3BucketName);
        s3FileTransferRequestParamsDto.setS3KeyPrefix(s3KeyPrefix);
        s3FileTransferRequestParamsDto.setS3Endpoint(s3Endpoint);
        when(mockS3Operations.putObject(any(), any())).then(new Answer<PutObjectResult>() {

            @Override
            public PutObjectResult answer(InvocationOnMock invocation) throws Throwable {
                AmazonS3Client amazonS3Client = invocation.getArgument(1);
                assertEquals(new URI("https://" + s3Endpoint), ReflectionTestUtils.getField(amazonS3Client, "endpoint"));
                return new PutObjectResult();
            }
        });
        s3Dao.createDirectory(s3FileTransferRequestParamsDto);
    } finally {
        ReflectionTestUtils.setField(s3Dao, "s3Operations", originalS3Operations);
    }
}
Also used : AmazonS3Client(com.amazonaws.services.s3.AmazonS3Client) S3FileTransferRequestParamsDto(org.finra.herd.model.dto.S3FileTransferRequestParamsDto) PutObjectResult(com.amazonaws.services.s3.model.PutObjectResult) InvocationOnMock(org.mockito.invocation.InvocationOnMock) URI(java.net.URI) Test(org.junit.Test)

Example 45 with S3FileTransferRequestParamsDto

use of org.finra.herd.model.dto.S3FileTransferRequestParamsDto in project herd by FINRAOS.

the class S3DaoTest method testAbortMultipartUploadsAssertAbortOnlyBeforeThreshold.

@Test
public void testAbortMultipartUploadsAssertAbortOnlyBeforeThreshold() {
    S3Operations originalS3Operations = (S3Operations) ReflectionTestUtils.getField(s3Dao, "s3Operations");
    S3Operations mockS3Operations = mock(S3Operations.class);
    ReflectionTestUtils.setField(s3Dao, "s3Operations", mockS3Operations);
    try {
        String s3BucketName = "s3BucketName";
        String uploadKey = "uploadKey1";
        String uploadId = "uploadId1";
        Date uploadInitiated = new Date(0);
        S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto = new S3FileTransferRequestParamsDto();
        s3FileTransferRequestParamsDto.setS3BucketName(s3BucketName);
        Date thresholdDate = new Date(1);
        when(mockS3Operations.listMultipartUploads(any(), any())).then(new Answer<MultipartUploadListing>() {

            @Override
            public MultipartUploadListing answer(InvocationOnMock invocation) throws Throwable {
                ListMultipartUploadsRequest listMultipartUploadsRequest = invocation.getArgument(0);
                assertEquals(s3BucketName, listMultipartUploadsRequest.getBucketName());
                MultipartUploadListing multipartUploadListing = new MultipartUploadListing();
                {
                    MultipartUpload multipartUpload = new MultipartUpload();
                    multipartUpload.setUploadId(uploadId);
                    multipartUpload.setKey(uploadKey);
                    multipartUpload.setInitiated(uploadInitiated);
                    multipartUploadListing.getMultipartUploads().add(multipartUpload);
                }
                // This upload is not aborted since the initiated date is greater than the threshold
                {
                    MultipartUpload multipartUpload = new MultipartUpload();
                    multipartUpload.setUploadId("uploadId2");
                    multipartUpload.setKey("uploadKey2");
                    multipartUpload.setInitiated(new Date(2));
                    multipartUploadListing.getMultipartUploads().add(multipartUpload);
                }
                return multipartUploadListing;
            }
        });
        assertEquals(1, s3Dao.abortMultipartUploads(s3FileTransferRequestParamsDto, thresholdDate));
        verify(mockS3Operations).listMultipartUploads(any(), any());
        /*
             * Assert that S3Operations.abortMultipartUpload is called exactly ONCE with arguments matching the given ArgumentMatcher
             */
        verify(mockS3Operations).abortMultipartUpload(argThat(argument -> Objects.equal(s3BucketName, argument.getBucketName()) && Objects.equal(uploadKey, argument.getKey()) && Objects.equal(uploadId, argument.getUploadId())), any());
        // Assert that no other interactions occur with the mock
        verifyNoMoreInteractions(mockS3Operations);
    } finally {
        ReflectionTestUtils.setField(s3Dao, "s3Operations", originalS3Operations);
    }
}
Also used : MultiObjectDeleteException(com.amazonaws.services.s3.model.MultiObjectDeleteException) S3FileTransferRequestParamsDto(org.finra.herd.model.dto.S3FileTransferRequestParamsDto) Arrays(java.util.Arrays) ObjectNotFoundException(org.finra.herd.model.ObjectNotFoundException) Date(java.util.Date) DefaultAWSCredentialsProviderChain(com.amazonaws.auth.DefaultAWSCredentialsProviderChain) TransferState(com.amazonaws.services.s3.transfer.Transfer.TransferState) TransferProgress(com.amazonaws.services.s3.transfer.TransferProgress) ObjectListing(com.amazonaws.services.s3.model.ObjectListing) Copy(com.amazonaws.services.s3.transfer.Copy) ByteArrayInputStream(java.io.ByteArrayInputStream) Mockito.verifyNoMoreInteractions(org.mockito.Mockito.verifyNoMoreInteractions) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) After(org.junit.After) KeyVersion(com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion) Assert.fail(org.junit.Assert.fail) URI(java.net.URI) S3DaoImpl(org.finra.herd.dao.impl.S3DaoImpl) Objects(com.google.common.base.Objects) Path(java.nio.file.Path) AWSCredentials(com.amazonaws.auth.AWSCredentials) AmazonServiceException(com.amazonaws.AmazonServiceException) Upload(com.amazonaws.services.s3.transfer.Upload) AmazonS3Client(com.amazonaws.services.s3.AmazonS3Client) Transfer(com.amazonaws.services.s3.transfer.Transfer) Matchers.any(org.mockito.Matchers.any) S3FileTransferResultsDto(org.finra.herd.model.dto.S3FileTransferResultsDto) List(java.util.List) Matchers.argThat(org.mockito.Matchers.argThat) HerdAWSCredentialsProvider(org.finra.herd.model.dto.HerdAWSCredentialsProvider) MultipartUploadListing(com.amazonaws.services.s3.model.MultipartUploadListing) DeletedObject(com.amazonaws.services.s3.model.DeleteObjectsResult.DeletedObject) AmazonClientException(com.amazonaws.AmazonClientException) Mockito.mock(org.mockito.Mockito.mock) AWSCredentialsProviderChain(com.amazonaws.auth.AWSCredentialsProviderChain) MultipartUpload(com.amazonaws.services.s3.model.MultipartUpload) S3FileCopyRequestParamsDto(org.finra.herd.model.dto.S3FileCopyRequestParamsDto) VersionListing(com.amazonaws.services.s3.model.VersionListing) ArrayList(java.util.ArrayList) BasicSessionCredentials(com.amazonaws.auth.BasicSessionCredentials) Answer(org.mockito.stubbing.Answer) InvocationOnMock(org.mockito.invocation.InvocationOnMock) S3Object(com.amazonaws.services.s3.model.S3Object) AWSCredentialsProvider(com.amazonaws.auth.AWSCredentialsProvider) ObjectTagging(com.amazonaws.services.s3.model.ObjectTagging) S3ObjectSummary(com.amazonaws.services.s3.model.S3ObjectSummary) PutObjectResult(com.amazonaws.services.s3.model.PutObjectResult) Before(org.junit.Before) AmazonS3Exception(com.amazonaws.services.s3.model.AmazonS3Exception) JavaPropertiesHelper(org.finra.herd.dao.helper.JavaPropertiesHelper) Properties(java.util.Properties) ListMultipartUploadsRequest(com.amazonaws.services.s3.model.ListMultipartUploadsRequest) Files(java.nio.file.Files) ListObjectsRequest(com.amazonaws.services.s3.model.ListObjectsRequest) StringWriter(java.io.StringWriter) Assert.assertTrue(org.junit.Assert.assertTrue) ReflectionTestUtils(org.springframework.test.util.ReflectionTestUtils) Mockito.times(org.mockito.Mockito.times) IOException(java.io.IOException) FileUtils(org.apache.commons.io.FileUtils) Test(org.junit.Test) LogLevel(org.finra.herd.core.helper.LogLevel) Mockito.when(org.mockito.Mockito.when) Tag(com.amazonaws.services.s3.model.Tag) GetObjectTaggingRequest(com.amazonaws.services.s3.model.GetObjectTaggingRequest) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest) File(java.io.File) Mockito.verify(org.mockito.Mockito.verify) StaticCredentialsProvider(com.amazonaws.internal.StaticCredentialsProvider) DeleteError(com.amazonaws.services.s3.model.MultiObjectDeleteException.DeleteError) StorageClass(com.amazonaws.services.s3.model.StorageClass) Assert.assertNull(org.junit.Assert.assertNull) ClientConfiguration(com.amazonaws.ClientConfiguration) Headers(com.amazonaws.services.s3.Headers) Ignore(org.junit.Ignore) Paths(java.nio.file.Paths) ListVersionsRequest(com.amazonaws.services.s3.model.ListVersionsRequest) S3VersionSummary(com.amazonaws.services.s3.model.S3VersionSummary) MockS3OperationsImpl(org.finra.herd.dao.impl.MockS3OperationsImpl) GetObjectTaggingResult(com.amazonaws.services.s3.model.GetObjectTaggingResult) Assert(org.junit.Assert) AwsCredential(org.finra.herd.model.api.xml.AwsCredential) Assert.assertEquals(org.junit.Assert.assertEquals) InputStream(java.io.InputStream) S3FileTransferRequestParamsDto(org.finra.herd.model.dto.S3FileTransferRequestParamsDto) InvocationOnMock(org.mockito.invocation.InvocationOnMock) MultipartUploadListing(com.amazonaws.services.s3.model.MultipartUploadListing) MultipartUpload(com.amazonaws.services.s3.model.MultipartUpload) Date(java.util.Date) ListMultipartUploadsRequest(com.amazonaws.services.s3.model.ListMultipartUploadsRequest) Test(org.junit.Test)

Aggregations

S3FileTransferRequestParamsDto (org.finra.herd.model.dto.S3FileTransferRequestParamsDto)160 Test (org.junit.Test)119 File (java.io.File)41 PutObjectRequest (com.amazonaws.services.s3.model.PutObjectRequest)31 ByteArrayInputStream (java.io.ByteArrayInputStream)30 BusinessObjectDataKey (org.finra.herd.model.api.xml.BusinessObjectDataKey)27 ObjectMetadata (com.amazonaws.services.s3.model.ObjectMetadata)23 ObjectNotFoundException (org.finra.herd.model.ObjectNotFoundException)23 S3FileTransferResultsDto (org.finra.herd.model.dto.S3FileTransferResultsDto)23 IOException (java.io.IOException)21 AmazonServiceException (com.amazonaws.AmazonServiceException)20 MultiObjectDeleteException (com.amazonaws.services.s3.model.MultiObjectDeleteException)20 InvocationOnMock (org.mockito.invocation.InvocationOnMock)20 AmazonClientException (com.amazonaws.AmazonClientException)19 AmazonS3Exception (com.amazonaws.services.s3.model.AmazonS3Exception)19 ArrayList (java.util.ArrayList)19 S3ObjectSummary (com.amazonaws.services.s3.model.S3ObjectSummary)18 Tag (com.amazonaws.services.s3.model.Tag)16 StorageFile (org.finra.herd.model.api.xml.StorageFile)15 AmazonS3Client (com.amazonaws.services.s3.AmazonS3Client)14