Search in sources :

Example 26 with AWSCredentialsProvider

use of com.amazonaws.auth.AWSCredentialsProvider in project components by Talend.

the class S3Connection method createClient.

public static AmazonS3 createClient(S3OutputProperties properties) {
    S3DatasetProperties data_set = properties.getDatasetProperties();
    S3DatastoreProperties data_store = properties.getDatasetProperties().getDatastoreProperties();
    com.amazonaws.auth.AWSCredentials credentials = new com.amazonaws.auth.BasicAWSCredentials(data_store.accessKey.getValue(), data_store.secretKey.getValue());
    Region region = RegionUtils.getRegion(data_set.region.getValue().getValue());
    Boolean clientSideEnc = data_set.encryptDataInMotion.getValue();
    AmazonS3 conn = null;
    if (clientSideEnc != null && clientSideEnc) {
        String kms_cmk = data_set.kmsForDataInMotion.getValue();
        KMSEncryptionMaterialsProvider encryptionMaterialsProvider = new KMSEncryptionMaterialsProvider(kms_cmk);
        conn = new AmazonS3EncryptionClient(credentials, encryptionMaterialsProvider, new CryptoConfiguration().withAwsKmsRegion(region));
    } else {
        AWSCredentialsProvider basicCredentialsProvider = new StaticCredentialsProvider(credentials);
        conn = new AmazonS3Client(basicCredentialsProvider);
    }
    conn.setRegion(region);
    return conn;
}
Also used : AmazonS3(com.amazonaws.services.s3.AmazonS3) AmazonS3EncryptionClient(com.amazonaws.services.s3.AmazonS3EncryptionClient) StaticCredentialsProvider(com.amazonaws.internal.StaticCredentialsProvider) KMSEncryptionMaterialsProvider(com.amazonaws.services.s3.model.KMSEncryptionMaterialsProvider) CryptoConfiguration(com.amazonaws.services.s3.model.CryptoConfiguration) S3DatasetProperties(org.talend.components.simplefileio.s3.S3DatasetProperties) AmazonS3Client(com.amazonaws.services.s3.AmazonS3Client) Region(com.amazonaws.regions.Region) S3DatastoreProperties(org.talend.components.simplefileio.s3.S3DatastoreProperties) AWSCredentialsProvider(com.amazonaws.auth.AWSCredentialsProvider)

Example 27 with AWSCredentialsProvider

use of com.amazonaws.auth.AWSCredentialsProvider in project herd by FINRAOS.

the class S3DaoTest method testGetAWSCredentialsProviderAssertStaticCredentialsSet.

@Test
public void testGetAWSCredentialsProviderAssertStaticCredentialsSet() {
    S3Operations originalS3Operations = (S3Operations) ReflectionTestUtils.getField(s3Dao, "s3Operations");
    S3Operations mockS3Operations = mock(S3Operations.class);
    ReflectionTestUtils.setField(s3Dao, "s3Operations", mockS3Operations);
    try {
        String s3BucketName = "s3BucketName";
        String s3KeyPrefix = "s3KeyPrefix";
        String s3AccessKey = "s3AccessKey";
        String s3SecretKey = "s3SecretKey";
        S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto = new S3FileTransferRequestParamsDto();
        s3FileTransferRequestParamsDto.setS3BucketName(s3BucketName);
        s3FileTransferRequestParamsDto.setS3KeyPrefix(s3KeyPrefix);
        s3FileTransferRequestParamsDto.setAwsAccessKeyId(s3AccessKey);
        s3FileTransferRequestParamsDto.setAwsSecretKey(s3SecretKey);
        when(mockS3Operations.putObject(any(), any())).then(new Answer<PutObjectResult>() {

            @SuppressWarnings("unchecked")
            @Override
            public PutObjectResult answer(InvocationOnMock invocation) throws Throwable {
                AmazonS3Client amazonS3Client = invocation.getArgument(1);
                AWSCredentialsProviderChain awsCredentialsProviderChain = (AWSCredentialsProviderChain) ReflectionTestUtils.getField(amazonS3Client, "awsCredentialsProvider");
                List<AWSCredentialsProvider> credentialsProviders = (List<AWSCredentialsProvider>) ReflectionTestUtils.getField(awsCredentialsProviderChain, "credentialsProviders");
                // Expect 2 providers: the static provider, and the default provider
                assertEquals(2, credentialsProviders.size());
                // Only verify the static value
                assertEquals(StaticCredentialsProvider.class, credentialsProviders.get(0).getClass());
                StaticCredentialsProvider staticCredentialsProvider = (StaticCredentialsProvider) credentialsProviders.get(0);
                assertEquals(s3AccessKey, staticCredentialsProvider.getCredentials().getAWSAccessKeyId());
                assertEquals(s3SecretKey, staticCredentialsProvider.getCredentials().getAWSSecretKey());
                return new PutObjectResult();
            }
        });
        s3Dao.createDirectory(s3FileTransferRequestParamsDto);
    } finally {
        ReflectionTestUtils.setField(s3Dao, "s3Operations", originalS3Operations);
    }
}
Also used : S3FileTransferRequestParamsDto(org.finra.herd.model.dto.S3FileTransferRequestParamsDto) PutObjectResult(com.amazonaws.services.s3.model.PutObjectResult) StaticCredentialsProvider(com.amazonaws.internal.StaticCredentialsProvider) DefaultAWSCredentialsProviderChain(com.amazonaws.auth.DefaultAWSCredentialsProviderChain) AWSCredentialsProviderChain(com.amazonaws.auth.AWSCredentialsProviderChain) AmazonS3Client(com.amazonaws.services.s3.AmazonS3Client) InvocationOnMock(org.mockito.invocation.InvocationOnMock) List(java.util.List) ArrayList(java.util.ArrayList) HerdAWSCredentialsProvider(org.finra.herd.model.dto.HerdAWSCredentialsProvider) AWSCredentialsProvider(com.amazonaws.auth.AWSCredentialsProvider) Test(org.junit.Test)

Example 28 with AWSCredentialsProvider

use of com.amazonaws.auth.AWSCredentialsProvider in project herd by FINRAOS.

the class S3DaoTest method testGetAWSCredentialsProviderAssertAdditionalProviderIsSet.

/**
 * A case where additional credentials provider is given in the request params. The credentials returned should be an AWS session credential where the
 * values are from the provided custom credentials provider.
 */
@Test
public void testGetAWSCredentialsProviderAssertAdditionalProviderIsSet() throws Exception {
    S3Operations originalS3Operations = (S3Operations) ReflectionTestUtils.getField(s3Dao, "s3Operations");
    S3Operations mockS3Operations = mock(S3Operations.class);
    ReflectionTestUtils.setField(s3Dao, "s3Operations", mockS3Operations);
    try {
        String s3BucketName = "s3BucketName";
        String s3KeyPrefix = "s3KeyPrefix";
        String awsAccessKey = "awsAccessKey";
        String awsSecretKey = "awsSecretKey";
        String awsSessionToken = "awsSessionToken";
        S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto = new S3FileTransferRequestParamsDto();
        s3FileTransferRequestParamsDto.setS3BucketName(s3BucketName);
        s3FileTransferRequestParamsDto.setS3KeyPrefix(s3KeyPrefix);
        s3FileTransferRequestParamsDto.setAdditionalAwsCredentialsProviders(Arrays.asList(new HerdAWSCredentialsProvider() {

            @Override
            public AwsCredential getAwsCredential() {
                return new AwsCredential(awsAccessKey, awsSecretKey, awsSessionToken, null);
            }
        }));
        when(mockS3Operations.putObject(any(), any())).then(new Answer<PutObjectResult>() {

            @SuppressWarnings("unchecked")
            @Override
            public PutObjectResult answer(InvocationOnMock invocation) throws Throwable {
                AmazonS3Client amazonS3Client = invocation.getArgument(1);
                AWSCredentialsProviderChain awsCredentialsProviderChain = (AWSCredentialsProviderChain) ReflectionTestUtils.getField(amazonS3Client, "awsCredentialsProvider");
                List<AWSCredentialsProvider> credentialsProviders = (List<AWSCredentialsProvider>) ReflectionTestUtils.getField(awsCredentialsProviderChain, "credentialsProviders");
                assertEquals(2, credentialsProviders.size());
                // refresh() does nothing, but gives code coverage
                credentialsProviders.get(0).refresh();
                /*
                     * We can't inspect the field directly since the class definition is private.
                     * Instead we call the getCredentials() and verify that it returns the credentials staged as part of this test.
                     */
                AWSCredentials credentials = awsCredentialsProviderChain.getCredentials();
                assertEquals(BasicSessionCredentials.class, credentials.getClass());
                BasicSessionCredentials basicSessionCredentials = (BasicSessionCredentials) credentials;
                assertEquals(awsAccessKey, basicSessionCredentials.getAWSAccessKeyId());
                assertEquals(awsSecretKey, basicSessionCredentials.getAWSSecretKey());
                assertEquals(awsSessionToken, basicSessionCredentials.getSessionToken());
                return new PutObjectResult();
            }
        });
        s3Dao.createDirectory(s3FileTransferRequestParamsDto);
    } finally {
        ReflectionTestUtils.setField(s3Dao, "s3Operations", originalS3Operations);
    }
}
Also used : BasicSessionCredentials(com.amazonaws.auth.BasicSessionCredentials) S3FileTransferRequestParamsDto(org.finra.herd.model.dto.S3FileTransferRequestParamsDto) PutObjectResult(com.amazonaws.services.s3.model.PutObjectResult) AWSCredentials(com.amazonaws.auth.AWSCredentials) DefaultAWSCredentialsProviderChain(com.amazonaws.auth.DefaultAWSCredentialsProviderChain) AWSCredentialsProviderChain(com.amazonaws.auth.AWSCredentialsProviderChain) HerdAWSCredentialsProvider(org.finra.herd.model.dto.HerdAWSCredentialsProvider) AmazonS3Client(com.amazonaws.services.s3.AmazonS3Client) InvocationOnMock(org.mockito.invocation.InvocationOnMock) List(java.util.List) ArrayList(java.util.ArrayList) AwsCredential(org.finra.herd.model.api.xml.AwsCredential) HerdAWSCredentialsProvider(org.finra.herd.model.dto.HerdAWSCredentialsProvider) AWSCredentialsProvider(com.amazonaws.auth.AWSCredentialsProvider) Test(org.junit.Test)

Example 29 with AWSCredentialsProvider

use of com.amazonaws.auth.AWSCredentialsProvider in project herd by FINRAOS.

the class S3DaoTest method testGetAWSCredentialsProviderAssertStaticCredentialsIsNotSetWhenSecretKeyIsNull.

@Test
public void testGetAWSCredentialsProviderAssertStaticCredentialsIsNotSetWhenSecretKeyIsNull() {
    S3Operations originalS3Operations = (S3Operations) ReflectionTestUtils.getField(s3Dao, "s3Operations");
    S3Operations mockS3Operations = mock(S3Operations.class);
    ReflectionTestUtils.setField(s3Dao, "s3Operations", mockS3Operations);
    try {
        String s3BucketName = "s3BucketName";
        String s3KeyPrefix = "s3KeyPrefix";
        String s3AccessKey = "s3AccessKey";
        String s3SecretKey = null;
        S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto = new S3FileTransferRequestParamsDto();
        s3FileTransferRequestParamsDto.setS3BucketName(s3BucketName);
        s3FileTransferRequestParamsDto.setS3KeyPrefix(s3KeyPrefix);
        s3FileTransferRequestParamsDto.setAwsAccessKeyId(s3AccessKey);
        s3FileTransferRequestParamsDto.setAwsSecretKey(s3SecretKey);
        when(mockS3Operations.putObject(any(), any())).then(new Answer<PutObjectResult>() {

            @SuppressWarnings("unchecked")
            @Override
            public PutObjectResult answer(InvocationOnMock invocation) throws Throwable {
                AmazonS3Client amazonS3Client = invocation.getArgument(1);
                AWSCredentialsProviderChain awsCredentialsProviderChain = (AWSCredentialsProviderChain) ReflectionTestUtils.getField(amazonS3Client, "awsCredentialsProvider");
                List<AWSCredentialsProvider> credentialsProviders = (List<AWSCredentialsProvider>) ReflectionTestUtils.getField(awsCredentialsProviderChain, "credentialsProviders");
                assertEquals(1, credentialsProviders.size());
                assertEquals(DefaultAWSCredentialsProviderChain.class, credentialsProviders.get(0).getClass());
                return new PutObjectResult();
            }
        });
        s3Dao.createDirectory(s3FileTransferRequestParamsDto);
    } finally {
        ReflectionTestUtils.setField(s3Dao, "s3Operations", originalS3Operations);
    }
}
Also used : DefaultAWSCredentialsProviderChain(com.amazonaws.auth.DefaultAWSCredentialsProviderChain) S3FileTransferRequestParamsDto(org.finra.herd.model.dto.S3FileTransferRequestParamsDto) PutObjectResult(com.amazonaws.services.s3.model.PutObjectResult) DefaultAWSCredentialsProviderChain(com.amazonaws.auth.DefaultAWSCredentialsProviderChain) AWSCredentialsProviderChain(com.amazonaws.auth.AWSCredentialsProviderChain) AmazonS3Client(com.amazonaws.services.s3.AmazonS3Client) InvocationOnMock(org.mockito.invocation.InvocationOnMock) List(java.util.List) ArrayList(java.util.ArrayList) HerdAWSCredentialsProvider(org.finra.herd.model.dto.HerdAWSCredentialsProvider) AWSCredentialsProvider(com.amazonaws.auth.AWSCredentialsProvider) Test(org.junit.Test)

Example 30 with AWSCredentialsProvider

use of com.amazonaws.auth.AWSCredentialsProvider in project herd by FINRAOS.

the class S3DaoImpl method getAWSCredentialsProvider.

/**
 * <p> Gets the {@link AWSCredentialsProvider} based on the credentials in the given parameters. </p> <p> Returns {@link DefaultAWSCredentialsProviderChain}
 * if either access or secret key is {@code null}. Otherwise returns a {@link StaticCredentialsProvider} with the credentials. </p>
 *
 * @param params - Access parameters
 *
 * @return AWS credentials provider implementation
 */
private AWSCredentialsProvider getAWSCredentialsProvider(S3FileTransferRequestParamsDto params) {
    List<AWSCredentialsProvider> providers = new ArrayList<>();
    String accessKey = params.getAwsAccessKeyId();
    String secretKey = params.getAwsSecretKey();
    if (accessKey != null && secretKey != null) {
        providers.add(new StaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)));
    }
    for (HerdAWSCredentialsProvider herdAWSCredentialsProvider : params.getAdditionalAwsCredentialsProviders()) {
        providers.add(new HerdAwsCredentialsProviderWrapper(herdAWSCredentialsProvider));
    }
    providers.add(new DefaultAWSCredentialsProviderChain());
    return new AWSCredentialsProviderChain(providers.toArray(new AWSCredentialsProvider[providers.size()]));
}
Also used : DefaultAWSCredentialsProviderChain(com.amazonaws.auth.DefaultAWSCredentialsProviderChain) HerdAWSCredentialsProvider(org.finra.herd.model.dto.HerdAWSCredentialsProvider) StaticCredentialsProvider(com.amazonaws.internal.StaticCredentialsProvider) ArrayList(java.util.ArrayList) HerdAWSCredentialsProvider(org.finra.herd.model.dto.HerdAWSCredentialsProvider) AWSCredentialsProvider(com.amazonaws.auth.AWSCredentialsProvider) BasicAWSCredentials(com.amazonaws.auth.BasicAWSCredentials) DefaultAWSCredentialsProviderChain(com.amazonaws.auth.DefaultAWSCredentialsProviderChain) AWSCredentialsProviderChain(com.amazonaws.auth.AWSCredentialsProviderChain)

Aggregations

AWSCredentialsProvider (com.amazonaws.auth.AWSCredentialsProvider)118 Test (org.junit.Test)71 DefaultAWSCredentialsProviderChain (com.amazonaws.auth.DefaultAWSCredentialsProviderChain)24 BasicAWSCredentials (com.amazonaws.auth.BasicAWSCredentials)19 AWSStaticCredentialsProvider (com.amazonaws.auth.AWSStaticCredentialsProvider)17 AWSCredentials (com.amazonaws.auth.AWSCredentials)15 TestRunner (org.apache.nifi.util.TestRunner)15 ClientConfiguration (com.amazonaws.ClientConfiguration)12 AmazonS3Client (com.amazonaws.services.s3.AmazonS3Client)12 STSAssumeRoleSessionCredentialsProvider (com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider)8 ProfileCredentialsProvider (com.amazonaws.auth.profile.ProfileCredentialsProvider)8 PropertyDescriptor (org.apache.nifi.components.PropertyDescriptor)8 ClasspathPropertiesFileCredentialsProvider (com.amazonaws.auth.ClasspathPropertiesFileCredentialsProvider)7 SystemPropertiesCredentialsProvider (com.amazonaws.auth.SystemPropertiesCredentialsProvider)7 Properties (java.util.Properties)7 PrepareForTest (org.powermock.core.classloader.annotations.PrepareForTest)7 AWSCredentialsProviderChain (com.amazonaws.auth.AWSCredentialsProviderChain)6 EnvironmentVariableCredentialsProvider (com.amazonaws.auth.EnvironmentVariableCredentialsProvider)6 Regions (com.amazonaws.regions.Regions)6 ArrayList (java.util.ArrayList)6