Search in sources :

Example 86 with ClientConfiguration

use of com.amazonaws.ClientConfiguration in project herd by FINRAOS.

the class CredStashHelperTest method testGetCredentialFromCredStash.

@Test
public void testGetCredentialFromCredStash() throws Exception {
    // Build AWS parameters.
    AwsParamsDto awsParamsDto = new AwsParamsDto(NO_AWS_ACCESS_KEY, NO_AWS_SECRET_KEY, NO_SESSION_TOKEN, HTTP_PROXY_HOST, HTTP_PROXY_PORT);
    // Build AWS client configuration.
    ClientConfiguration clientConfiguration = new ClientConfiguration();
    // Create CredStash encryption context map.
    Map<String, String> credStashEncryptionContextMap = new HashMap<>();
    credStashEncryptionContextMap.put(KEY, VALUE);
    // Mock the CredStash.
    CredStash credStash = mock(CredStash.class);
    when(credStash.getCredential(USER_CREDENTIAL_NAME, credStashEncryptionContextMap)).thenReturn(PASSWORD);
    // Mock the external calls.
    when(configurationHelper.getProperty(ConfigurationValue.CREDSTASH_AWS_REGION_NAME)).thenReturn(AWS_REGION_NAME);
    when(configurationHelper.getProperty(ConfigurationValue.CREDSTASH_TABLE_NAME)).thenReturn(TABLE_NAME);
    when(awsHelper.getAwsParamsDto()).thenReturn(awsParamsDto);
    when(awsHelper.getClientConfiguration(awsParamsDto)).thenReturn(clientConfiguration);
    when(credStashFactory.getCredStash(AWS_REGION_NAME, TABLE_NAME, clientConfiguration)).thenReturn(credStash);
    when(jsonHelper.unmarshallJsonToObject(Map.class, CREDSTASH_ENCRYPTION_CONTEXT)).thenReturn(credStashEncryptionContextMap);
    // Call the method under test.
    String result = credStashHelper.getCredentialFromCredStash(CREDSTASH_ENCRYPTION_CONTEXT, USER_CREDENTIAL_NAME);
    // Verify the external calls.
    verify(configurationHelper).getProperty(ConfigurationValue.CREDSTASH_AWS_REGION_NAME);
    verify(configurationHelper).getProperty(ConfigurationValue.CREDSTASH_TABLE_NAME);
    verify(awsHelper).getAwsParamsDto();
    verify(awsHelper).getClientConfiguration(awsParamsDto);
    verify(credStashFactory).getCredStash(AWS_REGION_NAME, TABLE_NAME, clientConfiguration);
    verify(jsonHelper).unmarshallJsonToObject(Map.class, CREDSTASH_ENCRYPTION_CONTEXT);
    verify(credStash).getCredential(USER_CREDENTIAL_NAME, credStashEncryptionContextMap);
    verifyNoMoreInteractions(credStash);
    verifyNoMoreInteractionsHelper();
    // Validate the results.
    assertEquals(PASSWORD, result);
}
Also used : AwsParamsDto(org.finra.herd.model.dto.AwsParamsDto) HashMap(java.util.HashMap) CredStash(org.finra.herd.dao.credstash.CredStash) ClientConfiguration(com.amazonaws.ClientConfiguration) Test(org.junit.Test) AbstractDaoTest(org.finra.herd.dao.AbstractDaoTest)

Example 87 with ClientConfiguration

use of com.amazonaws.ClientConfiguration in project herd by FINRAOS.

the class S3DaoImpl method getAmazonS3.

/**
 * Gets a new S3 client based on the specified parameters. The HTTP proxy information will be added if the host and port are specified in the parameters.
 *
 * @param params the parameters.
 *
 * @return the Amazon S3 client.
 */
private AmazonS3Client getAmazonS3(S3FileTransferRequestParamsDto params) {
    AmazonS3Client amazonS3Client;
    ClientConfiguration clientConfiguration = new ClientConfiguration().withRetryPolicy(retryPolicyFactory.getRetryPolicy());
    // Set the proxy configuration, if proxy is specified.
    if (StringUtils.isNotBlank(params.getHttpProxyHost()) && params.getHttpProxyPort() != null) {
        clientConfiguration.setProxyHost(params.getHttpProxyHost());
        clientConfiguration.setProxyPort(params.getHttpProxyPort());
    }
    // Sign all S3 API's with V4 signing.
    // AmazonS3Client.upgradeToSigV4 already has some scenarios where it will "upgrade" the signing approach to use V4 if not already present (e.g.
    // GetObjectRequest and KMS PutObjectRequest), but setting it here (especially when KMS is used) will ensure it isn't missed when required (e.g.
    // copying objects between KMS encrypted buckets). Otherwise, AWS will return a bad request error and retry which isn't desirable.
    clientConfiguration.setSignerOverride(SIGNER_OVERRIDE_V4);
    // Set the optional socket timeout, if configured.
    if (params.getSocketTimeout() != null) {
        clientConfiguration.setSocketTimeout(params.getSocketTimeout());
    }
    // Create an S3 client using passed in credentials and HTTP proxy information.
    if (StringUtils.isNotBlank(params.getAwsAccessKeyId()) && StringUtils.isNotBlank(params.getAwsSecretKey()) && StringUtils.isNotBlank(params.getSessionToken())) {
        // Create an S3 client using basic session credentials.
        amazonS3Client = new AmazonS3Client(new BasicSessionCredentials(params.getAwsAccessKeyId(), params.getAwsSecretKey(), params.getSessionToken()), clientConfiguration);
    } else {
        // Create an S3 client using AWS credentials provider.
        amazonS3Client = new AmazonS3Client(getAWSCredentialsProvider(params), clientConfiguration);
    }
    // Set the optional endpoint, if specified.
    if (StringUtils.isNotBlank(params.getS3Endpoint())) {
        LOGGER.info("Configured S3 Endpoint: " + params.getS3Endpoint());
        amazonS3Client.setEndpoint(params.getS3Endpoint());
    }
    // Return the newly created client.
    return amazonS3Client;
}
Also used : AmazonS3Client(com.amazonaws.services.s3.AmazonS3Client) BasicSessionCredentials(com.amazonaws.auth.BasicSessionCredentials) ClientConfiguration(com.amazonaws.ClientConfiguration)

Example 88 with ClientConfiguration

use of com.amazonaws.ClientConfiguration in project herd by FINRAOS.

the class EmrDaoTest method getEmrClientAssertClientConfigurationNotSetWhenProxyHostIsBlank.

@Test
public void getEmrClientAssertClientConfigurationNotSetWhenProxyHostIsBlank() throws Exception {
    String httpProxyHost = "";
    Integer httpProxyPort = 1234;
    AwsParamsDto awsParamsDto = new AwsParamsDto();
    awsParamsDto.setHttpProxyHost(httpProxyHost);
    awsParamsDto.setHttpProxyPort(httpProxyPort);
    AmazonElasticMapReduceClient amazonElasticMapReduceClient = emrDao.getEmrClient(awsParamsDto);
    ClientConfiguration clientConfiguration = (ClientConfiguration) ReflectionTestUtils.getField(amazonElasticMapReduceClient, "clientConfiguration");
    assertNotNull(clientConfiguration);
    assertNull(clientConfiguration.getProxyHost());
}
Also used : AwsParamsDto(org.finra.herd.model.dto.AwsParamsDto) AmazonElasticMapReduceClient(com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient) ClientConfiguration(com.amazonaws.ClientConfiguration) Test(org.junit.Test)

Example 89 with ClientConfiguration

use of com.amazonaws.ClientConfiguration in project herd by FINRAOS.

the class S3DaoTest method testGetAmazonS3AssertProxyIsSet.

@Test
public void testGetAmazonS3AssertProxyIsSet() {
    S3Operations originalS3Operations = (S3Operations) ReflectionTestUtils.getField(s3Dao, "s3Operations");
    S3Operations mockS3Operations = mock(S3Operations.class);
    ReflectionTestUtils.setField(s3Dao, "s3Operations", mockS3Operations);
    try {
        String s3BucketName = "s3BucketName";
        String s3KeyPrefix = "s3KeyPrefix";
        String httpProxyHost = "httpProxyHost";
        Integer httpProxyPort = 1234;
        S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto = new S3FileTransferRequestParamsDto();
        s3FileTransferRequestParamsDto.setS3BucketName(s3BucketName);
        s3FileTransferRequestParamsDto.setS3KeyPrefix(s3KeyPrefix);
        s3FileTransferRequestParamsDto.setHttpProxyHost(httpProxyHost);
        s3FileTransferRequestParamsDto.setHttpProxyPort(httpProxyPort);
        when(mockS3Operations.putObject(any(), any())).then(new Answer<PutObjectResult>() {

            @Override
            public PutObjectResult answer(InvocationOnMock invocation) throws Throwable {
                AmazonS3Client amazonS3Client = invocation.getArgument(1);
                ClientConfiguration clientConfiguration = (ClientConfiguration) ReflectionTestUtils.getField(amazonS3Client, "clientConfiguration");
                assertEquals(httpProxyHost, clientConfiguration.getProxyHost());
                assertEquals(httpProxyPort.intValue(), clientConfiguration.getProxyPort());
                return new PutObjectResult();
            }
        });
        s3Dao.createDirectory(s3FileTransferRequestParamsDto);
    } finally {
        ReflectionTestUtils.setField(s3Dao, "s3Operations", originalS3Operations);
    }
}
Also used : AmazonS3Client(com.amazonaws.services.s3.AmazonS3Client) S3FileTransferRequestParamsDto(org.finra.herd.model.dto.S3FileTransferRequestParamsDto) PutObjectResult(com.amazonaws.services.s3.model.PutObjectResult) InvocationOnMock(org.mockito.invocation.InvocationOnMock) ClientConfiguration(com.amazonaws.ClientConfiguration) Test(org.junit.Test)

Example 90 with ClientConfiguration

use of com.amazonaws.ClientConfiguration in project herd by FINRAOS.

the class S3DaoTest method testGetAmazonS3AssertProxyIsNotSetWhenProxyPortIsNull.

@Test
public void testGetAmazonS3AssertProxyIsNotSetWhenProxyPortIsNull() {
    S3Operations originalS3Operations = (S3Operations) ReflectionTestUtils.getField(s3Dao, "s3Operations");
    S3Operations mockS3Operations = mock(S3Operations.class);
    ReflectionTestUtils.setField(s3Dao, "s3Operations", mockS3Operations);
    try {
        String s3BucketName = "s3BucketName";
        String s3KeyPrefix = "s3KeyPrefix";
        String httpProxyHost = "httpProxyHost";
        Integer httpProxyPort = null;
        S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto = new S3FileTransferRequestParamsDto();
        s3FileTransferRequestParamsDto.setS3BucketName(s3BucketName);
        s3FileTransferRequestParamsDto.setS3KeyPrefix(s3KeyPrefix);
        s3FileTransferRequestParamsDto.setHttpProxyHost(httpProxyHost);
        s3FileTransferRequestParamsDto.setHttpProxyPort(httpProxyPort);
        when(mockS3Operations.putObject(any(), any())).then(new Answer<PutObjectResult>() {

            @Override
            public PutObjectResult answer(InvocationOnMock invocation) throws Throwable {
                AmazonS3Client amazonS3Client = invocation.getArgument(1);
                ClientConfiguration clientConfiguration = (ClientConfiguration) ReflectionTestUtils.getField(amazonS3Client, "clientConfiguration");
                assertNull(clientConfiguration.getProxyHost());
                return new PutObjectResult();
            }
        });
        s3Dao.createDirectory(s3FileTransferRequestParamsDto);
    } finally {
        ReflectionTestUtils.setField(s3Dao, "s3Operations", originalS3Operations);
    }
}
Also used : AmazonS3Client(com.amazonaws.services.s3.AmazonS3Client) S3FileTransferRequestParamsDto(org.finra.herd.model.dto.S3FileTransferRequestParamsDto) PutObjectResult(com.amazonaws.services.s3.model.PutObjectResult) InvocationOnMock(org.mockito.invocation.InvocationOnMock) ClientConfiguration(com.amazonaws.ClientConfiguration) Test(org.junit.Test)

Aggregations

ClientConfiguration (com.amazonaws.ClientConfiguration)134 Test (org.junit.Test)35 BasicAWSCredentials (com.amazonaws.auth.BasicAWSCredentials)29 AmazonS3Client (com.amazonaws.services.s3.AmazonS3Client)17 AWSCredentials (com.amazonaws.auth.AWSCredentials)14 AWSCredentialsProvider (com.amazonaws.auth.AWSCredentialsProvider)13 AWSStaticCredentialsProvider (com.amazonaws.auth.AWSStaticCredentialsProvider)13 AwsClientBuilder (com.amazonaws.client.builder.AwsClientBuilder)10 AwsParamsDto (org.finra.herd.model.dto.AwsParamsDto)8 ClientConfigurationFactory (com.amazonaws.ClientConfigurationFactory)7 EnvVars (hudson.EnvVars)7 File (java.io.File)7 AmazonS3ClientBuilder (com.amazonaws.services.s3.AmazonS3ClientBuilder)6 Configuration (org.apache.hadoop.conf.Configuration)6 AmazonClientException (com.amazonaws.AmazonClientException)5 DefaultAWSCredentialsProviderChain (com.amazonaws.auth.DefaultAWSCredentialsProviderChain)5 EndpointConfiguration (com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration)5 URI (java.net.URI)5 Properties (java.util.Properties)5 Test (org.testng.annotations.Test)5