Search in sources :

Example 61 with S3ObjectSummary

use of com.amazonaws.services.s3.model.S3ObjectSummary in project apex-malhar by apache.

the class S3RecordReaderModuleAppTest method deleteBucketAndContent.

public void deleteBucketAndContent() {
    // Get the list of objects
    ObjectListing objectListing = client.listObjects(testMeta.bucketKey);
    for (Iterator<?> iterator = objectListing.getObjectSummaries().iterator(); iterator.hasNext(); ) {
        S3ObjectSummary objectSummary = (S3ObjectSummary) iterator.next();
        LOG.info("Deleting an object: {}", objectSummary.getKey());
        client.deleteObject(testMeta.bucketKey, objectSummary.getKey());
    }
    client.deleteBucket(testMeta.bucketKey);
}
Also used : ObjectListing(com.amazonaws.services.s3.model.ObjectListing) S3ObjectSummary(com.amazonaws.services.s3.model.S3ObjectSummary)

Example 62 with S3ObjectSummary

use of com.amazonaws.services.s3.model.S3ObjectSummary in project syndesis-qe by syndesisio.

the class S3Utils method deleteS3Bucket.

public void deleteS3Bucket(String bucketName) {
    try {
        final ObjectListing bucketObjects = s3client.listObjects(bucketName);
        for (Iterator<?> iterator = bucketObjects.getObjectSummaries().iterator(); iterator.hasNext(); ) {
            final S3ObjectSummary summary = (S3ObjectSummary) iterator.next();
            s3client.deleteObject(bucketName, summary.getKey());
        }
        s3client.deleteBucket(bucketName);
    } catch (AmazonServiceException e) {
        log.error("Could not delete the S3 bucket: {}", e.getErrorMessage());
    }
}
Also used : AmazonServiceException(com.amazonaws.AmazonServiceException) ObjectListing(com.amazonaws.services.s3.model.ObjectListing) S3ObjectSummary(com.amazonaws.services.s3.model.S3ObjectSummary)

Example 63 with S3ObjectSummary

use of com.amazonaws.services.s3.model.S3ObjectSummary in project athenz by yahoo.

the class S3ChangeLogStoreTest method testGetUpdatedSignedDomainsNoChanges.

@Test
public void testGetUpdatedSignedDomainsNoChanges() {
    MockS3ChangeLogStore store = new MockS3ChangeLogStore(null);
    ArrayList<S3ObjectSummary> objectList = new ArrayList<>();
    S3ObjectSummary objectSummary = new S3ObjectSummary();
    objectSummary.setKey("iaas");
    objectSummary.setLastModified(new Date(100));
    objectList.add(objectSummary);
    objectSummary = new S3ObjectSummary();
    objectSummary.setKey("iaas.athenz");
    objectSummary.setLastModified(new Date(200));
    objectList.add(objectSummary);
    ObjectListing objectListing = mock(ObjectListing.class);
    when(objectListing.getObjectSummaries()).thenReturn(objectList);
    when(objectListing.isTruncated()).thenReturn(false);
    when(store.awsS3Client.listObjects(any(ListObjectsRequest.class))).thenReturn(objectListing);
    // set the last modification time to not return any of the domains
    store.lastModTime = (new Date(250)).getTime();
    StringBuilder lastModTimeBuffer = new StringBuilder(512);
    SignedDomains signedDomains = store.getUpdatedSignedDomains(lastModTimeBuffer);
    assertTrue(lastModTimeBuffer.length() > 0);
    assertEquals(signedDomains.getDomains().size(), 0);
}
Also used : ListObjectsRequest(com.amazonaws.services.s3.model.ListObjectsRequest) ArrayList(java.util.ArrayList) ObjectListing(com.amazonaws.services.s3.model.ObjectListing) S3ObjectSummary(com.amazonaws.services.s3.model.S3ObjectSummary) SignedDomains(com.yahoo.athenz.zms.SignedDomains) Date(java.util.Date) Test(org.testng.annotations.Test)

Example 64 with S3ObjectSummary

use of com.amazonaws.services.s3.model.S3ObjectSummary in project athenz by yahoo.

the class S3ChangeLogStoreTest method testListObjectsAllObjectsNoPage.

@Test
public void testListObjectsAllObjectsNoPage() {
    MockS3ChangeLogStore store = new MockS3ChangeLogStore(null);
    ArrayList<S3ObjectSummary> objectList = new ArrayList<>();
    S3ObjectSummary objectSummary = new S3ObjectSummary();
    objectSummary.setKey("iaas");
    objectList.add(objectSummary);
    objectSummary = new S3ObjectSummary();
    objectSummary.setKey("iaas.athenz");
    objectList.add(objectSummary);
    ObjectListing objectListing = mock(ObjectListing.class);
    when(objectListing.getObjectSummaries()).thenReturn(objectList);
    when(objectListing.isTruncated()).thenReturn(false);
    when(store.awsS3Client.listObjects(any(ListObjectsRequest.class))).thenReturn(objectListing);
    ArrayList<String> domains = new ArrayList<>();
    store.listObjects(store.awsS3Client, domains, 0);
    assertEquals(domains.size(), 2);
    assertTrue(domains.contains("iaas"));
    assertTrue(domains.contains("iaas.athenz"));
}
Also used : ListObjectsRequest(com.amazonaws.services.s3.model.ListObjectsRequest) ArrayList(java.util.ArrayList) ObjectListing(com.amazonaws.services.s3.model.ObjectListing) S3ObjectSummary(com.amazonaws.services.s3.model.S3ObjectSummary) Test(org.testng.annotations.Test)

Example 65 with S3ObjectSummary

use of com.amazonaws.services.s3.model.S3ObjectSummary in project athenz by yahoo.

the class S3ChangeLogStoreTest method testGetLocalDomains.

@Test
public void testGetLocalDomains() {
    MockS3ChangeLogStore store = new MockS3ChangeLogStore(null);
    ArrayList<S3ObjectSummary> objectList = new ArrayList<>();
    S3ObjectSummary objectSummary = new S3ObjectSummary();
    objectSummary.setKey("iaas");
    objectList.add(objectSummary);
    objectSummary = new S3ObjectSummary();
    objectSummary.setKey("iaas.athenz");
    objectList.add(objectSummary);
    ObjectListing objectListing = mock(ObjectListing.class);
    when(objectListing.getObjectSummaries()).thenReturn(objectList);
    when(objectListing.isTruncated()).thenReturn(false);
    when(store.awsS3Client.listObjects(any(ListObjectsRequest.class))).thenReturn(objectListing);
    // verify that our last mod time is 0 before the call
    assertEquals(store.lastModTime, 0);
    // retrieve the list of domains
    List<String> domains = store.getLocalDomainList();
    assertEquals(domains.size(), 2);
    assertTrue(domains.contains("iaas"));
    assertTrue(domains.contains("iaas.athenz"));
    // also verify that last mod time is updated
    assertTrue(store.lastModTime > 0);
}
Also used : ListObjectsRequest(com.amazonaws.services.s3.model.ListObjectsRequest) ArrayList(java.util.ArrayList) ObjectListing(com.amazonaws.services.s3.model.ObjectListing) S3ObjectSummary(com.amazonaws.services.s3.model.S3ObjectSummary) Test(org.testng.annotations.Test)

Aggregations

S3ObjectSummary (com.amazonaws.services.s3.model.S3ObjectSummary)60 ObjectListing (com.amazonaws.services.s3.model.ObjectListing)51 ArrayList (java.util.ArrayList)28 ListObjectsRequest (com.amazonaws.services.s3.model.ListObjectsRequest)24 DeleteObjectsRequest (com.amazonaws.services.s3.model.DeleteObjectsRequest)13 AmazonS3 (com.amazonaws.services.s3.AmazonS3)11 Test (org.testng.annotations.Test)9 AmazonS3Client (com.amazonaws.services.s3.AmazonS3Client)8 AmazonClientException (com.amazonaws.AmazonClientException)7 Path (org.apache.hadoop.fs.Path)7 AmazonServiceException (com.amazonaws.AmazonServiceException)6 DeleteObjectsResult (com.amazonaws.services.s3.model.DeleteObjectsResult)6 Properties (java.util.Properties)6 S3Object (com.amazonaws.services.s3.model.S3Object)4 IOException (java.io.IOException)4 HashMap (java.util.HashMap)4 Test (org.junit.Test)4 TransferManager (com.amazonaws.services.s3.transfer.TransferManager)3 Date (java.util.Date)3 HashSet (java.util.HashSet)3