Search in sources :

Example 1 with ObjectIdentifier

use of software.amazon.awssdk.services.s3.model.ObjectIdentifier in project beam by apache.

the class S3FileSystem method delete.

private void delete(String bucket, Collection<String> keys) throws IOException {
    checkArgument(keys.size() <= MAX_DELETE_OBJECTS_PER_REQUEST, "only %s keys can be deleted per request, but got %s", MAX_DELETE_OBJECTS_PER_REQUEST, keys.size());
    List<ObjectIdentifier> deleteKeyVersions = keys.stream().map((key) -> ObjectIdentifier.builder().key(key).build()).collect(Collectors.toList());
    Delete delete = Delete.builder().objects(deleteKeyVersions).quiet(true).build();
    DeleteObjectsRequest deleteObjectsRequest = DeleteObjectsRequest.builder().bucket(bucket).delete(delete).build();
    try {
        s3Client.get().deleteObjects(deleteObjectsRequest);
    } catch (SdkServiceException e) {
        throw new IOException(e);
    }
}
Also used : CompletedMultipartUpload(software.amazon.awssdk.services.s3.model.CompletedMultipartUpload) Date(java.util.Date) LoggerFactory(org.slf4j.LoggerFactory) CreateMultipartUploadResponse(software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse) S3Exception(software.amazon.awssdk.services.s3.model.S3Exception) ImmutableSet(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet) Future(java.util.concurrent.Future) Supplier(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier) Strings(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings) Map(java.util.Map) Iterables(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables) ArrayListMultimap(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap) ObjectIdentifier(software.amazon.awssdk.services.s3.model.ObjectIdentifier) CompleteMultipartUploadResponse(software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse) CopyObjectResponse(software.amazon.awssdk.services.s3.model.CopyObjectResponse) Suppliers(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers) MoreExecutors(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors) Collection(java.util.Collection) CompleteMultipartUploadRequest(software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest) HeadObjectRequest(software.amazon.awssdk.services.s3.model.HeadObjectRequest) ListObjectsV2Request(software.amazon.awssdk.services.s3.model.ListObjectsV2Request) Collectors(java.util.stream.Collectors) S3Options(org.apache.beam.sdk.io.aws2.options.S3Options) FileNotFoundException(java.io.FileNotFoundException) Executors(java.util.concurrent.Executors) ThreadFactoryBuilder(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder) Objects(java.util.Objects) List(java.util.List) DeleteObjectsRequest(software.amazon.awssdk.services.s3.model.DeleteObjectsRequest) Stream(java.util.stream.Stream) MoveOptions(org.apache.beam.sdk.io.fs.MoveOptions) VisibleForTesting(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting) HeadObjectResponse(software.amazon.awssdk.services.s3.model.HeadObjectResponse) AutoValue(com.google.auto.value.AutoValue) ListeningExecutorService(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService) ImmutableList(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList) Pattern(java.util.regex.Pattern) SdkServiceException(software.amazon.awssdk.core.exception.SdkServiceException) Delete(software.amazon.awssdk.services.s3.model.Delete) FileSystem(org.apache.beam.sdk.io.FileSystem) Preconditions.checkNotNull(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull) MatchResult(org.apache.beam.sdk.io.fs.MatchResult) S3Object(software.amazon.awssdk.services.s3.model.S3Object) CopyObjectRequest(software.amazon.awssdk.services.s3.model.CopyObjectRequest) FileSystemUtils.wildcardToRegexp(org.apache.beam.sdk.io.FileSystemUtils.wildcardToRegexp) Callable(java.util.concurrent.Callable) UploadPartCopyRequest(software.amazon.awssdk.services.s3.model.UploadPartCopyRequest) ListObjectsV2Response(software.amazon.awssdk.services.s3.model.ListObjectsV2Response) ArrayList(java.util.ArrayList) Multimap(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap) Preconditions.checkArgument(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument) Nullable(org.checkerframework.checker.nullness.qual.Nullable) ReadableByteChannel(java.nio.channels.ReadableByteChannel) Logger(org.slf4j.Logger) Iterator(java.util.Iterator) CreateOptions(org.apache.beam.sdk.io.fs.CreateOptions) S3Client(software.amazon.awssdk.services.s3.S3Client) IOException(java.io.IOException) CopyPartResult(software.amazon.awssdk.services.s3.model.CopyPartResult) CompletedPart(software.amazon.awssdk.services.s3.model.CompletedPart) ExecutionException(java.util.concurrent.ExecutionException) Preconditions.checkState(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState) MoreFutures(org.apache.beam.sdk.util.MoreFutures) WritableByteChannel(java.nio.channels.WritableByteChannel) CreateMultipartUploadRequest(software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest) Delete(software.amazon.awssdk.services.s3.model.Delete) SdkServiceException(software.amazon.awssdk.core.exception.SdkServiceException) IOException(java.io.IOException) ObjectIdentifier(software.amazon.awssdk.services.s3.model.ObjectIdentifier) DeleteObjectsRequest(software.amazon.awssdk.services.s3.model.DeleteObjectsRequest)

Example 2 with ObjectIdentifier

use of software.amazon.awssdk.services.s3.model.ObjectIdentifier in project aws-doc-sdk-examples by awsdocs.

the class DeleteMultiObjects method deleteBucketObjects.

// snippet-start:[s3.java2.delete_many_objects.main]
public static void deleteBucketObjects(S3Client s3, String bucketName) {
    // Upload three sample objects to the specfied Amazon S3 bucket.
    ArrayList<ObjectIdentifier> keys = new ArrayList<>();
    PutObjectRequest putOb = null;
    ObjectIdentifier objectId = null;
    for (int i = 0; i < 3; i++) {
        String keyName = "delete object example " + i;
        objectId = ObjectIdentifier.builder().key(keyName).build();
        putOb = PutObjectRequest.builder().bucket(bucketName).key(keyName).build();
        s3.putObject(putOb, RequestBody.fromString(keyName));
        keys.add(objectId);
    }
    System.out.println(keys.size() + " objects successfully created.");
    // Delete multiple objects in one request.
    Delete del = Delete.builder().objects(keys).build();
    try {
        DeleteObjectsRequest multiObjectDeleteRequest = DeleteObjectsRequest.builder().bucket(bucketName).delete(del).build();
        s3.deleteObjects(multiObjectDeleteRequest);
        System.out.println("Multiple objects are deleted!");
    } catch (S3Exception e) {
        System.err.println(e.awsErrorDetails().errorMessage());
        System.exit(1);
    }
}
Also used : Delete(software.amazon.awssdk.services.s3.model.Delete) S3Exception(software.amazon.awssdk.services.s3.model.S3Exception) ArrayList(java.util.ArrayList) PutObjectRequest(software.amazon.awssdk.services.s3.model.PutObjectRequest) ObjectIdentifier(software.amazon.awssdk.services.s3.model.ObjectIdentifier) DeleteObjectsRequest(software.amazon.awssdk.services.s3.model.DeleteObjectsRequest)

Example 3 with ObjectIdentifier

use of software.amazon.awssdk.services.s3.model.ObjectIdentifier in project ddf by codice.

the class DeleteBean method delete.

public void delete(@Header(AWS2S3Constants.KEY) String s3Key) {
    LOGGER.trace("Deleting: {} / {}", s3Bucket, s3Key);
    ArrayList<ObjectIdentifier> toDelete = new ArrayList<>();
    toDelete.add(ObjectIdentifier.builder().key(s3Key).build());
    DeleteObjectsRequest deleteRequest = DeleteObjectsRequest.builder().bucket(s3Bucket).delete(Delete.builder().objects(toDelete).build()).build();
    s3Client.deleteObjects(deleteRequest);
}
Also used : ArrayList(java.util.ArrayList) ObjectIdentifier(software.amazon.awssdk.services.s3.model.ObjectIdentifier) DeleteObjectsRequest(software.amazon.awssdk.services.s3.model.DeleteObjectsRequest)

Example 4 with ObjectIdentifier

use of software.amazon.awssdk.services.s3.model.ObjectIdentifier in project aws-doc-sdk-examples by awsdocs.

the class DeleteObjects method deleteBucketObjects.

// snippet-start:[s3.java2.delete_objects.main]
public static void deleteBucketObjects(S3Client s3, String bucketName, String objectName) {
    ArrayList<ObjectIdentifier> toDelete = new ArrayList<ObjectIdentifier>();
    toDelete.add(ObjectIdentifier.builder().key(objectName).build());
    try {
        DeleteObjectsRequest dor = DeleteObjectsRequest.builder().bucket(bucketName).delete(Delete.builder().objects(toDelete).build()).build();
        s3.deleteObjects(dor);
    } catch (S3Exception e) {
        System.err.println(e.awsErrorDetails().errorMessage());
        System.exit(1);
    }
    System.out.println("Done!");
}
Also used : S3Exception(software.amazon.awssdk.services.s3.model.S3Exception) ArrayList(java.util.ArrayList) ObjectIdentifier(software.amazon.awssdk.services.s3.model.ObjectIdentifier) DeleteObjectsRequest(software.amazon.awssdk.services.s3.model.DeleteObjectsRequest)

Example 5 with ObjectIdentifier

use of software.amazon.awssdk.services.s3.model.ObjectIdentifier in project hazelcast by hazelcast.

the class S3SinkTest method deleteObjects.

@AfterClass
public static void deleteObjects() {
    S3Client client = client();
    List<ObjectIdentifier> identifiers = listObjects(client).contents().stream().map(S3Object::key).map(key -> ObjectIdentifier.builder().key(key).build()).collect(Collectors.toList());
    if (!identifiers.isEmpty()) {
        client.deleteObjects(b -> b.bucket(BUCKET).delete(d -> d.objects(identifiers)));
    }
    int sleepMillis = (int) (SECONDS.toMillis(WAIT_AFTER_CLEANUP_IN_SECS) / 10);
    long deadline = System.currentTimeMillis() + SECONDS.toMillis(WAIT_AFTER_CLEANUP_IN_SECS);
    int keyCount;
    while ((keyCount = listObjects(client).keyCount()) != 0 && System.currentTimeMillis() < deadline) {
        logger.info("After sending the object cleanup request to S3, the bucket, " + BUCKET + ", still has " + keyCount + " keys.");
        sleepMillis(sleepMillis);
    }
    if (keyCount == 0) {
        logger.info("We finished waiting because we observe that the keys were deleted.");
    } else {
        logger.warning("There may still be keys in the bucket that are not deleted.\n" + "At our last check, keyCount was " + keyCount + "\n We finished waiting because of the timeout.");
    }
}
Also used : Logger(com.hazelcast.logging.Logger) NightlyTest(com.hazelcast.test.annotation.NightlyTest) AfterClass(org.junit.AfterClass) S3Client(software.amazon.awssdk.services.s3.S3Client) S3Object(software.amazon.awssdk.services.s3.model.S3Object) RunWith(org.junit.runner.RunWith) Test(org.junit.Test) ListObjectsV2Request(software.amazon.awssdk.services.s3.model.ListObjectsV2Request) Category(org.junit.experimental.categories.Category) ListObjectsV2Response(software.amazon.awssdk.services.s3.model.ListObjectsV2Response) Collectors(java.util.stream.Collectors) SupplierEx(com.hazelcast.function.SupplierEx) HazelcastSerialClassRunner(com.hazelcast.test.HazelcastSerialClassRunner) List(java.util.List) ILogger(com.hazelcast.logging.ILogger) ObjectIdentifier(software.amazon.awssdk.services.s3.model.ObjectIdentifier) Region(software.amazon.awssdk.regions.Region) SECONDS(java.util.concurrent.TimeUnit.SECONDS) S3Object(software.amazon.awssdk.services.s3.model.S3Object) S3Client(software.amazon.awssdk.services.s3.S3Client) ObjectIdentifier(software.amazon.awssdk.services.s3.model.ObjectIdentifier) AfterClass(org.junit.AfterClass)

Aggregations

ObjectIdentifier (software.amazon.awssdk.services.s3.model.ObjectIdentifier)5 ArrayList (java.util.ArrayList)4 DeleteObjectsRequest (software.amazon.awssdk.services.s3.model.DeleteObjectsRequest)4 S3Exception (software.amazon.awssdk.services.s3.model.S3Exception)3 List (java.util.List)2 Collectors (java.util.stream.Collectors)2 Delete (software.amazon.awssdk.services.s3.model.Delete)2 AutoValue (com.google.auto.value.AutoValue)1 SupplierEx (com.hazelcast.function.SupplierEx)1 ILogger (com.hazelcast.logging.ILogger)1 Logger (com.hazelcast.logging.Logger)1 HazelcastSerialClassRunner (com.hazelcast.test.HazelcastSerialClassRunner)1 NightlyTest (com.hazelcast.test.annotation.NightlyTest)1 FileNotFoundException (java.io.FileNotFoundException)1 IOException (java.io.IOException)1 ReadableByteChannel (java.nio.channels.ReadableByteChannel)1 WritableByteChannel (java.nio.channels.WritableByteChannel)1 Collection (java.util.Collection)1 Date (java.util.Date)1 Iterator (java.util.Iterator)1