Search in sources :

Example 6 with BatchRequest

use of com.google.api.client.googleapis.batch.BatchRequest in project beam by apache.

the class GcsUtil method makeRemoveBatches.

List<BatchRequest> makeRemoveBatches(Collection<String> filenames) throws IOException {
    List<BatchRequest> batches = new LinkedList<>();
    for (List<String> filesToDelete : Lists.partition(Lists.newArrayList(filenames), MAX_REQUESTS_PER_BATCH)) {
        BatchRequest batch = createBatchRequest();
        for (String file : filesToDelete) {
            enqueueDelete(GcsPath.fromUri(file), batch);
        }
        batches.add(batch);
    }
    return batches;
}
Also used : BatchRequest(com.google.api.client.googleapis.batch.BatchRequest) LinkedList(java.util.LinkedList)

Aggregations

BatchRequest (com.google.api.client.googleapis.batch.BatchRequest)6 LinkedList (java.util.LinkedList)4 IOException (java.io.IOException)2 GcsPath (org.apache.beam.sdk.util.gcsfs.GcsPath)2 GoogleJsonError (com.google.api.client.googleapis.json.GoogleJsonError)1 HttpHeaders (com.google.api.client.http.HttpHeaders)1 Storage (com.google.api.services.storage.Storage)1 Objects (com.google.api.services.storage.model.Objects)1 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 ListenableFuture (com.google.common.util.concurrent.ListenableFuture)1 ListeningExecutorService (com.google.common.util.concurrent.ListeningExecutorService)1 FileNotFoundException (java.io.FileNotFoundException)1 ArrayList (java.util.ArrayList)1 Callable (java.util.concurrent.Callable)1 ExecutionException (java.util.concurrent.ExecutionException)1 ThreadPoolExecutor (java.util.concurrent.ThreadPoolExecutor)1 CountDown (org.elasticsearch.common.util.concurrent.CountDown)1