Search in sources :

Example 1 with RemoveBackupRequest

use of com.instaclustr.esop.impl.remove.RemoveBackupRequest in project esop by instaclustr.

the class BaseS3Restorer method delete.

@Override
public void delete(final Manifest.ManifestReporter.ManifestReport backupToDelete, final RemoveBackupRequest request) throws Exception {
    logger.info("Deleting backup {}", backupToDelete.name);
    if (backupToDelete.reclaimableSpace > 0 && !backupToDelete.getRemovableEntries().isEmpty()) {
        // convert removable entries into S3 object keys
        String[] toRemove = backupToDelete.getRemovableEntries().stream().map(entry -> objectKeyToNodeAwareRemoteReference(Paths.get(entry)).canonicalPath).toArray(String[]::new);
        for (String remove : toRemove) {
            if (request.dry) {
                logger.info("Deletion of {} was executed in dry mode.", remove);
            } else {
                logger.info("Deleting file {}", remove);
            }
        }
        if (!request.dry) {
            amazonS3.deleteObjects(new DeleteObjectsRequest(request.storageLocation.bucket).withKeys(toRemove));
            logger.info("Deletion of files complete");
        }
    }
    Path key = backupToDelete.manifest.objectKey;
    // manifest and topology as the last
    if (!request.dry) {
        logger.info("Deleting file {} from S3", key);
        // delete in S3
        deleteNodeAwareKey(key);
        // delete in local cache
        logger.info("Deleting file {} from local cache", key);
        localFileRestorer.deleteNodeAwareKey(key);
    } else {
        logger.info("Deletion of {} was executed in dry mode.", key);
    }
}
Also used : TransferManager(com.amazonaws.services.s3.transfer.TransferManager) ListOperationRequest(com.instaclustr.esop.impl.list.ListOperationRequest) LoggerFactory(org.slf4j.LoggerFactory) ListOperationRequest.getForLocalListing(com.instaclustr.esop.impl.list.ListOperationRequest.getForLocalListing) GetObjectRequest(com.amazonaws.services.s3.model.GetObjectRequest) ObjectListing(com.amazonaws.services.s3.model.ObjectListing) ProgressEvent(com.amazonaws.event.ProgressEvent) DeleteObjectRequest(com.amazonaws.services.s3.model.DeleteObjectRequest) ArrayList(java.util.ArrayList) Collectors.toCollection(java.util.stream.Collectors.toCollection) RemoveBackupRequest(com.instaclustr.esop.impl.remove.RemoveBackupRequest) S3ProgressListener(com.amazonaws.services.s3.transfer.internal.S3ProgressListener) LocalFileRestorer(com.instaclustr.esop.local.LocalFileRestorer) S3Object(com.amazonaws.services.s3.model.S3Object) CharStreams(com.google.common.io.CharStreams) DeleteObjectsRequest(com.amazonaws.services.s3.model.DeleteObjectsRequest) RemoteObjectReference(com.instaclustr.esop.impl.RemoteObjectReference) AmazonS3(com.amazonaws.services.s3.AmazonS3) ProgressEventType(com.amazonaws.event.ProgressEventType) S3ObjectSummary(com.amazonaws.services.s3.model.S3ObjectSummary) Path(java.nio.file.Path) FileUtils(com.instaclustr.io.FileUtils) RestoreCommitLogsOperationRequest(com.instaclustr.esop.impl.restore.RestoreCommitLogsOperationRequest) Logger(org.slf4j.Logger) AmazonServiceException(com.amazonaws.AmazonServiceException) PersistableTransfer(com.amazonaws.services.s3.transfer.PersistableTransfer) StorageLocation(com.instaclustr.esop.impl.StorageLocation) Restorer(com.instaclustr.esop.impl.restore.Restorer) Files(java.nio.file.Files) Predicate(java.util.function.Predicate) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) RetriableException(com.instaclustr.esop.impl.retry.Retrier.RetriableException) IOException(java.io.IOException) Manifest(com.instaclustr.esop.impl.Manifest) InputStreamReader(java.io.InputStreamReader) Collectors(java.util.stream.Collectors) String.format(java.lang.String.format) RetrierFactory(com.instaclustr.esop.impl.retry.RetrierFactory) Consumer(java.util.function.Consumer) List(java.util.List) Paths(java.nio.file.Paths) RestoreOperationRequest(com.instaclustr.esop.impl.restore.RestoreOperationRequest) AmazonClientException(com.amazonaws.AmazonClientException) InputStream(java.io.InputStream) Path(java.nio.file.Path) DeleteObjectsRequest(com.amazonaws.services.s3.model.DeleteObjectsRequest)

Example 2 with RemoveBackupRequest

use of com.instaclustr.esop.impl.remove.RemoveBackupRequest in project esop by instaclustr.

the class RemoveBackupApplication method run.

@Override
public void run() {
    Esop.logCommandVersionInformation(spec);
    final List<Module> modules = Collections.singletonList(new RemoveBackupModule());
    Esop.init(this, jmxSpec, new HashSpec(), request, logger, modules);
    if (rate.value == 0) {
        final Operation<?> operation = operationsService.submitOperationRequest(request);
        await().forever().until(() -> operation.state.isTerminalState());
        if (operation.state == FAILED) {
            throw new IllegalStateException(format("List operation %s was not successful.", operation.id));
        }
    } else {
        final Supplier<RemoveBackupOperation> supplier = () -> new RemoveBackupOperation(request, cassandraJMXService, restorerFactoryMap, objectMapper);
        final DaemonScheduler<RemoveBackupRequest, RemoveBackupOperation> scheduler = new DaemonScheduler<>(rate, supplier);
        scheduler.setup();
        scheduler.execute();
    }
}
Also used : RemoveBackupModule(com.instaclustr.esop.impl.remove.RemoveBackupModule) HashSpec(com.instaclustr.esop.impl.hash.HashSpec) RemoveBackupOperation(com.instaclustr.esop.impl.remove.RemoveBackupOperation) DaemonScheduler(com.instaclustr.scheduling.DaemonScheduler) Module(com.google.inject.Module) RemoveBackupModule(com.instaclustr.esop.impl.remove.RemoveBackupModule) RemoveBackupRequest(com.instaclustr.esop.impl.remove.RemoveBackupRequest)

Aggregations

RemoveBackupRequest (com.instaclustr.esop.impl.remove.RemoveBackupRequest)2 AmazonClientException (com.amazonaws.AmazonClientException)1 AmazonServiceException (com.amazonaws.AmazonServiceException)1 ProgressEvent (com.amazonaws.event.ProgressEvent)1 ProgressEventType (com.amazonaws.event.ProgressEventType)1 AmazonS3 (com.amazonaws.services.s3.AmazonS3)1 DeleteObjectRequest (com.amazonaws.services.s3.model.DeleteObjectRequest)1 DeleteObjectsRequest (com.amazonaws.services.s3.model.DeleteObjectsRequest)1 GetObjectRequest (com.amazonaws.services.s3.model.GetObjectRequest)1 ObjectListing (com.amazonaws.services.s3.model.ObjectListing)1 S3Object (com.amazonaws.services.s3.model.S3Object)1 S3ObjectSummary (com.amazonaws.services.s3.model.S3ObjectSummary)1 PersistableTransfer (com.amazonaws.services.s3.transfer.PersistableTransfer)1 TransferManager (com.amazonaws.services.s3.transfer.TransferManager)1 S3ProgressListener (com.amazonaws.services.s3.transfer.internal.S3ProgressListener)1 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)1 CharStreams (com.google.common.io.CharStreams)1 Module (com.google.inject.Module)1 Manifest (com.instaclustr.esop.impl.Manifest)1 RemoteObjectReference (com.instaclustr.esop.impl.RemoteObjectReference)1