Search in sources :

Example 1 with ManifestEntry

use of com.instaclustr.esop.impl.ManifestEntry in project esop by instaclustr.

the class LocalFileRestorer method listManifests.

@Override
public List<Manifest> listManifests() throws Exception {
    assert objectMapper != null;
    final Path path = Paths.get(storageLocation.rawLocation.replaceAll("file://", ""), "manifests");
    if (!Files.exists(path))
        return Collections.emptyList();
    final List<Path> manifests = Files.list(path).sorted(new ManifestAgePathComparator()).collect(toList());
    final List<Manifest> manifestsList = new ArrayList<>();
    for (final Path manifest : manifests) {
        final Manifest read = Manifest.read(manifest, objectMapper);
        read.setManifest(new ManifestEntry(Paths.get("manifests", manifest.getFileName().toString()), manifest, Type.FILE, null));
        manifestsList.add(read);
    }
    return manifestsList;
}
Also used : Path(java.nio.file.Path) ManifestAgePathComparator(com.instaclustr.esop.impl.Manifest.ManifestAgePathComparator) ManifestEntry(com.instaclustr.esop.impl.ManifestEntry) ArrayList(java.util.ArrayList) Manifest(com.instaclustr.esop.impl.Manifest)

Example 2 with ManifestEntry

use of com.instaclustr.esop.impl.ManifestEntry in project esop by instaclustr.

the class BaseBackupOperationCoordinator method coordinate.

@Override
public void coordinate(final Operation<BackupOperationRequest> operation) {
    final BackupOperationRequest request = operation.request;
    logger.info(request.toString());
    try {
        assert cassandraJMXService != null;
        assert backuperFactoryMap != null;
        assert bucketServiceFactoryMap != null;
        assert objectMapper != null;
        if (!request.skipBucketVerification) {
            try (final BucketService bucketService = bucketServiceFactoryMap.get(request.storageLocation.storageProvider).createBucketService(request)) {
                bucketService.checkBucket(request.storageLocation.bucket, request.createMissingBucket);
            }
        }
        final CassandraData cassandraData = CassandraData.parse(request.dataDirs.get(0));
        cassandraData.setDatabaseEntitiesFromRequest(request.entities);
        final List<String> tokens = new CassandraTokens(cassandraJMXService).act();
        logger.info("Tokens {}", tokens);
        if (!Snapshots.snapshotContainsTimestamp(operation.request.snapshotTag)) {
            if (operation.request.schemaVersion == null) {
                operation.request.schemaVersion = new CassandraSchemaVersion(cassandraJMXService).act();
            }
            operation.request.snapshotTag = resolveSnapshotTag(operation.request, System.currentTimeMillis());
        }
        logger.info("Taking snapshot with name {}", request.snapshotTag);
        new TakeSnapshotOperation(cassandraJMXService, new TakeSnapshotOperationRequest(request.entities, request.snapshotTag), cassandraVersionProvider).run0();
        Snapshots.hashSpec = hashSpec;
        final Snapshots snapshots = Snapshots.parse(request.dataDirs, request.snapshotTag);
        final Optional<Snapshot> snapshot = snapshots.get(request.snapshotTag);
        if (!snapshot.isPresent()) {
            throw new IllegalStateException(format("There is not any snapshot of tag %s", request.snapshotTag));
        }
        final Manifest manifest = Manifest.from(snapshot.get());
        manifest.setSchemaVersion(request.schemaVersion);
        manifest.setTokens(tokens);
        // manifest
        final Path localManifestPath = getLocalManifestPath(request.snapshotTag);
        Manifest.write(manifest, localManifestPath, objectMapper);
        manifest.setManifest(getManifestAsManifestEntry(localManifestPath));
        try (final Backuper backuper = backuperFactoryMap.get(request.storageLocation.storageProvider).createBackuper(request)) {
            final List<ManifestEntry> manifestEntries = manifest.getManifestEntries();
            Session<UploadUnit> uploadSession = null;
            try {
                uploadSession = uploadTracker.submit(backuper, operation, manifestEntries, request.snapshotTag, operation.request.concurrentConnections);
                uploadSession.waitUntilConsideredFinished();
                uploadTracker.cancelIfNecessary(uploadSession);
                final List<UploadUnit> failedUnits = uploadSession.getFailedUnits();
                if (!failedUnits.isEmpty()) {
                    final String message = failedUnits.stream().map(unit -> unit.getManifestEntry().objectKey.toString()).collect(Collectors.joining(","));
                    logger.error(message);
                    throw new IOException(format("Unable to upload some files successfully: %s", message));
                }
            } finally {
                uploadTracker.removeSession(uploadSession);
                uploadSession = null;
            }
            if (operation.request.uploadClusterTopology) {
                // here we will upload all topology because we do not know what restore might look like (what dc a restorer will restore against if any)
                final ClusterTopology topology = new CassandraClusterTopology(cassandraJMXService, null).act();
                ClusterTopology.upload(backuper, topology, objectMapper, operation.request.snapshotTag);
            }
        } finally {
            manifest.cleanup();
        }
    } catch (final Exception ex) {
        operation.addError(Error.from(ex));
    } finally {
        final ClearSnapshotOperation cso = new ClearSnapshotOperation(cassandraJMXService, new ClearSnapshotOperationRequest(request.snapshotTag));
        try {
            cso.run0();
        } catch (final Exception ex) {
            operation.addErrors(cso.errors);
        }
    }
}
Also used : CassandraData(com.instaclustr.esop.impl.CassandraData) ManifestEntry(com.instaclustr.esop.impl.ManifestEntry) Error(com.instaclustr.operations.Operation.Error) Manifest.getManifestAsManifestEntry(com.instaclustr.esop.impl.Manifest.getManifestAsManifestEntry) Provider(javax.inject.Provider) HashSpec(com.instaclustr.esop.impl.hash.HashSpec) OperationCoordinator(com.instaclustr.operations.OperationCoordinator) LoggerFactory(org.slf4j.LoggerFactory) Manifest.getLocalManifestPath(com.instaclustr.esop.impl.Manifest.getLocalManifestPath) BackuperFactory(com.instaclustr.esop.guice.BackuperFactory) Map(java.util.Map) Session(com.instaclustr.esop.impl.AbstractTracker.Session) CassandraClusterTopology(com.instaclustr.esop.topology.CassandraClusterTopology) Path(java.nio.file.Path) CassandraData(com.instaclustr.esop.impl.CassandraData) BucketServiceFactory(com.instaclustr.esop.guice.BucketServiceFactory) BucketService(com.instaclustr.esop.impl.BucketService) Backuper(com.instaclustr.esop.impl.backup.Backuper) CassandraSchemaVersion(com.instaclustr.esop.impl.interaction.CassandraSchemaVersion) Logger(org.slf4j.Logger) UploadTracker(com.instaclustr.esop.impl.backup.UploadTracker) Operation(com.instaclustr.operations.Operation) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) IOException(java.io.IOException) Manifest(com.instaclustr.esop.impl.Manifest) Snapshot(com.instaclustr.esop.impl.Snapshots.Snapshot) Collectors(java.util.stream.Collectors) String.format(java.lang.String.format) ClearSnapshotOperationRequest(com.instaclustr.esop.impl.backup.coordination.ClearSnapshotOperation.ClearSnapshotOperationRequest) List(java.util.List) Snapshots(com.instaclustr.esop.impl.Snapshots) UploadUnit(com.instaclustr.esop.impl.backup.UploadTracker.UploadUnit) TakeSnapshotOperationRequest(com.instaclustr.esop.impl.backup.coordination.TakeSnapshotOperation.TakeSnapshotOperationRequest) CassandraVersion(com.instaclustr.cassandra.CassandraVersion) Optional(java.util.Optional) BackupOperationRequest(com.instaclustr.esop.impl.backup.BackupOperationRequest) CassandraTokens(com.instaclustr.esop.impl.interaction.CassandraTokens) ClusterTopology(com.instaclustr.esop.topology.CassandraClusterTopology.ClusterTopology) CassandraJMXService(jmx.org.apache.cassandra.service.CassandraJMXService) ClearSnapshotOperationRequest(com.instaclustr.esop.impl.backup.coordination.ClearSnapshotOperation.ClearSnapshotOperationRequest) TakeSnapshotOperationRequest(com.instaclustr.esop.impl.backup.coordination.TakeSnapshotOperation.TakeSnapshotOperationRequest) CassandraClusterTopology(com.instaclustr.esop.topology.CassandraClusterTopology) ClusterTopology(com.instaclustr.esop.topology.CassandraClusterTopology.ClusterTopology) CassandraSchemaVersion(com.instaclustr.esop.impl.interaction.CassandraSchemaVersion) UploadUnit(com.instaclustr.esop.impl.backup.UploadTracker.UploadUnit) CassandraClusterTopology(com.instaclustr.esop.topology.CassandraClusterTopology) CassandraTokens(com.instaclustr.esop.impl.interaction.CassandraTokens) Backuper(com.instaclustr.esop.impl.backup.Backuper) Snapshots(com.instaclustr.esop.impl.Snapshots) Manifest.getLocalManifestPath(com.instaclustr.esop.impl.Manifest.getLocalManifestPath) Path(java.nio.file.Path) BucketService(com.instaclustr.esop.impl.BucketService) IOException(java.io.IOException) Manifest(com.instaclustr.esop.impl.Manifest) IOException(java.io.IOException) Snapshot(com.instaclustr.esop.impl.Snapshots.Snapshot) BackupOperationRequest(com.instaclustr.esop.impl.backup.BackupOperationRequest) ManifestEntry(com.instaclustr.esop.impl.ManifestEntry) Manifest.getManifestAsManifestEntry(com.instaclustr.esop.impl.Manifest.getManifestAsManifestEntry)

Example 3 with ManifestEntry

use of com.instaclustr.esop.impl.ManifestEntry in project esop by instaclustr.

the class RestoreCommitLogsOperation method downloadCommitLogs.

private void downloadCommitLogs(final Restorer restorer) throws Exception {
    final RemoteObjectReference remoteObjectReference = restorer.objectKeyToNodeAwareRemoteReference(Paths.get("commitlog"));
    final Pattern commitlogPattern = Pattern.compile(".*(CommitLog-\\d+-\\d+\\.log)\\.(\\d+)");
    final Set<ManifestEntry> parsedCommitlogList = new HashSet<>();
    logger.info("Commencing processing of commit log listing");
    final AtomicReference<ManifestEntry> overhangingManifestEntry = new AtomicReference<>();
    final AtomicLong overhangingTimestamp = new AtomicLong(Long.MAX_VALUE);
    restorer.consumeFiles(remoteObjectReference, commitlogFile -> {
        final Matcher matcherCommitlog = commitlogPattern.matcher(commitlogFile.getObjectKey().toString());
        if (matcherCommitlog.matches()) {
            final long commitlogTimestamp = Long.parseLong(matcherCommitlog.group(2));
            if (commitlogTimestamp >= request.timestampStart && commitlogTimestamp <= request.timestampEnd) {
                parsedCommitlogList.add(new ManifestEntry(commitlogFile.getObjectKey(), request.commitlogDownloadDir.resolve(matcherCommitlog.group(1)), COMMIT_LOG, 0, null, null));
            } else if (commitlogTimestamp > request.timestampEnd && commitlogTimestamp < overhangingTimestamp.get()) {
                // Make sure we also catch the first commitlog that goes past the end of the timestamp
                overhangingTimestamp.set(commitlogTimestamp);
                overhangingManifestEntry.set(new ManifestEntry(commitlogFile.getObjectKey(), request.commitlogDownloadDir.resolve(matcherCommitlog.group(1)), COMMIT_LOG, 0, null, null));
            }
        }
    });
    if (overhangingManifestEntry.get() != null) {
        parsedCommitlogList.add(overhangingManifestEntry.get());
    }
    logger.info("Found {} commit logs to download", parsedCommitlogList.size());
    if (parsedCommitlogList.size() == 0) {
        return;
    }
    Session<DownloadUnit> downloadSession = null;
    try {
        downloadSession = downloadTracker.submit(restorer, this, parsedCommitlogList, null, this.request.concurrentConnections);
        downloadSession.waitUntilConsideredFinished();
        downloadTracker.cancelIfNecessary(downloadSession);
    } finally {
        downloadTracker.removeSession(downloadSession);
    }
}
Also used : Pattern(java.util.regex.Pattern) AtomicLong(java.util.concurrent.atomic.AtomicLong) RemoteObjectReference(com.instaclustr.esop.impl.RemoteObjectReference) ManifestEntry(com.instaclustr.esop.impl.ManifestEntry) Matcher(java.util.regex.Matcher) AtomicReference(java.util.concurrent.atomic.AtomicReference) DownloadUnit(com.instaclustr.esop.impl.restore.DownloadTracker.DownloadUnit) HashSet(java.util.HashSet)

Example 4 with ManifestEntry

use of com.instaclustr.esop.impl.ManifestEntry in project esop by instaclustr.

the class ManifestComponentsTest method testManifestFilterNoRestoreSystemKeyspaceYesNewCluster.

@Test
public void testManifestFilterNoRestoreSystemKeyspaceYesNewCluster() throws Exception {
    Manifest manifest = parseManifest();
    manifest.enrichManifestEntries();
    List<ManifestEntry> manifestFiles = manifest.getManifestFiles(new DatabaseEntities(), // restoreSystemKeyspace
    false, // restoreSystemAuth
    false, // newCluster
    true, // withSchemas
    false);
    Assert.assertTrue(manifestFiles.stream().allMatch(entry -> {
        if (entry.keyspaceTable.keyspace.equals("system")) {
            return entry.keyspaceTable.table.startsWith("schema_");
        } else if (entry.keyspaceTable.keyspace.equals("system_schema")) {
            return true;
        } else
            return !KeyspaceTable.isSystemKeyspace(entry.keyspaceTable.keyspace);
    }));
}
Also used : DatabaseEntities(com.instaclustr.esop.impl.DatabaseEntities) ManifestEntry(com.instaclustr.esop.impl.ManifestEntry) Module(com.google.inject.Module) Inject(com.google.inject.Inject) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) BeforeMethod(org.testng.annotations.BeforeMethod) Keyspace(com.instaclustr.esop.impl.Snapshots.Snapshot.Keyspace) Type(com.instaclustr.esop.impl.ManifestEntry.Type) JacksonModule(com.instaclustr.jackson.JacksonModule) Test(org.testng.annotations.Test) HashMap(java.util.HashMap) Manifest(com.instaclustr.esop.impl.Manifest) Snapshot(com.instaclustr.esop.impl.Snapshots.Snapshot) ArrayList(java.util.ArrayList) Injector(com.google.inject.Injector) DatabaseEntities(com.instaclustr.esop.impl.DatabaseEntities) List(java.util.List) Assert(org.testng.Assert) Paths(java.nio.file.Paths) Table(com.instaclustr.esop.impl.Snapshots.Snapshot.Keyspace.Table) Guice(com.google.inject.Guice) KeyspaceTable(com.instaclustr.esop.impl.KeyspaceTable) Path(java.nio.file.Path) ManifestEntry(com.instaclustr.esop.impl.ManifestEntry) Manifest(com.instaclustr.esop.impl.Manifest) Test(org.testng.annotations.Test)

Example 5 with ManifestEntry

use of com.instaclustr.esop.impl.ManifestEntry in project esop by instaclustr.

the class ManifestComponentsTest method testManifestFilterYesRestoreSystemKeyspaceYesNewCluster.

@Test
public void testManifestFilterYesRestoreSystemKeyspaceYesNewCluster() throws Exception {
    Manifest manifest = parseManifest();
    manifest.enrichManifestEntries();
    List<ManifestEntry> manifestFiles = manifest.getManifestFiles(new DatabaseEntities(), // restoreSystemKeyspace
    true, // restoreSystemAuth
    false, // newCluster
    true, // withSchemas
    false);
    Assert.assertEquals(manifestFiles.size(), 304);
}
Also used : DatabaseEntities(com.instaclustr.esop.impl.DatabaseEntities) ManifestEntry(com.instaclustr.esop.impl.ManifestEntry) Manifest(com.instaclustr.esop.impl.Manifest) Test(org.testng.annotations.Test)

Aggregations

ManifestEntry (com.instaclustr.esop.impl.ManifestEntry)16 Manifest (com.instaclustr.esop.impl.Manifest)11 Path (java.nio.file.Path)11 Test (org.testng.annotations.Test)11 Snapshot (com.instaclustr.esop.impl.Snapshots.Snapshot)8 DatabaseEntities (com.instaclustr.esop.impl.DatabaseEntities)7 KeyspaceTable (com.instaclustr.esop.impl.KeyspaceTable)7 List (java.util.List)7 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)6 Keyspace (com.instaclustr.esop.impl.Snapshots.Snapshot.Keyspace)6 ArrayList (java.util.ArrayList)6 Table (com.instaclustr.esop.impl.Snapshots.Snapshot.Keyspace.Table)5 HashMap (java.util.HashMap)5 Guice (com.google.inject.Guice)4 Inject (com.google.inject.Inject)4 Injector (com.google.inject.Injector)4 Module (com.google.inject.Module)4 Type (com.instaclustr.esop.impl.ManifestEntry.Type)4 Snapshots (com.instaclustr.esop.impl.Snapshots)4 JacksonModule (com.instaclustr.jackson.JacksonModule)4