Examples with Manifest org.elasticsearch.cluster.metadata.Manifest used on opensource projects

Example 6 with Manifest

use of org.elasticsearch.cluster.metadata.Manifest in project crate by crate.

the class MetaStateService method loadFullState.

/**
 * Loads the full state, which includes both the global state and all the indices meta data. <br>
 * When loading, manifest file is consulted (represented by {@link Manifest} class), to load proper generations. <br>
 * If there is no manifest file on disk, this method fallbacks to BWC mode, where latest generation of global and indices
 * metadata is loaded. Please note that currently there is no way to distinguish between manifest file being removed and manifest
 * file was not yet created. It means that this method always fallbacks to BWC mode, if there is no manifest file.
 *
 * @return tuple of {@link Manifest} and {@link Metadata} with global metadata and indices metadata. If there is no state on disk,
 * meta state with globalGeneration -1 and empty meta data is returned.
 * @throws IOException if some IOException when loading files occurs or there is no metadata referenced by manifest file.
 */
public Tuple<Manifest, Metadata> loadFullState() throws IOException {
    final Manifest manifest = MANIFEST_FORMAT.loadLatestState(LOGGER, namedXContentRegistry, nodeEnv.nodeDataPaths());
    if (manifest == null) {
        return loadFullStateBWC();
    }
    final Metadata.Builder metadataBuilder;
    if (manifest.isGlobalGenerationMissing()) {
        metadataBuilder = Metadata.builder();
    } else {
        final Metadata globalMetadata = META_DATA_FORMAT.loadGeneration(LOGGER, namedXContentRegistry, manifest.getGlobalGeneration(), nodeEnv.nodeDataPaths());
        if (globalMetadata != null) {
            metadataBuilder = Metadata.builder(globalMetadata);
        } else {
            throw new IOException("failed to find global metadata [generation: " + manifest.getGlobalGeneration() + "]");
        }
    }
    for (Map.Entry<Index, Long> entry : manifest.getIndexGenerations().entrySet()) {
        final Index index = entry.getKey();
        final long generation = entry.getValue();
        final String indexFolderName = index.getUUID();
        final IndexMetadata indexMetadata = INDEX_META_DATA_FORMAT.loadGeneration(LOGGER, namedXContentRegistry, generation, nodeEnv.resolveIndexFolder(indexFolderName));
        if (indexMetadata != null) {
            metadataBuilder.put(indexMetadata, false);
        } else {
            throw new IOException("failed to find metadata for existing index " + index.getName() + " [location: " + indexFolderName + ", generation: " + generation + "]");
        }
    }
    return new Tuple<>(manifest, metadataBuilder.build());
}

Example 7 with Manifest

use of org.elasticsearch.cluster.metadata.Manifest in project crate by crate.

the class IncrementalClusterStateWriterTests method testAtomicityWithFailures.

public void testAtomicityWithFailures() throws IOException {
    try (NodeEnvironment env = newNodeEnvironment()) {
        MetaStateServiceWithFailures metaStateService = new MetaStateServiceWithFailures(randomIntBetween(100, 1000), env, xContentRegistry());
        // We only guarantee atomicity of writes, if there is initial Manifest file
        Manifest manifest = Manifest.empty();
        Metadata metadata = Metadata.EMPTY_METADATA;
        metaStateService.writeManifestAndCleanup("startup", Manifest.empty());
        long currentTerm = randomNonNegativeLong();
        long clusterStateVersion = randomNonNegativeLong();
        metaStateService.failRandomly();
        Set<Metadata> possibleMetadata = new HashSet<>();
        possibleMetadata.add(metadata);
        for (int i = 0; i < randomIntBetween(1, 5); i++) {
            IncrementalClusterStateWriter.AtomicClusterStateWriter writer = new IncrementalClusterStateWriter.AtomicClusterStateWriter(metaStateService, manifest);
            metadata = randomMetadataForTx();
            Map<Index, Long> indexGenerations = new HashMap<>();
            try {
                long globalGeneration = writer.writeGlobalState("global", metadata);
                for (IndexMetadata indexMetadata : metadata) {
                    long generation = writer.writeIndex("index", indexMetadata);
                    indexGenerations.put(indexMetadata.getIndex(), generation);
                }
                Manifest newManifest = new Manifest(currentTerm, clusterStateVersion, globalGeneration, indexGenerations);
                writer.writeManifestAndCleanup("manifest", newManifest);
                possibleMetadata.clear();
                possibleMetadata.add(metadata);
                manifest = newManifest;
            } catch (WriteStateException e) {
                if (e.isDirty()) {
                    possibleMetadata.add(metadata);
                    /*
                         * If dirty WriteStateException occurred, it's only safe to proceed if there is subsequent
                         * successful write of metadata and Manifest. We prefer to break here, not to over complicate test logic.
                         * See also MetadataStateFormat#testFailRandomlyAndReadAnyState, that does not break.
                         */
                    break;
                }
            }
        }
        metaStateService.noFailures();
        Tuple<Manifest, Metadata> manifestAndMetadata = metaStateService.loadFullState();
        Metadata loadedMetadata = manifestAndMetadata.v2();
        assertTrue(possibleMetadata.stream().anyMatch(md -> metadataEquals(md, loadedMetadata)));
    }
}