Examples with IndexCommit org.apache.lucene.index.IndexCommit used on opensource projects

Example 51 with IndexCommit

use of org.apache.lucene.index.IndexCommit in project neo4j by neo4j.

the class LuceneIndexSnapshots method forIndex.

/**
 * Create index snapshot iterator for a read only index.
 * @param indexFolder index location folder
 * @param directory index directory
 * @return index file name resource iterator
 * @throws IOException
 */
public static ResourceIterator<Path> forIndex(Path indexFolder, Directory directory) throws IOException {
    if (!hasCommits(directory)) {
        return emptyResourceIterator();
    }
    Collection<IndexCommit> indexCommits = DirectoryReader.listCommits(directory);
    IndexCommit indexCommit = Iterables.last(indexCommits);
    return new ReadOnlyIndexSnapshotFileIterator(indexFolder, indexCommit);
}

Example 52 with IndexCommit

use of org.apache.lucene.index.IndexCommit in project crate by crate.

the class CombinedDeletionPolicy method releaseCommit.

/**
 * Releases an index commit that acquired by {@link #acquireIndexCommit(boolean)}.
 *
 * @return true if the snapshotting commit can be clean up.
 */
synchronized boolean releaseCommit(final IndexCommit snapshotCommit) {
    final IndexCommit releasingCommit = ((SnapshotIndexCommit) snapshotCommit).delegate;
    assert snapshottedCommits.containsKey(releasingCommit) : "Release non-snapshotted commit;" + "snapshotted commits [" + snapshottedCommits + "], releasing commit [" + releasingCommit + "]";
    // release refCount
    final int refCount = snapshottedCommits.addTo(releasingCommit, -1);
    assert refCount >= 0 : "Number of snapshots can not be negative [" + refCount + "]";
    if (refCount == 0) {
        snapshottedCommits.remove(releasingCommit);
    }
    // The commit can be clean up only if no pending snapshot and it is neither the safe commit nor last commit.
    return refCount == 0 && releasingCommit.equals(safeCommit) == false && releasingCommit.equals(lastCommit) == false;
}

Example 53 with IndexCommit

use of org.apache.lucene.index.IndexCommit in project crate by crate.

the class Store method trimUnsafeCommits.

/**
 * Keeping existing unsafe commits when opening an engine can be problematic because these commits are not safe
 * at the recovering time but they can suddenly become safe in the future.
 * The following issues can happen if unsafe commits are kept oninit.
 * <p>
 * 1. Replica can use unsafe commit in peer-recovery. This happens when a replica with a safe commit c1(max_seqno=1)
 * and an unsafe commit c2(max_seqno=2) recovers from a primary with c1(max_seqno=1). If a new document(seqno=2)
 * is added without flushing, the global checkpoint is advanced to 2; and the replica recovers again, it will use
 * the unsafe commit c2(max_seqno=2 at most gcp=2) as the starting commit for sequenced-based recovery even the
 * commit c2 contains a stale operation and the document(with seqno=2) will not be replicated to the replica.
 * <p>
 * 2. Min translog gen for recovery can go backwards in peer-recovery. This happens when are replica with a safe commit
 * c1(local_checkpoint=1, recovery_translog_gen=1) and an unsafe commit c2(local_checkpoint=2, recovery_translog_gen=2).
 * The replica recovers from a primary, and keeps c2 as the last commit, then sets last_translog_gen to 2. Flushing a new
 * commit on the replica will cause exception as the new last commit c3 will have recovery_translog_gen=1. The recovery
 * translog generation of a commit is calculated based on the current local checkpoint. The local checkpoint of c3 is 1
 * while the local checkpoint of c2 is 2.
 * <p>
 * 3. Commit without translog can be used in recovery. An old index, which was created before multiple-commits is introduced
 * (v6.2), may not have a safe commit. If that index has a snapshotted commit without translog and an unsafe commit,
 * the policy can consider the snapshotted commit as a safe commit for recovery even the commit does not have translog.
 */
public void trimUnsafeCommits(final long lastSyncedGlobalCheckpoint, final long minRetainedTranslogGen, final org.elasticsearch.Version indexVersionCreated) throws IOException {
    metadataLock.writeLock().lock();
    try {
        final List<IndexCommit> existingCommits = DirectoryReader.listCommits(directory);
        if (existingCommits.isEmpty()) {
            throw new IllegalArgumentException("No index found to trim");
        }
        final IndexCommit lastIndexCommitCommit = existingCommits.get(existingCommits.size() - 1);
        final String translogUUID = lastIndexCommitCommit.getUserData().get(Translog.TRANSLOG_UUID_KEY);
        final IndexCommit startingIndexCommit;
        // To avoid this issue, we only select index commits whose translog are fully retained.
        if (indexVersionCreated.before(org.elasticsearch.Version.V_3_2_0)) {
            final List<IndexCommit> recoverableCommits = new ArrayList<>();
            for (IndexCommit commit : existingCommits) {
                final String translogGeneration = commit.getUserData().get("translog_generation");
                if (translogGeneration == null || minRetainedTranslogGen <= Long.parseLong(translogGeneration)) {
                    recoverableCommits.add(commit);
                }
            }
            assert recoverableCommits.isEmpty() == false : "No commit point with translog found; " + "commits [" + existingCommits + "], minRetainedTranslogGen [" + minRetainedTranslogGen + "]";
            startingIndexCommit = CombinedDeletionPolicy.findSafeCommitPoint(recoverableCommits, lastSyncedGlobalCheckpoint);
        } else {
            // TODO: Asserts the starting commit is a safe commit once peer-recovery sets global checkpoint.
            startingIndexCommit = CombinedDeletionPolicy.findSafeCommitPoint(existingCommits, lastSyncedGlobalCheckpoint);
        }
        if (translogUUID.equals(startingIndexCommit.getUserData().get(Translog.TRANSLOG_UUID_KEY)) == false) {
            throw new IllegalStateException("starting commit translog uuid [" + startingIndexCommit.getUserData().get(Translog.TRANSLOG_UUID_KEY) + "] is not equal to last commit's translog uuid [" + translogUUID + "]");
        }
        logger.debug("starting index commit [{}]", startingIndexCommit.getUserData());
        if (startingIndexCommit.equals(lastIndexCommitCommit) == false) {
            try (IndexWriter writer = newAppendingIndexWriter(directory, startingIndexCommit)) {
                // this achieves two things:
                // - by committing a new commit based on the starting commit, it make sure the starting commit will be opened
                // - deletes any other commit (by lucene standard deletion policy)
                // 
                // note that we can't just use IndexCommit.delete() as we really want to make sure that those files won't be used
                // even if a virus scanner causes the files not to be used.
                // The new commit will use segment files from the starting commit but userData from the last commit by default.
                // Thus, we need to manually set the userData from the starting commit to the new commit.
                writer.setLiveCommitData(startingIndexCommit.getUserData().entrySet());
                writer.commit();
            }
        }
    } finally {
        metadataLock.writeLock().unlock();
    }
}

Example 54 with IndexCommit

use of org.apache.lucene.index.IndexCommit in project crate by crate.

the class InternalEngineTests method testKeepMinRetainedSeqNoByMergePolicy.

@Test
public void testKeepMinRetainedSeqNoByMergePolicy() throws IOException {
    IOUtils.close(engine, store);
    Settings.Builder settings = Settings.builder().put(defaultSettings.getSettings()).put(IndexSettings.INDEX_SOFT_DELETES_SETTING.getKey(), true).put(IndexSettings.INDEX_SOFT_DELETES_RETENTION_OPERATIONS_SETTING.getKey(), randomLongBetween(0, 10));
    final IndexMetadata indexMetadata = IndexMetadata.builder(defaultSettings.getIndexMetadata()).settings(settings).build();
    final IndexSettings indexSettings = IndexSettingsModule.newIndexSettings(indexMetadata);
    final AtomicLong globalCheckpoint = new AtomicLong(SequenceNumbers.NO_OPS_PERFORMED);
    final long primaryTerm = randomLongBetween(1, Long.MAX_VALUE);
    final AtomicLong retentionLeasesVersion = new AtomicLong();
    final AtomicReference<RetentionLeases> retentionLeasesHolder = new AtomicReference<>(new RetentionLeases(primaryTerm, retentionLeasesVersion.get(), Collections.emptyList()));
    final List<Engine.Operation> operations = generateSingleDocHistory(true, randomFrom(VersionType.INTERNAL, VersionType.EXTERNAL), false, 2, 10, 300, "2");
    Randomness.shuffle(operations);
    Set<Long> existingSeqNos = new HashSet<>();
    store = createStore();
    engine = createEngine(config(indexSettings, store, createTempDir(), newMergePolicy(), null, null, globalCheckpoint::get, retentionLeasesHolder::get));
    assertThat(engine.getMinRetainedSeqNo(), equalTo(0L));
    long lastMinRetainedSeqNo = engine.getMinRetainedSeqNo();
    for (Engine.Operation op : operations) {
        final Engine.Result result;
        if (op instanceof Engine.Index) {
            result = engine.index((Engine.Index) op);
        } else {
            result = engine.delete((Engine.Delete) op);
        }
        existingSeqNos.add(result.getSeqNo());
        if (randomBoolean()) {
            // advance persisted local checkpoint
            engine.syncTranslog();
            assertEquals(engine.getProcessedLocalCheckpoint(), engine.getPersistedLocalCheckpoint());
            globalCheckpoint.set(randomLongBetween(globalCheckpoint.get(), engine.getLocalCheckpointTracker().getPersistedCheckpoint()));
        }
        if (randomBoolean()) {
            retentionLeasesVersion.incrementAndGet();
            final int length = randomIntBetween(0, 8);
            final List<RetentionLease> leases = new ArrayList<>(length);
            for (int i = 0; i < length; i++) {
                final String id = randomAlphaOfLength(8);
                final long retainingSequenceNumber = randomLongBetween(0, Math.max(0, globalCheckpoint.get()));
                final long timestamp = randomLongBetween(0L, Long.MAX_VALUE);
                final String source = randomAlphaOfLength(8);
                leases.add(new RetentionLease(id, retainingSequenceNumber, timestamp, source));
            }
            retentionLeasesHolder.set(new RetentionLeases(primaryTerm, retentionLeasesVersion.get(), leases));
        }
        if (rarely()) {
            settings.put(IndexSettings.INDEX_SOFT_DELETES_RETENTION_OPERATIONS_SETTING.getKey(), randomLongBetween(0, 10));
            indexSettings.updateIndexMetadata(IndexMetadata.builder(defaultSettings.getIndexMetadata()).settings(settings).build());
            engine.onSettingsChanged(indexSettings.getTranslogRetentionAge(), indexSettings.getTranslogRetentionSize(), indexSettings.getSoftDeleteRetentionOperations());
        }
        if (rarely()) {
            engine.refresh("test");
        }
        if (rarely()) {
            engine.flush(true, true);
            assertThat(Long.parseLong(engine.getLastCommittedSegmentInfos().userData.get(Engine.MIN_RETAINED_SEQNO)), equalTo(engine.getMinRetainedSeqNo()));
        }
        if (rarely()) {
            engine.forceMerge(randomBoolean(), 1, false, false, false, UUIDs.randomBase64UUID());
        }
        try (Closeable ignored = engine.acquireHistoryRetentionLock(Engine.HistorySource.INDEX)) {
            long minRetainSeqNos = engine.getMinRetainedSeqNo();
            assertThat(minRetainSeqNos, lessThanOrEqualTo(globalCheckpoint.get() + 1));
            Long[] expectedOps = existingSeqNos.stream().filter(seqno -> seqno >= minRetainSeqNos).toArray(Long[]::new);
            Set<Long> actualOps = readAllOperationsInLucene(engine, createMapperService("test")).stream().map(Translog.Operation::seqNo).collect(Collectors.toSet());
            assertThat(actualOps, containsInAnyOrder(expectedOps));
        }
        try (Engine.IndexCommitRef commitRef = engine.acquireSafeIndexCommit()) {
            IndexCommit safeCommit = commitRef.getIndexCommit();
            if (safeCommit.getUserData().containsKey(Engine.MIN_RETAINED_SEQNO)) {
                lastMinRetainedSeqNo = Long.parseLong(safeCommit.getUserData().get(Engine.MIN_RETAINED_SEQNO));
            }
        }
    }
    if (randomBoolean()) {
        engine.close();
    } else {
        engine.flushAndClose();
    }
    try (InternalEngine recoveringEngine = new InternalEngine(engine.config())) {
        assertThat(recoveringEngine.getMinRetainedSeqNo(), equalTo(lastMinRetainedSeqNo));
    }
}

Example 55 with IndexCommit

use of org.apache.lucene.index.IndexCommit in project crate by crate.

the class InternalEngineTests method testCleanUpCommitsWhenGlobalCheckpointAdvanced.

@Test
public void testCleanUpCommitsWhenGlobalCheckpointAdvanced() throws Exception {
    IOUtils.close(engine, store);
    final IndexSettings indexSettings = IndexSettingsModule.newIndexSettings("test", Settings.builder().put(defaultSettings.getSettings()).put(IndexSettings.INDEX_TRANSLOG_RETENTION_SIZE_SETTING.getKey(), -1).put(IndexSettings.INDEX_TRANSLOG_RETENTION_AGE_SETTING.getKey(), -1).build());
    final AtomicLong globalCheckpoint = new AtomicLong(SequenceNumbers.NO_OPS_PERFORMED);
    try (Store store = createStore();
        InternalEngine engine = createEngine(config(indexSettings, store, createTempDir(), newMergePolicy(), null, null, globalCheckpoint::get))) {
        final int numDocs = scaledRandomIntBetween(10, 100);
        for (int docId = 0; docId < numDocs; docId++) {
            index(engine, docId);
            if (rarely()) {
                engine.flush(randomBoolean(), true);
            }
        }
        engine.flush(false, randomBoolean());
        globalCheckpoint.set(randomLongBetween(globalCheckpoint.get(), engine.getPersistedLocalCheckpoint()));
        engine.syncTranslog();
        List<IndexCommit> commits = DirectoryReader.listCommits(store.directory());
        assertThat(Long.parseLong(commits.get(0).getUserData().get(SequenceNumbers.MAX_SEQ_NO)), lessThanOrEqualTo(globalCheckpoint.get()));
        for (int i = 1; i < commits.size(); i++) {
            assertThat(Long.parseLong(commits.get(i).getUserData().get(SequenceNumbers.MAX_SEQ_NO)), greaterThan(globalCheckpoint.get()));
        }
        // Global checkpoint advanced enough - only the last commit is kept.
        globalCheckpoint.set(randomLongBetween(engine.getPersistedLocalCheckpoint(), Long.MAX_VALUE));
        engine.syncTranslog();
        assertThat(DirectoryReader.listCommits(store.directory()), contains(commits.get(commits.size() - 1)));
        assertThat(engine.getTranslog().totalOperations(), equalTo(0));
    }
}