Search in sources :

Example 21 with FlushRequest

use of org.graylog.shaded.elasticsearch7.org.elasticsearch.action.admin.indices.flush.FlushRequest in project crate by crate.

the class TransportVerifyShardBeforeCloseAction method executeShardOperation.

private void executeShardOperation(final ShardRequest request, final IndexShard indexShard) throws IOException {
    final ShardId shardId = indexShard.shardId();
    if (indexShard.getActiveOperationsCount() != IndexShard.OPERATIONS_BLOCKED) {
        throw new IllegalStateException("Index shard " + shardId + " is not blocking all operations during closing");
    }
    final ClusterBlocks clusterBlocks = clusterService.state().blocks();
    if (clusterBlocks.hasIndexBlock(shardId.getIndexName(), request.clusterBlock()) == false) {
        throw new IllegalStateException("Index shard " + shardId + " must be blocked by " + request.clusterBlock() + " before closing");
    }
    if (request.isPhase1()) {
        // in order to advance the global checkpoint to the maximum sequence number, the (persisted) local checkpoint needs to be
        // advanced first, which, when using async translog syncing, does not automatically hold at the time where we have acquired
        // all operation permits. Instead, this requires and explicit sync, which communicates the updated (persisted) local checkpoint
        // to the primary (we call this phase1), and phase2 can then use the fact that the global checkpoint has moved to the maximum
        // sequence number to pass the verifyShardBeforeIndexClosing check and create a safe commit where the maximum sequence number
        // is equal to the global checkpoint.
        indexShard.sync();
    } else {
        indexShard.verifyShardBeforeIndexClosing();
        indexShard.flush(new FlushRequest().force(true).waitIfOngoing(true));
        logger.trace("{} shard is ready for closing", shardId);
    }
}
Also used : ShardId(org.elasticsearch.index.shard.ShardId) ClusterBlocks(org.elasticsearch.cluster.block.ClusterBlocks) FlushRequest(org.elasticsearch.action.admin.indices.flush.FlushRequest)

Example 22 with FlushRequest

use of org.graylog.shaded.elasticsearch7.org.elasticsearch.action.admin.indices.flush.FlushRequest in project crate by crate.

the class IndexShard method resetEngineToGlobalCheckpoint.

/**
 * Rollback the current engine to the safe commit, then replay local translog up to the global checkpoint.
 */
void resetEngineToGlobalCheckpoint() throws IOException {
    assert Thread.holdsLock(mutex) == false : "resetting engine under mutex";
    assert getActiveOperationsCount() == OPERATIONS_BLOCKED : "resetting engine without blocking operations; active operations are [" + getActiveOperations() + ']';
    // persist the global checkpoint to disk
    sync();
    final SeqNoStats seqNoStats = seqNoStats();
    final TranslogStats translogStats = translogStats();
    // flush to make sure the latest commit, which will be opened by the read-only engine, includes all operations.
    flush(new FlushRequest().waitIfOngoing(true));
    SetOnce<Engine> newEngineReference = new SetOnce<>();
    final long globalCheckpoint = getLastKnownGlobalCheckpoint();
    assert globalCheckpoint == getLastSyncedGlobalCheckpoint();
    synchronized (engineMutex) {
        verifyNotClosed();
        // we must create both new read-only engine and new read-write engine under engineMutex to ensure snapshotStoreMetadata,
        // acquireXXXCommit and close works.
        final Engine readOnlyEngine = new ReadOnlyEngine(newEngineConfig(replicationTracker), seqNoStats, translogStats, false, Function.identity()) {

            @Override
            public IndexCommitRef acquireLastIndexCommit(boolean flushFirst) {
                synchronized (engineMutex) {
                    if (newEngineReference.get() == null) {
                        throw new AlreadyClosedException("engine was closed");
                    }
                    // ignore flushFirst since we flushed above and we do not want to interfere with ongoing translog replay
                    return newEngineReference.get().acquireLastIndexCommit(false);
                }
            }

            @Override
            public IndexCommitRef acquireSafeIndexCommit() {
                synchronized (engineMutex) {
                    if (newEngineReference.get() == null) {
                        throw new AlreadyClosedException("engine was closed");
                    }
                    return newEngineReference.get().acquireSafeIndexCommit();
                }
            }

            @Override
            public void close() throws IOException {
                assert Thread.holdsLock(engineMutex);
                Engine newEngine = newEngineReference.get();
                if (newEngine == currentEngineReference.get()) {
                    // we successfully installed the new engine so do not close it.
                    newEngine = null;
                }
                IOUtils.close(super::close, newEngine);
            }
        };
        IOUtils.close(currentEngineReference.getAndSet(readOnlyEngine));
        newEngineReference.set(engineFactory.newReadWriteEngine(newEngineConfig(replicationTracker)));
        onNewEngine(newEngineReference.get());
    }
    final Engine.TranslogRecoveryRunner translogRunner = (engine, snapshot) -> runTranslogRecovery(engine, snapshot, Engine.Operation.Origin.LOCAL_RESET, () -> {
    // TODO: add a dedicate recovery stats for the reset translog
    });
    newEngineReference.get().recoverFromTranslog(translogRunner, globalCheckpoint);
    newEngineReference.get().refresh("reset_engine");
    synchronized (engineMutex) {
        verifyNotClosed();
        IOUtils.close(currentEngineReference.getAndSet(newEngineReference.get()));
        // We set active because we are now writing operations to the engine; this way,
        // if we go idle after some time and become inactive, we still give sync'd flush a chance to run.
        active.set(true);
    }
    // time elapses after the engine is created above (pulling the config settings) until we set the engine reference, during
    // which settings changes could possibly have happened, so here we forcefully push any config changes to the new engine.
    onSettingsChanged();
}
Also used : Query(org.apache.lucene.search.Query) UpgradeRequest(org.elasticsearch.action.admin.indices.upgrade.post.UpgradeRequest) LongSupplier(java.util.function.LongSupplier) BigArrays(org.elasticsearch.common.util.BigArrays) IndexMetadata(org.elasticsearch.cluster.metadata.IndexMetadata) Term(org.apache.lucene.index.Term) AlreadyClosedException(org.apache.lucene.store.AlreadyClosedException) RecoveryStats(org.elasticsearch.index.recovery.RecoveryStats) ReferenceManager(org.apache.lucene.search.ReferenceManager) SeqNoStats(org.elasticsearch.index.seqno.SeqNoStats) UsageTrackingQueryCachingPolicy(org.apache.lucene.search.UsageTrackingQueryCachingPolicy) EngineConfig(org.elasticsearch.index.engine.EngineConfig) WriteStateException(org.elasticsearch.gateway.WriteStateException) IndexNotFoundException(org.elasticsearch.index.IndexNotFoundException) Map(java.util.Map) ObjectLongMap(com.carrotsearch.hppc.ObjectLongMap) QueryCachingPolicy(org.apache.lucene.search.QueryCachingPolicy) CheckedRunnable(org.elasticsearch.common.CheckedRunnable) EnumSet(java.util.EnumSet) PeerRecoveryTargetService(org.elasticsearch.indices.recovery.PeerRecoveryTargetService) Set(java.util.Set) StandardCharsets(java.nio.charset.StandardCharsets) ClosedByInterruptException(java.nio.channels.ClosedByInterruptException) Booleans(io.crate.common.Booleans) CountDownLatch(java.util.concurrent.CountDownLatch) RecoverySource(org.elasticsearch.cluster.routing.RecoverySource) AbstractRunnable(org.elasticsearch.common.util.concurrent.AbstractRunnable) Exceptions(io.crate.exceptions.Exceptions) Logger(org.apache.logging.log4j.Logger) RestStatus(org.elasticsearch.rest.RestStatus) ReplicationTracker(org.elasticsearch.index.seqno.ReplicationTracker) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) ThreadInterruptedException(org.apache.lucene.util.ThreadInterruptedException) IndexCommit(org.apache.lucene.index.IndexCommit) StoreStats(org.elasticsearch.index.store.StoreStats) Tuple(io.crate.common.collections.Tuple) RecoveryFailedException(org.elasticsearch.indices.recovery.RecoveryFailedException) BytesStreamOutput(org.elasticsearch.common.io.stream.BytesStreamOutput) IndexModule(org.elasticsearch.index.IndexModule) CodecService(org.elasticsearch.index.codec.CodecService) ArrayList(java.util.ArrayList) CircuitBreakerService(org.elasticsearch.indices.breaker.CircuitBreakerService) XContentHelper(org.elasticsearch.common.xcontent.XContentHelper) RetentionLease(org.elasticsearch.index.seqno.RetentionLease) RetentionLeases(org.elasticsearch.index.seqno.RetentionLeases) IndexCache(org.elasticsearch.index.cache.IndexCache) Store(org.elasticsearch.index.store.Store) BiConsumer(java.util.function.BiConsumer) StreamSupport(java.util.stream.StreamSupport) IndicesService(org.elasticsearch.indices.IndicesService) TranslogConfig(org.elasticsearch.index.translog.TranslogConfig) Nullable(javax.annotation.Nullable) EngineException(org.elasticsearch.index.engine.EngineException) SourceToParse(org.elasticsearch.index.mapper.SourceToParse) AsyncIOProcessor(org.elasticsearch.common.util.concurrent.AsyncIOProcessor) SequenceNumbers(org.elasticsearch.index.seqno.SequenceNumbers) SetOnce(org.apache.lucene.util.SetOnce) IdFieldMapper(org.elasticsearch.index.mapper.IdFieldMapper) IndexService(org.elasticsearch.index.IndexService) IOUtils(io.crate.common.io.IOUtils) IOException(java.io.IOException) ParsedDocument(org.elasticsearch.index.mapper.ParsedDocument) RepositoriesService(org.elasticsearch.repositories.RepositoriesService) Segment(org.elasticsearch.index.engine.Segment) AtomicLong(java.util.concurrent.atomic.AtomicLong) ReplicationResponse(org.elasticsearch.action.support.replication.ReplicationResponse) CounterMetric(org.elasticsearch.common.metrics.CounterMetric) ActionListener(org.elasticsearch.action.ActionListener) ElasticsearchException(org.elasticsearch.ElasticsearchException) SafeCommitInfo(org.elasticsearch.index.engine.SafeCommitInfo) TimeoutException(java.util.concurrent.TimeoutException) SnapshotRecoverySource(org.elasticsearch.cluster.routing.RecoverySource.SnapshotRecoverySource) VersionType(org.elasticsearch.index.VersionType) StoreFileMetadata(org.elasticsearch.index.store.StoreFileMetadata) Settings(org.elasticsearch.common.settings.Settings) ResyncTask(org.elasticsearch.index.shard.PrimaryReplicaSyncer.ResyncTask) Locale(java.util.Locale) ThreadPool(org.elasticsearch.threadpool.ThreadPool) ActionRunnable(org.elasticsearch.action.ActionRunnable) Releasable(org.elasticsearch.common.lease.Releasable) ByteSizeValue(org.elasticsearch.common.unit.ByteSizeValue) RefreshFailedEngineException(org.elasticsearch.index.engine.RefreshFailedEngineException) CheckIndex(org.apache.lucene.index.CheckIndex) UNASSIGNED_SEQ_NO(org.elasticsearch.index.seqno.SequenceNumbers.UNASSIGNED_SEQ_NO) IndexShardRoutingTable(org.elasticsearch.cluster.routing.IndexShardRoutingTable) Collectors(java.util.stream.Collectors) SegmentInfos(org.apache.lucene.index.SegmentInfos) ReadOnlyEngine(org.elasticsearch.index.engine.ReadOnlyEngine) Engine(org.elasticsearch.index.engine.Engine) Objects(java.util.Objects) MapperService(org.elasticsearch.index.mapper.MapperService) TranslogStats(org.elasticsearch.index.translog.TranslogStats) List(java.util.List) Version(org.elasticsearch.Version) MeanMetric(org.elasticsearch.common.metrics.MeanMetric) RetentionLeaseStats(org.elasticsearch.index.seqno.RetentionLeaseStats) MappingMetadata(org.elasticsearch.cluster.metadata.MappingMetadata) IndicesClusterStateService(org.elasticsearch.indices.cluster.IndicesClusterStateService) RecoveryState(org.elasticsearch.indices.recovery.RecoveryState) RetentionLeaseSyncer(org.elasticsearch.index.seqno.RetentionLeaseSyncer) TimeValue(io.crate.common.unit.TimeValue) Optional(java.util.Optional) ShardRouting(org.elasticsearch.cluster.routing.ShardRouting) CommitStats(org.elasticsearch.index.engine.CommitStats) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) CheckedConsumer(org.elasticsearch.common.CheckedConsumer) Index(org.elasticsearch.index.Index) Lucene(org.elasticsearch.common.lucene.Lucene) AtomicReference(java.util.concurrent.atomic.AtomicReference) Function(java.util.function.Function) ParameterizedMessage(org.apache.logging.log4j.message.ParameterizedMessage) HashSet(java.util.HashSet) ForceMergeRequest(org.elasticsearch.action.admin.indices.forcemerge.ForceMergeRequest) RootObjectMapper(org.elasticsearch.index.mapper.RootObjectMapper) MetadataSnapshot(org.elasticsearch.index.store.Store.MetadataSnapshot) IndexSettings(org.elasticsearch.index.IndexSettings) Mapping(org.elasticsearch.index.mapper.Mapping) DocumentMapper(org.elasticsearch.index.mapper.DocumentMapper) PrintStream(java.io.PrintStream) Repository(org.elasticsearch.repositories.Repository) Uid(org.elasticsearch.index.mapper.Uid) RecoveryTarget(org.elasticsearch.indices.recovery.RecoveryTarget) EngineFactory(org.elasticsearch.index.engine.EngineFactory) IndexingMemoryController(org.elasticsearch.indices.IndexingMemoryController) TimeUnit(java.util.concurrent.TimeUnit) Consumer(java.util.function.Consumer) ExceptionsHelper(org.elasticsearch.ExceptionsHelper) FlushRequest(org.elasticsearch.action.admin.indices.flush.FlushRequest) Closeable(java.io.Closeable) Assertions(org.elasticsearch.Assertions) Translog(org.elasticsearch.index.translog.Translog) Collections(java.util.Collections) RunOnce(org.elasticsearch.common.util.concurrent.RunOnce) SeqNoStats(org.elasticsearch.index.seqno.SeqNoStats) FlushRequest(org.elasticsearch.action.admin.indices.flush.FlushRequest) SetOnce(org.apache.lucene.util.SetOnce) ReadOnlyEngine(org.elasticsearch.index.engine.ReadOnlyEngine) TranslogStats(org.elasticsearch.index.translog.TranslogStats) AlreadyClosedException(org.apache.lucene.store.AlreadyClosedException) ReadOnlyEngine(org.elasticsearch.index.engine.ReadOnlyEngine) Engine(org.elasticsearch.index.engine.Engine)

Example 23 with FlushRequest

use of org.graylog.shaded.elasticsearch7.org.elasticsearch.action.admin.indices.flush.FlushRequest in project graylog2-server by Graylog2.

the class IndicesAdapterES7 method flush.

@Override
public void flush(String index) {
    final FlushRequest request = new FlushRequest(index);
    client.execute((c, requestOptions) -> c.indices().flush(request, requestOptions), "Unable to flush index " + index);
}
Also used : FlushRequest(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.admin.indices.flush.FlushRequest)

Aggregations

FlushRequest (org.elasticsearch.action.admin.indices.flush.FlushRequest)22 IOException (java.io.IOException)7 Engine (org.elasticsearch.index.engine.Engine)6 IndexShard (org.elasticsearch.index.shard.IndexShard)5 ClosedByInterruptException (java.nio.channels.ClosedByInterruptException)3 ArrayList (java.util.ArrayList)3 TimeoutException (java.util.concurrent.TimeoutException)3 ElasticsearchException (org.elasticsearch.ElasticsearchException)3 ForceMergeRequest (org.elasticsearch.action.admin.indices.forcemerge.ForceMergeRequest)3 BytesArray (org.elasticsearch.common.bytes.BytesArray)3 Settings (org.elasticsearch.common.settings.Settings)3 AbstractRunnable (org.elasticsearch.common.util.concurrent.AbstractRunnable)3 IndexNotFoundException (org.elasticsearch.index.IndexNotFoundException)3 Term (org.apache.lucene.index.Term)2 AlreadyClosedException (org.apache.lucene.store.AlreadyClosedException)2 ThreadInterruptedException (org.apache.lucene.util.ThreadInterruptedException)2 UpgradeRequest (org.elasticsearch.action.admin.indices.upgrade.post.UpgradeRequest)2 EngineException (org.elasticsearch.index.engine.EngineException)2 RefreshFailedEngineException (org.elasticsearch.index.engine.RefreshFailedEngineException)2 ParsedDocument (org.elasticsearch.index.mapper.ParsedDocument)2