Search in sources :

Example 16 with ReplicationResponse

use of org.elasticsearch.action.support.replication.ReplicationResponse in project elasticsearch by elastic.

the class GlobalCheckpointSyncAction method shardOperationOnPrimary.

@Override
protected PrimaryResult shardOperationOnPrimary(PrimaryRequest request, IndexShard indexShard) throws Exception {
    long checkpoint = indexShard.getGlobalCheckpoint();
    indexShard.getTranslog().sync();
    return new PrimaryResult(new ReplicaRequest(request, checkpoint), new ReplicationResponse());
}
Also used : ReplicationResponse(org.elasticsearch.action.support.replication.ReplicationResponse)

Example 17 with ReplicationResponse

use of org.elasticsearch.action.support.replication.ReplicationResponse in project crate by crate.

the class TransportShardRefreshAction method shardOperationOnPrimary.

@Override
protected void shardOperationOnPrimary(BasicReplicationRequest shardRequest, IndexShard primary, ActionListener<PrimaryResult<BasicReplicationRequest, ReplicationResponse>> listener) {
    ActionListener.completeWith(listener, () -> {
        primary.refresh("api");
        logger.trace("{} refresh request executed on primary", primary.shardId());
        return new PrimaryResult<>(shardRequest, new ReplicationResponse());
    });
}
Also used : ReplicationResponse(org.elasticsearch.action.support.replication.ReplicationResponse)

Example 18 with ReplicationResponse

use of org.elasticsearch.action.support.replication.ReplicationResponse in project crate by crate.

the class RecoverySourceHandler method createRetentionLease.

void createRetentionLease(final long startingSeqNo, ActionListener<RetentionLease> listener) {
    runUnderPrimaryPermit(() -> {
        // Clone the peer recovery retention lease belonging to the source shard. We are retaining history between the the local
        // checkpoint of the safe commit we're creating and this lease's retained seqno with the retention lock, and by cloning an
        // existing lease we (approximately) know that all our peers are also retaining history as requested by the cloned lease. If
        // the recovery now fails before copying enough history over then a subsequent attempt will find this lease, determine it is
        // not enough, and fall back to a file-based recovery.
        // 
        // (approximately) because we do not guarantee to be able to satisfy every lease on every peer.
        logger.trace("cloning primary's retention lease");
        try {
            final StepListener<ReplicationResponse> cloneRetentionLeaseStep = new StepListener<>();
            final RetentionLease clonedLease = shard.cloneLocalPeerRecoveryRetentionLease(request.targetNode().getId(), new ThreadedActionListener<>(logger, shard.getThreadPool(), ThreadPool.Names.GENERIC, cloneRetentionLeaseStep, false));
            logger.trace("cloned primary's retention lease as [{}]", clonedLease);
            cloneRetentionLeaseStep.whenComplete(rr -> listener.onResponse(clonedLease), listener::onFailure);
        } catch (RetentionLeaseNotFoundException e) {
            // recovery as a conservative estimate for the global checkpoint.
            assert shard.indexSettings().getIndexVersionCreated().before(Version.V_4_3_0) || shard.indexSettings().isSoftDeleteEnabled() == false;
            final StepListener<ReplicationResponse> addRetentionLeaseStep = new StepListener<>();
            final long estimatedGlobalCheckpoint = startingSeqNo - 1;
            final RetentionLease newLease = shard.addPeerRecoveryRetentionLease(request.targetNode().getId(), estimatedGlobalCheckpoint, new ThreadedActionListener<>(logger, shard.getThreadPool(), ThreadPool.Names.GENERIC, addRetentionLeaseStep, false));
            addRetentionLeaseStep.whenComplete(rr -> listener.onResponse(newLease), listener::onFailure);
            logger.trace("created retention lease with estimated checkpoint of [{}]", estimatedGlobalCheckpoint);
        }
    }, shardId + " establishing retention lease for [" + request.targetAllocationId() + "]", shard, cancellableThreads, logger);
}
Also used : CancellableThreads(org.elasticsearch.common.util.CancellableThreads) Arrays(java.util.Arrays) IndexFormatTooNewException(org.apache.lucene.index.IndexFormatTooNewException) Releasables(org.elasticsearch.common.lease.Releasables) RecoveryEngineException(org.elasticsearch.index.engine.RecoveryEngineException) RetentionLeaseNotFoundException(org.elasticsearch.index.seqno.RetentionLeaseNotFoundException) CorruptIndexException(org.apache.lucene.index.CorruptIndexException) StoreFileMetadata(org.elasticsearch.index.store.StoreFileMetadata) Transports(org.elasticsearch.transport.Transports) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) Locale(java.util.Locale) ThreadPool(org.elasticsearch.threadpool.ThreadPool) ActionRunnable(org.elasticsearch.action.ActionRunnable) IOContext(org.apache.lucene.store.IOContext) StepListener(org.elasticsearch.action.StepListener) Releasable(org.elasticsearch.common.lease.Releasable) ByteSizeValue(org.elasticsearch.common.unit.ByteSizeValue) PlainActionFuture(org.elasticsearch.action.support.PlainActionFuture) IndexShardRoutingTable(org.elasticsearch.cluster.routing.IndexShardRoutingTable) BytesReference(org.elasticsearch.common.bytes.BytesReference) Engine(org.elasticsearch.index.engine.Engine) RemoteTransportException(org.elasticsearch.transport.RemoteTransportException) List(java.util.List) Logger(org.apache.logging.log4j.Logger) Version(org.elasticsearch.Version) InputStreamIndexInput(org.elasticsearch.common.lucene.store.InputStreamIndexInput) TimeValue(io.crate.common.unit.TimeValue) ReplicationTracker(org.elasticsearch.index.seqno.ReplicationTracker) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) IndexCommit(org.apache.lucene.index.IndexCommit) ShardRouting(org.elasticsearch.cluster.routing.ShardRouting) IndexShardClosedException(org.elasticsearch.index.shard.IndexShardClosedException) IndexShardRelocatedException(org.elasticsearch.index.shard.IndexShardRelocatedException) CompletableFuture(java.util.concurrent.CompletableFuture) Deque(java.util.Deque) ParameterizedMessage(org.apache.logging.log4j.message.ParameterizedMessage) ArrayList(java.util.ArrayList) BytesArray(org.elasticsearch.common.bytes.BytesArray) RetentionLease(org.elasticsearch.index.seqno.RetentionLease) RetentionLeases(org.elasticsearch.index.seqno.RetentionLeases) Store(org.elasticsearch.index.store.Store) StreamSupport(java.util.stream.StreamSupport) IntSupplier(java.util.function.IntSupplier) Loggers(org.elasticsearch.common.logging.Loggers) ArrayUtil(org.apache.lucene.util.ArrayUtil) FutureUtils(org.elasticsearch.common.util.concurrent.FutureUtils) SequenceNumbers(org.elasticsearch.index.seqno.SequenceNumbers) IndexShardState(org.elasticsearch.index.shard.IndexShardState) IndexInput(org.apache.lucene.store.IndexInput) SetOnce(org.apache.lucene.util.SetOnce) IOUtils(io.crate.common.io.IOUtils) IndexShard(org.elasticsearch.index.shard.IndexShard) IOException(java.io.IOException) StopWatch(org.elasticsearch.common.StopWatch) IndexFormatTooOldException(org.apache.lucene.index.IndexFormatTooOldException) ConcurrentLinkedDeque(java.util.concurrent.ConcurrentLinkedDeque) Consumer(java.util.function.Consumer) ExceptionsHelper(org.elasticsearch.ExceptionsHelper) AtomicLong(java.util.concurrent.atomic.AtomicLong) ReplicationResponse(org.elasticsearch.action.support.replication.ReplicationResponse) Closeable(java.io.Closeable) Translog(org.elasticsearch.index.translog.Translog) ThreadedActionListener(org.elasticsearch.action.support.ThreadedActionListener) Comparator(java.util.Comparator) Collections(java.util.Collections) RateLimiter(org.apache.lucene.store.RateLimiter) ActionListener(org.elasticsearch.action.ActionListener) RetentionLeaseNotFoundException(org.elasticsearch.index.seqno.RetentionLeaseNotFoundException) ThreadedActionListener(org.elasticsearch.action.support.ThreadedActionListener) RetentionLease(org.elasticsearch.index.seqno.RetentionLease) StepListener(org.elasticsearch.action.StepListener) ReplicationResponse(org.elasticsearch.action.support.replication.ReplicationResponse)

Example 19 with ReplicationResponse

use of org.elasticsearch.action.support.replication.ReplicationResponse in project crate by crate.

the class ReplicationTracker method createMissingPeerRecoveryRetentionLeases.

/**
 * Create any required peer-recovery retention leases that do not currently exist because we just did a rolling upgrade from a version
 * prior to {@link Version#V_4_3_0} that does not create peer-recovery retention leases.
 */
public synchronized void createMissingPeerRecoveryRetentionLeases(ActionListener<Void> listener) {
    if (hasAllPeerRecoveryRetentionLeases == false) {
        final List<ShardRouting> shardRoutings = routingTable.assignedShards();
        final GroupedActionListener<ReplicationResponse> groupedActionListener = new GroupedActionListener<>(ActionListener.wrap(vs -> {
            setHasAllPeerRecoveryRetentionLeases();
            listener.onResponse(null);
        }, listener::onFailure), shardRoutings.size());
        for (ShardRouting shardRouting : shardRoutings) {
            if (retentionLeases.contains(getPeerRecoveryRetentionLeaseId(shardRouting))) {
                groupedActionListener.onResponse(null);
            } else {
                final CheckpointState checkpointState = checkpoints.get(shardRouting.allocationId().getId());
                if (checkpointState.tracked == false) {
                    groupedActionListener.onResponse(null);
                } else {
                    logger.trace("createMissingPeerRecoveryRetentionLeases: adding missing lease for {}", shardRouting);
                    try {
                        addPeerRecoveryRetentionLease(shardRouting.currentNodeId(), Math.max(SequenceNumbers.NO_OPS_PERFORMED, checkpointState.globalCheckpoint), groupedActionListener);
                    } catch (Exception e) {
                        groupedActionListener.onFailure(e);
                    }
                }
            }
        }
    } else {
        logger.trace("createMissingPeerRecoveryRetentionLeases: nothing to do");
        listener.onResponse(null);
    }
}
Also used : ShardRouting(org.elasticsearch.cluster.routing.ShardRouting) ShardId(org.elasticsearch.index.shard.ShardId) SuppressForbidden(io.crate.common.SuppressForbidden) StreamOutput(org.elasticsearch.common.io.stream.StreamOutput) SafeCommitInfo(org.elasticsearch.index.engine.SafeCommitInfo) Tuple(io.crate.common.collections.Tuple) LongSupplier(java.util.function.LongSupplier) IndexMetadata(org.elasticsearch.cluster.metadata.IndexMetadata) HashMap(java.util.HashMap) Function(java.util.function.Function) Supplier(java.util.function.Supplier) HashSet(java.util.HashSet) OptionalLong(java.util.OptionalLong) ObjectLongHashMap(com.carrotsearch.hppc.ObjectLongHashMap) WriteStateException(org.elasticsearch.gateway.WriteStateException) Map(java.util.Map) BiConsumer(java.util.function.BiConsumer) IndexSettings(org.elasticsearch.index.IndexSettings) ObjectLongMap(com.carrotsearch.hppc.ObjectLongMap) StreamSupport(java.util.stream.StreamSupport) ToLongFunction(java.util.function.ToLongFunction) Path(java.nio.file.Path) NamedXContentRegistry(org.elasticsearch.common.xcontent.NamedXContentRegistry) LongStream(java.util.stream.LongStream) Collection(java.util.Collection) IndexShard(org.elasticsearch.index.shard.IndexShard) Set(java.util.Set) IOException(java.io.IOException) GroupedActionListener(org.elasticsearch.action.support.GroupedActionListener) IndexShardRoutingTable(org.elasticsearch.cluster.routing.IndexShardRoutingTable) Collectors(java.util.stream.Collectors) LongConsumer(java.util.function.LongConsumer) Objects(java.util.Objects) AbstractIndexShardComponent(org.elasticsearch.index.shard.AbstractIndexShardComponent) List(java.util.List) ReplicationResponse(org.elasticsearch.action.support.replication.ReplicationResponse) Stream(java.util.stream.Stream) Version(org.elasticsearch.Version) StreamInput(org.elasticsearch.common.io.stream.StreamInput) AllocationId(org.elasticsearch.cluster.routing.AllocationId) Writeable(org.elasticsearch.common.io.stream.Writeable) ReplicationGroup(org.elasticsearch.index.shard.ReplicationGroup) Collections(java.util.Collections) ActionListener(org.elasticsearch.action.ActionListener) GroupedActionListener(org.elasticsearch.action.support.GroupedActionListener) ShardRouting(org.elasticsearch.cluster.routing.ShardRouting) WriteStateException(org.elasticsearch.gateway.WriteStateException) IOException(java.io.IOException) ReplicationResponse(org.elasticsearch.action.support.replication.ReplicationResponse)

Example 20 with ReplicationResponse

use of org.elasticsearch.action.support.replication.ReplicationResponse in project crate by crate.

the class RetentionLeaseBackgroundSyncAction method backgroundSync.

final void backgroundSync(ShardId shardId, String primaryAllocationId, long primaryTerm, RetentionLeases retentionLeases) {
    final Request request = new Request(shardId, retentionLeases);
    transportService.sendChildRequest(clusterService.localNode(), transportPrimaryAction, new ConcreteShardRequest<>(request, primaryAllocationId, primaryTerm), transportOptions, new TransportResponseHandler<ReplicationResponse>() {

        @Override
        public ReplicationResponse read(StreamInput in) throws IOException {
            return newResponseInstance(in);
        }

        @Override
        public String executor() {
            return ThreadPool.Names.SAME;
        }

        @Override
        public void handleResponse(ReplicationResponse response) {
        }

        @Override
        public void handleException(TransportException e) {
            if (ExceptionsHelper.unwrap(e, NodeClosedException.class) != null) {
                // node shutting down
                return;
            }
            if (ExceptionsHelper.unwrap(e, AlreadyClosedException.class, IndexShardClosedException.class) != null) {
                // the shard is closed
                return;
            }
            getLogger().warn(new ParameterizedMessage("{} retention lease background sync failed", shardId), e);
        }
    });
}
Also used : ReplicationRequest(org.elasticsearch.action.support.replication.ReplicationRequest) StreamInput(org.elasticsearch.common.io.stream.StreamInput) ParameterizedMessage(org.apache.logging.log4j.message.ParameterizedMessage) IOException(java.io.IOException) TransportException(org.elasticsearch.transport.TransportException) ReplicationResponse(org.elasticsearch.action.support.replication.ReplicationResponse)

Aggregations

ReplicationResponse (org.elasticsearch.action.support.replication.ReplicationResponse)24 Test (org.junit.Test)10 CountDownLatch (java.util.concurrent.CountDownLatch)9 IndexShard (org.elasticsearch.index.shard.IndexShard)9 ShardId (org.elasticsearch.index.shard.ShardId)8 ShardRouting (org.elasticsearch.cluster.routing.ShardRouting)7 Collections (java.util.Collections)6 List (java.util.List)6 AtomicBoolean (java.util.concurrent.atomic.AtomicBoolean)6 PlainActionFuture (org.elasticsearch.action.support.PlainActionFuture)6 IndexShardRoutingTable (org.elasticsearch.cluster.routing.IndexShardRoutingTable)6 IndicesService (org.elasticsearch.indices.IndicesService)6 IOException (java.io.IOException)5 ArrayList (java.util.ArrayList)5 Arrays (java.util.Arrays)5 BytesArray (org.elasticsearch.common.bytes.BytesArray)5 ByteSizeValue (org.elasticsearch.common.unit.ByteSizeValue)5 Engine (org.elasticsearch.index.engine.Engine)5 SequenceNumbers (org.elasticsearch.index.seqno.SequenceNumbers)5 Translog (org.elasticsearch.index.translog.Translog)5