Search in sources :

Example 1 with Tuple

use of io.crate.common.collections.Tuple in project crate by crate.

the class HttpAuthUpstreamHandler method credentialsFromRequest.

@VisibleForTesting
static Tuple<String, SecureString> credentialsFromRequest(HttpRequest request, @Nullable SSLSession session, Settings settings) {
    String username = null;
    if (request.headers().contains(HttpHeaderNames.AUTHORIZATION.toString())) {
        // Prefer Http Basic Auth
        return Headers.extractCredentialsFromHttpBasicAuthHeader(request.headers().get(HttpHeaderNames.AUTHORIZATION.toString()));
    } else {
        // prefer commonName as userName over AUTH_TRUST_HTTP_DEFAULT_HEADER user
        if (session != null) {
            try {
                Certificate certificate = session.getPeerCertificates()[0];
                username = SSL.extractCN(certificate);
            } catch (ArrayIndexOutOfBoundsException | SSLPeerUnverifiedException ignored) {
            // client cert is optional
            }
        }
        if (username == null) {
            username = AuthSettings.AUTH_TRUST_HTTP_DEFAULT_HEADER.get(settings);
        }
    }
    return new Tuple<>(username, null);
}
Also used : SSLPeerUnverifiedException(javax.net.ssl.SSLPeerUnverifiedException) SecureString(org.elasticsearch.common.settings.SecureString) Tuple(io.crate.common.collections.Tuple) Certificate(java.security.cert.Certificate) VisibleForTesting(io.crate.common.annotations.VisibleForTesting)

Example 2 with Tuple

use of io.crate.common.collections.Tuple in project crate by crate.

the class JobLauncher method createHandlerPhaseAndReceivers.

private List<Tuple<ExecutionPhase, RowConsumer>> createHandlerPhaseAndReceivers(List<ExecutionPhase> handlerPhases, List<RowConsumer> handlerReceivers, InitializationTracker initializationTracker) {
    List<Tuple<ExecutionPhase, RowConsumer>> handlerPhaseAndReceiver = new ArrayList<>();
    ListIterator<RowConsumer> consumerIt = handlerReceivers.listIterator();
    for (ExecutionPhase handlerPhase : handlerPhases) {
        InterceptingRowConsumer interceptingBatchConsumer = new InterceptingRowConsumer(jobId, consumerIt.next(), initializationTracker, executor, transportKillJobsNodeAction);
        handlerPhaseAndReceiver.add(new Tuple<>(handlerPhase, interceptingBatchConsumer));
    }
    return handlerPhaseAndReceiver;
}
Also used : ArrayList(java.util.ArrayList) ExecutionPhase(io.crate.execution.dsl.phases.ExecutionPhase) CollectingRowConsumer(io.crate.data.CollectingRowConsumer) RowConsumer(io.crate.data.RowConsumer) Tuple(io.crate.common.collections.Tuple)

Example 3 with Tuple

use of io.crate.common.collections.Tuple in project crate by crate.

the class InsertSourceFromCells method addDefaults.

private static Tuple<List<Reference>, List<Input<?>>> addDefaults(List<Reference> targets, DocTableInfo table, TransactionContext txnCtx, NodeContext nodeCtx) {
    if (table.defaultExpressionColumns().isEmpty()) {
        return new Tuple<>(targets, List.of());
    }
    InputFactory inputFactory = new InputFactory(nodeCtx);
    Context<CollectExpression<Row, ?>> ctx = inputFactory.ctxForInputColumns(txnCtx);
    ArrayList<Reference> defaultColumns = new ArrayList<>(table.defaultExpressionColumns().size());
    ArrayList<Input<?>> defaultValues = new ArrayList<>();
    for (Reference ref : table.defaultExpressionColumns()) {
        if (targets.contains(ref) == false) {
            defaultColumns.add(ref);
            defaultValues.add(ctx.add(ref.defaultExpression()));
        }
    }
    List<Reference> allColumns;
    if (defaultColumns.isEmpty()) {
        allColumns = targets;
    } else {
        allColumns = Lists2.concat(targets, defaultColumns);
    }
    return new Tuple<>(allColumns, defaultValues);
}
Also used : InputFactory(io.crate.expression.InputFactory) Input(io.crate.data.Input) Reference(io.crate.metadata.Reference) ArrayList(java.util.ArrayList) CollectExpression(io.crate.execution.engine.collect.CollectExpression) NestableCollectExpression(io.crate.execution.engine.collect.NestableCollectExpression) InputCollectExpression(io.crate.execution.engine.collect.InputCollectExpression) Tuple(io.crate.common.collections.Tuple)

Example 4 with Tuple

use of io.crate.common.collections.Tuple in project crate by crate.

the class IndexShard method syncRetentionLeases.

/**
 * Syncs the current retention leases to all replicas.
 */
public void syncRetentionLeases() {
    assert assertPrimaryMode();
    verifyNotClosed();
    replicationTracker.renewPeerRecoveryRetentionLeases();
    final Tuple<Boolean, RetentionLeases> retentionLeases = getRetentionLeases(true);
    if (retentionLeases.v1()) {
        logger.trace("syncing retention leases [{}] after expiration check", retentionLeases.v2());
        retentionLeaseSyncer.sync(shardId, shardRouting.allocationId().getId(), getPendingPrimaryTerm(), retentionLeases.v2(), ActionListener.wrap(r -> {
        }, e -> logger.warn(new ParameterizedMessage("failed to sync retention leases [{}] after expiration check", retentionLeases), e)));
    } else {
        logger.trace("background syncing retention leases [{}] after expiration check", retentionLeases.v2());
        retentionLeaseSyncer.backgroundSync(shardId, shardRouting.allocationId().getId(), getPendingPrimaryTerm(), retentionLeases.v2());
    }
}
Also used : Query(org.apache.lucene.search.Query) UpgradeRequest(org.elasticsearch.action.admin.indices.upgrade.post.UpgradeRequest) LongSupplier(java.util.function.LongSupplier) BigArrays(org.elasticsearch.common.util.BigArrays) IndexMetadata(org.elasticsearch.cluster.metadata.IndexMetadata) Term(org.apache.lucene.index.Term) AlreadyClosedException(org.apache.lucene.store.AlreadyClosedException) RecoveryStats(org.elasticsearch.index.recovery.RecoveryStats) ReferenceManager(org.apache.lucene.search.ReferenceManager) SeqNoStats(org.elasticsearch.index.seqno.SeqNoStats) UsageTrackingQueryCachingPolicy(org.apache.lucene.search.UsageTrackingQueryCachingPolicy) EngineConfig(org.elasticsearch.index.engine.EngineConfig) WriteStateException(org.elasticsearch.gateway.WriteStateException) IndexNotFoundException(org.elasticsearch.index.IndexNotFoundException) Map(java.util.Map) ObjectLongMap(com.carrotsearch.hppc.ObjectLongMap) QueryCachingPolicy(org.apache.lucene.search.QueryCachingPolicy) CheckedRunnable(org.elasticsearch.common.CheckedRunnable) EnumSet(java.util.EnumSet) PeerRecoveryTargetService(org.elasticsearch.indices.recovery.PeerRecoveryTargetService) Set(java.util.Set) StandardCharsets(java.nio.charset.StandardCharsets) ClosedByInterruptException(java.nio.channels.ClosedByInterruptException) Booleans(io.crate.common.Booleans) CountDownLatch(java.util.concurrent.CountDownLatch) RecoverySource(org.elasticsearch.cluster.routing.RecoverySource) AbstractRunnable(org.elasticsearch.common.util.concurrent.AbstractRunnable) Exceptions(io.crate.exceptions.Exceptions) Logger(org.apache.logging.log4j.Logger) RestStatus(org.elasticsearch.rest.RestStatus) ReplicationTracker(org.elasticsearch.index.seqno.ReplicationTracker) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) ThreadInterruptedException(org.apache.lucene.util.ThreadInterruptedException) IndexCommit(org.apache.lucene.index.IndexCommit) StoreStats(org.elasticsearch.index.store.StoreStats) Tuple(io.crate.common.collections.Tuple) RecoveryFailedException(org.elasticsearch.indices.recovery.RecoveryFailedException) BytesStreamOutput(org.elasticsearch.common.io.stream.BytesStreamOutput) IndexModule(org.elasticsearch.index.IndexModule) CodecService(org.elasticsearch.index.codec.CodecService) ArrayList(java.util.ArrayList) CircuitBreakerService(org.elasticsearch.indices.breaker.CircuitBreakerService) XContentHelper(org.elasticsearch.common.xcontent.XContentHelper) RetentionLease(org.elasticsearch.index.seqno.RetentionLease) RetentionLeases(org.elasticsearch.index.seqno.RetentionLeases) IndexCache(org.elasticsearch.index.cache.IndexCache) Store(org.elasticsearch.index.store.Store) BiConsumer(java.util.function.BiConsumer) StreamSupport(java.util.stream.StreamSupport) IndicesService(org.elasticsearch.indices.IndicesService) TranslogConfig(org.elasticsearch.index.translog.TranslogConfig) Nullable(javax.annotation.Nullable) EngineException(org.elasticsearch.index.engine.EngineException) SourceToParse(org.elasticsearch.index.mapper.SourceToParse) AsyncIOProcessor(org.elasticsearch.common.util.concurrent.AsyncIOProcessor) SequenceNumbers(org.elasticsearch.index.seqno.SequenceNumbers) SetOnce(org.apache.lucene.util.SetOnce) IdFieldMapper(org.elasticsearch.index.mapper.IdFieldMapper) IndexService(org.elasticsearch.index.IndexService) IOUtils(io.crate.common.io.IOUtils) IOException(java.io.IOException) ParsedDocument(org.elasticsearch.index.mapper.ParsedDocument) RepositoriesService(org.elasticsearch.repositories.RepositoriesService) Segment(org.elasticsearch.index.engine.Segment) AtomicLong(java.util.concurrent.atomic.AtomicLong) ReplicationResponse(org.elasticsearch.action.support.replication.ReplicationResponse) CounterMetric(org.elasticsearch.common.metrics.CounterMetric) ActionListener(org.elasticsearch.action.ActionListener) ElasticsearchException(org.elasticsearch.ElasticsearchException) SafeCommitInfo(org.elasticsearch.index.engine.SafeCommitInfo) TimeoutException(java.util.concurrent.TimeoutException) SnapshotRecoverySource(org.elasticsearch.cluster.routing.RecoverySource.SnapshotRecoverySource) VersionType(org.elasticsearch.index.VersionType) StoreFileMetadata(org.elasticsearch.index.store.StoreFileMetadata) Settings(org.elasticsearch.common.settings.Settings) ResyncTask(org.elasticsearch.index.shard.PrimaryReplicaSyncer.ResyncTask) Locale(java.util.Locale) ThreadPool(org.elasticsearch.threadpool.ThreadPool) ActionRunnable(org.elasticsearch.action.ActionRunnable) Releasable(org.elasticsearch.common.lease.Releasable) ByteSizeValue(org.elasticsearch.common.unit.ByteSizeValue) RefreshFailedEngineException(org.elasticsearch.index.engine.RefreshFailedEngineException) CheckIndex(org.apache.lucene.index.CheckIndex) UNASSIGNED_SEQ_NO(org.elasticsearch.index.seqno.SequenceNumbers.UNASSIGNED_SEQ_NO) IndexShardRoutingTable(org.elasticsearch.cluster.routing.IndexShardRoutingTable) Collectors(java.util.stream.Collectors) SegmentInfos(org.apache.lucene.index.SegmentInfos) ReadOnlyEngine(org.elasticsearch.index.engine.ReadOnlyEngine) Engine(org.elasticsearch.index.engine.Engine) Objects(java.util.Objects) MapperService(org.elasticsearch.index.mapper.MapperService) TranslogStats(org.elasticsearch.index.translog.TranslogStats) List(java.util.List) Version(org.elasticsearch.Version) MeanMetric(org.elasticsearch.common.metrics.MeanMetric) RetentionLeaseStats(org.elasticsearch.index.seqno.RetentionLeaseStats) MappingMetadata(org.elasticsearch.cluster.metadata.MappingMetadata) IndicesClusterStateService(org.elasticsearch.indices.cluster.IndicesClusterStateService) RecoveryState(org.elasticsearch.indices.recovery.RecoveryState) RetentionLeaseSyncer(org.elasticsearch.index.seqno.RetentionLeaseSyncer) TimeValue(io.crate.common.unit.TimeValue) Optional(java.util.Optional) ShardRouting(org.elasticsearch.cluster.routing.ShardRouting) CommitStats(org.elasticsearch.index.engine.CommitStats) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) CheckedConsumer(org.elasticsearch.common.CheckedConsumer) Index(org.elasticsearch.index.Index) Lucene(org.elasticsearch.common.lucene.Lucene) AtomicReference(java.util.concurrent.atomic.AtomicReference) Function(java.util.function.Function) ParameterizedMessage(org.apache.logging.log4j.message.ParameterizedMessage) HashSet(java.util.HashSet) ForceMergeRequest(org.elasticsearch.action.admin.indices.forcemerge.ForceMergeRequest) RootObjectMapper(org.elasticsearch.index.mapper.RootObjectMapper) MetadataSnapshot(org.elasticsearch.index.store.Store.MetadataSnapshot) IndexSettings(org.elasticsearch.index.IndexSettings) Mapping(org.elasticsearch.index.mapper.Mapping) DocumentMapper(org.elasticsearch.index.mapper.DocumentMapper) PrintStream(java.io.PrintStream) Repository(org.elasticsearch.repositories.Repository) Uid(org.elasticsearch.index.mapper.Uid) RecoveryTarget(org.elasticsearch.indices.recovery.RecoveryTarget) EngineFactory(org.elasticsearch.index.engine.EngineFactory) IndexingMemoryController(org.elasticsearch.indices.IndexingMemoryController) TimeUnit(java.util.concurrent.TimeUnit) Consumer(java.util.function.Consumer) ExceptionsHelper(org.elasticsearch.ExceptionsHelper) FlushRequest(org.elasticsearch.action.admin.indices.flush.FlushRequest) Closeable(java.io.Closeable) Assertions(org.elasticsearch.Assertions) Translog(org.elasticsearch.index.translog.Translog) Collections(java.util.Collections) RunOnce(org.elasticsearch.common.util.concurrent.RunOnce) ParameterizedMessage(org.apache.logging.log4j.message.ParameterizedMessage) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) RetentionLeases(org.elasticsearch.index.seqno.RetentionLeases)

Example 5 with Tuple

use of io.crate.common.collections.Tuple in project crate by crate.

the class LinearizabilityChecker method isLinearizable.

private boolean isLinearizable(SequentialSpec spec, List<Event> history, BooleanSupplier terminateEarly) {
    LOGGER.debug("Checking history of size: {}: {}", history.size(), history);
    // the current state of the datatype
    Object state = spec.initialState();
    // the linearized prefix of the history
    final FixedBitSet linearized = new FixedBitSet(history.size() / 2);
    // cache of explored <state, linearized prefix> pairs
    final Cache cache = new Cache();
    // path we're currently exploring
    final Deque<Tuple<Entry, Object>> calls = new LinkedList<>();
    final Entry headEntry = createLinkedEntries(history);
    // current entry
    Entry entry = headEntry.next;
    while (headEntry.next != null) {
        if (terminateEarly.getAsBoolean()) {
            return false;
        }
        if (entry.match != null) {
            final Optional<Object> maybeNextState = spec.nextState(state, entry.event.value, entry.match.event.value);
            boolean shouldExploreNextState = false;
            if (maybeNextState.isPresent()) {
                // check if we have already explored this linearization
                final FixedBitSet updatedLinearized = linearized.clone();
                updatedLinearized.set(entry.id);
                shouldExploreNextState = cache.add(maybeNextState.get(), updatedLinearized);
            }
            if (shouldExploreNextState) {
                calls.push(new Tuple<>(entry, state));
                state = maybeNextState.get();
                linearized.set(entry.id);
                entry.lift();
                entry = headEntry.next;
            } else {
                entry = entry.next;
            }
        } else {
            if (calls.isEmpty()) {
                return false;
            }
            final Tuple<Entry, Object> top = calls.pop();
            entry = top.v1();
            state = top.v2();
            linearized.clear(entry.id);
            entry.unlift();
            entry = entry.next;
        }
    }
    return true;
}
Also used : FixedBitSet(org.apache.lucene.util.FixedBitSet) Tuple(io.crate.common.collections.Tuple) LinkedList(java.util.LinkedList)

Aggregations

Tuple (io.crate.common.collections.Tuple)42 ArrayList (java.util.ArrayList)15 List (java.util.List)15 IOException (java.io.IOException)13 Map (java.util.Map)12 Version (org.elasticsearch.Version)10 Collections (java.util.Collections)9 Collectors (java.util.stream.Collectors)9 Logger (org.apache.logging.log4j.Logger)9 DiscoveryNode (org.elasticsearch.cluster.node.DiscoveryNode)9 ESTestCase (org.elasticsearch.test.ESTestCase)9 HashMap (java.util.HashMap)8 Set (java.util.Set)8 TimeUnit (java.util.concurrent.TimeUnit)8 Settings (org.elasticsearch.common.settings.Settings)8 HashSet (java.util.HashSet)7 Consumer (java.util.function.Consumer)7 Nullable (javax.annotation.Nullable)7 LogManager (org.apache.logging.log4j.LogManager)7 IndexMetadata (org.elasticsearch.cluster.metadata.IndexMetadata)7