Search in sources :

Example 1 with Nullable

use of org.opensearch.common.Nullable in project OpenSearch by opensearch-project.

the class HdfsBlobContainer method listBlobsByPrefix.

@Override
public Map<String, BlobMetadata> listBlobsByPrefix(@Nullable final String prefix) throws IOException {
    FileStatus[] files = store.execute(fileContext -> fileContext.util().listStatus(path, path -> prefix == null || path.getName().startsWith(prefix)));
    Map<String, BlobMetadata> map = new LinkedHashMap<>();
    for (FileStatus file : files) {
        if (file.isFile()) {
            map.put(file.getPath().getName(), new PlainBlobMetadata(file.getPath().getName(), file.getLen()));
        }
    }
    return Collections.unmodifiableMap(map);
}
Also used : NoSuchFileException(java.nio.file.NoSuchFileException) Operation(org.opensearch.repositories.hdfs.HdfsBlobStore.Operation) BlobContainer(org.opensearch.common.blobstore.BlobContainer) FileStatus(org.apache.hadoop.fs.FileStatus) LinkedHashMap(java.util.LinkedHashMap) FSDataOutputStream(org.apache.hadoop.fs.FSDataOutputStream) Options(org.apache.hadoop.fs.Options) FilterInputStream(java.io.FilterInputStream) FileContext(org.apache.hadoop.fs.FileContext) Map(java.util.Map) Path(org.apache.hadoop.fs.Path) BlobMetadata(org.opensearch.common.blobstore.BlobMetadata) CreateFlag(org.apache.hadoop.fs.CreateFlag) EnumSet(java.util.EnumSet) IOException(java.io.IOException) FileAlreadyExistsException(java.nio.file.FileAlreadyExistsException) Nullable(org.opensearch.common.Nullable) FileNotFoundException(java.io.FileNotFoundException) CreateOpts(org.apache.hadoop.fs.Options.CreateOpts) List(java.util.List) DeleteResult(org.opensearch.common.blobstore.DeleteResult) BlobPath(org.opensearch.common.blobstore.BlobPath) PlainBlobMetadata(org.opensearch.common.blobstore.support.PlainBlobMetadata) FsBlobContainer(org.opensearch.common.blobstore.fs.FsBlobContainer) AbstractBlobContainer(org.opensearch.common.blobstore.support.AbstractBlobContainer) Collections(java.util.Collections) InputStream(java.io.InputStream) FileStatus(org.apache.hadoop.fs.FileStatus) BlobMetadata(org.opensearch.common.blobstore.BlobMetadata) PlainBlobMetadata(org.opensearch.common.blobstore.support.PlainBlobMetadata) PlainBlobMetadata(org.opensearch.common.blobstore.support.PlainBlobMetadata) LinkedHashMap(java.util.LinkedHashMap)

Example 2 with Nullable

use of org.opensearch.common.Nullable in project OpenSearch by opensearch-project.

the class TransportRolloverAction method evaluateConditions.

static Map<String, Boolean> evaluateConditions(final Collection<Condition<?>> conditions, @Nullable final DocsStats docsStats, @Nullable final IndexMetadata metadata) {
    if (metadata == null) {
        return conditions.stream().collect(Collectors.toMap(Condition::toString, cond -> false));
    }
    final long numDocs = docsStats == null ? 0 : docsStats.getCount();
    final long indexSize = docsStats == null ? 0 : docsStats.getTotalSizeInBytes();
    final Condition.Stats stats = new Condition.Stats(numDocs, metadata.getCreationDate(), new ByteSizeValue(indexSize));
    return conditions.stream().map(condition -> condition.evaluate(stats)).collect(Collectors.toMap(result -> result.condition.toString(), result -> result.matched));
}
Also used : Metadata(org.opensearch.cluster.metadata.Metadata) IndexMetadata(org.opensearch.cluster.metadata.IndexMetadata) ThreadPool(org.opensearch.threadpool.ThreadPool) ByteSizeValue(org.opensearch.common.unit.ByteSizeValue) TransportMasterNodeAction(org.opensearch.action.support.master.TransportMasterNodeAction) OpenSearchException(org.opensearch.OpenSearchException) IndicesOptions(org.opensearch.action.support.IndicesOptions) ClusterState(org.opensearch.cluster.ClusterState) Map(java.util.Map) Inject(org.opensearch.common.inject.Inject) ActionListener(org.opensearch.action.ActionListener) StreamInput(org.opensearch.common.io.stream.StreamInput) Client(org.opensearch.client.Client) ClusterBlockLevel(org.opensearch.cluster.block.ClusterBlockLevel) DocsStats(org.opensearch.index.shard.DocsStats) Collection(java.util.Collection) ClusterBlockException(org.opensearch.cluster.block.ClusterBlockException) IOException(java.io.IOException) IndicesStatsRequest(org.opensearch.action.admin.indices.stats.IndicesStatsRequest) Task(org.opensearch.tasks.Task) TransportService(org.opensearch.transport.TransportService) Collectors(java.util.stream.Collectors) Nullable(org.opensearch.common.Nullable) ActionFilters(org.opensearch.action.support.ActionFilters) List(java.util.List) ClusterStateUpdateTask(org.opensearch.cluster.ClusterStateUpdateTask) ActiveShardsObserver(org.opensearch.action.support.ActiveShardsObserver) IndicesStatsAction(org.opensearch.action.admin.indices.stats.IndicesStatsAction) ClusterService(org.opensearch.cluster.service.ClusterService) Optional(java.util.Optional) IndicesStatsResponse(org.opensearch.action.admin.indices.stats.IndicesStatsResponse) Collections(java.util.Collections) IndexNameExpressionResolver(org.opensearch.cluster.metadata.IndexNameExpressionResolver) DocsStats(org.opensearch.index.shard.DocsStats) ByteSizeValue(org.opensearch.common.unit.ByteSizeValue)

Example 3 with Nullable

use of org.opensearch.common.Nullable in project OpenSearch by opensearch-project.

the class RefreshListenersTests method setupListeners.

@Before
public void setupListeners() throws Exception {
    // Setup dependencies of the listeners
    maxListeners = randomIntBetween(1, 1000);
    // Now setup the InternalEngine which is much more complicated because we aren't mocking anything
    threadPool = new TestThreadPool(getTestName());
    refreshMetric = new MeanMetric();
    listeners = new RefreshListeners(() -> maxListeners, () -> engine.refresh("too-many-listeners"), logger, threadPool.getThreadContext(), refreshMetric);
    IndexSettings indexSettings = IndexSettingsModule.newIndexSettings("index", Settings.EMPTY);
    ShardId shardId = new ShardId(new Index("index", "_na_"), 1);
    String allocationId = UUIDs.randomBase64UUID(random());
    Directory directory = newDirectory();
    store = new Store(shardId, indexSettings, directory, new DummyShardLock(shardId));
    IndexWriterConfig iwc = newIndexWriterConfig();
    TranslogConfig translogConfig = new TranslogConfig(shardId, createTempDir("translog"), indexSettings, BigArrays.NON_RECYCLING_INSTANCE);
    Engine.EventListener eventListener = new Engine.EventListener() {

        @Override
        public void onFailedEngine(String reason, @Nullable Exception e) {
        // we don't need to notify anybody in this test
        }
    };
    store.createEmpty(Version.CURRENT.luceneVersion);
    final long primaryTerm = randomNonNegativeLong();
    final String translogUUID = Translog.createEmptyTranslog(translogConfig.getTranslogPath(), SequenceNumbers.NO_OPS_PERFORMED, shardId, primaryTerm);
    store.associateIndexWithNewTranslog(translogUUID);
    EngineConfig config = new EngineConfig(shardId, threadPool, indexSettings, null, store, newMergePolicy(), iwc.getAnalyzer(), iwc.getSimilarity(), new CodecService(null, logger), eventListener, IndexSearcher.getDefaultQueryCache(), IndexSearcher.getDefaultQueryCachingPolicy(), translogConfig, TimeValue.timeValueMinutes(5), Collections.singletonList(listeners), Collections.emptyList(), null, new NoneCircuitBreakerService(), () -> SequenceNumbers.NO_OPS_PERFORMED, () -> RetentionLeases.EMPTY, () -> primaryTerm, EngineTestCase.tombstoneDocSupplier());
    engine = new InternalEngine(config);
    engine.recoverFromTranslog((e, s) -> 0, Long.MAX_VALUE);
    listeners.setCurrentRefreshLocationSupplier(engine::getTranslogLastWriteLocation);
}
Also used : TranslogConfig(org.opensearch.index.translog.TranslogConfig) IndexSettings(org.opensearch.index.IndexSettings) Store(org.opensearch.index.store.Store) Index(org.opensearch.index.Index) TestThreadPool(org.opensearch.threadpool.TestThreadPool) IOException(java.io.IOException) InternalEngine(org.opensearch.index.engine.InternalEngine) MeanMetric(org.opensearch.common.metrics.MeanMetric) CodecService(org.opensearch.index.codec.CodecService) DummyShardLock(org.opensearch.test.DummyShardLock) EngineConfig(org.opensearch.index.engine.EngineConfig) InternalEngine(org.opensearch.index.engine.InternalEngine) Engine(org.opensearch.index.engine.Engine) Nullable(org.opensearch.common.Nullable) Directory(org.apache.lucene.store.Directory) IndexWriterConfig(org.apache.lucene.index.IndexWriterConfig) NoneCircuitBreakerService(org.opensearch.indices.breaker.NoneCircuitBreakerService) Before(org.junit.Before)

Example 4 with Nullable

use of org.opensearch.common.Nullable in project OpenSearch by opensearch-project.

the class SnapshotsService method shards.

/**
 * Calculates the assignment of shards to data nodes for a new snapshot based on the given cluster state and the
 * indices that should be included in the snapshot.
 *
 * @param indices             Indices to snapshot
 * @param useShardGenerations whether to write {@link ShardGenerations} during the snapshot
 * @return list of shard to be included into current snapshot
 */
private static ImmutableOpenMap<ShardId, SnapshotsInProgress.ShardSnapshotStatus> shards(SnapshotsInProgress snapshotsInProgress, @Nullable SnapshotDeletionsInProgress deletionsInProgress, Metadata metadata, RoutingTable routingTable, List<IndexId> indices, boolean useShardGenerations, RepositoryData repositoryData, String repoName) {
    ImmutableOpenMap.Builder<ShardId, SnapshotsInProgress.ShardSnapshotStatus> builder = ImmutableOpenMap.builder();
    final ShardGenerations shardGenerations = repositoryData.shardGenerations();
    final InFlightShardSnapshotStates inFlightShardStates = InFlightShardSnapshotStates.forRepo(repoName, snapshotsInProgress.entries());
    final boolean readyToExecute = deletionsInProgress == null || deletionsInProgress.getEntries().stream().noneMatch(entry -> entry.repository().equals(repoName) && entry.state() == SnapshotDeletionsInProgress.State.STARTED);
    for (IndexId index : indices) {
        final String indexName = index.getName();
        final boolean isNewIndex = repositoryData.getIndices().containsKey(indexName) == false;
        IndexMetadata indexMetadata = metadata.index(indexName);
        if (indexMetadata == null) {
            // The index was deleted before we managed to start the snapshot - mark it as missing.
            builder.put(new ShardId(indexName, IndexMetadata.INDEX_UUID_NA_VALUE, 0), ShardSnapshotStatus.MISSING);
        } else {
            final IndexRoutingTable indexRoutingTable = routingTable.index(indexName);
            for (int i = 0; i < indexMetadata.getNumberOfShards(); i++) {
                final ShardId shardId = indexRoutingTable.shard(i).shardId();
                final String shardRepoGeneration;
                if (useShardGenerations) {
                    final String inFlightGeneration = inFlightShardStates.generationForShard(index, shardId.id(), shardGenerations);
                    if (inFlightGeneration == null && isNewIndex) {
                        assert shardGenerations.getShardGen(index, shardId.getId()) == null : "Found shard generation for new index [" + index + "]";
                        shardRepoGeneration = ShardGenerations.NEW_SHARD_GEN;
                    } else {
                        shardRepoGeneration = inFlightGeneration;
                    }
                } else {
                    shardRepoGeneration = null;
                }
                final ShardSnapshotStatus shardSnapshotStatus;
                if (indexRoutingTable == null) {
                    shardSnapshotStatus = new SnapshotsInProgress.ShardSnapshotStatus(null, ShardState.MISSING, "missing routing table", shardRepoGeneration);
                } else {
                    ShardRouting primary = indexRoutingTable.shard(i).primaryShard();
                    if (readyToExecute == false || inFlightShardStates.isActive(indexName, i)) {
                        shardSnapshotStatus = ShardSnapshotStatus.UNASSIGNED_QUEUED;
                    } else if (primary == null || !primary.assignedToNode()) {
                        shardSnapshotStatus = new ShardSnapshotStatus(null, ShardState.MISSING, "primary shard is not allocated", shardRepoGeneration);
                    } else if (primary.relocating() || primary.initializing()) {
                        shardSnapshotStatus = new ShardSnapshotStatus(primary.currentNodeId(), ShardState.WAITING, shardRepoGeneration);
                    } else if (!primary.started()) {
                        shardSnapshotStatus = new ShardSnapshotStatus(primary.currentNodeId(), ShardState.MISSING, "primary shard hasn't been started yet", shardRepoGeneration);
                    } else {
                        shardSnapshotStatus = new ShardSnapshotStatus(primary.currentNodeId(), shardRepoGeneration);
                    }
                }
                builder.put(shardId, shardSnapshotStatus);
            }
        }
    }
    return builder.build();
}
Also used : RepositoryMissingException(org.opensearch.repositories.RepositoryMissingException) ImmutableOpenMap(org.opensearch.common.collect.ImmutableOpenMap) Arrays(java.util.Arrays) Metadata(org.opensearch.cluster.metadata.Metadata) Collections.unmodifiableList(java.util.Collections.unmodifiableList) DataStream(org.opensearch.cluster.metadata.DataStream) Version(org.opensearch.Version) ClusterStateApplier(org.opensearch.cluster.ClusterStateApplier) Regex(org.opensearch.common.regex.Regex) Strings(org.opensearch.common.Strings) GroupedActionListener(org.opensearch.action.support.GroupedActionListener) Map(java.util.Map) ActionListener(org.opensearch.action.ActionListener) EnumSet(java.util.EnumSet) Repository(org.opensearch.repositories.Repository) TimeValue(org.opensearch.common.unit.TimeValue) Index(org.opensearch.index.Index) ExceptionsHelper(org.opensearch.ExceptionsHelper) Set(java.util.Set) ClusterStateTaskExecutor(org.opensearch.cluster.ClusterStateTaskExecutor) Settings(org.opensearch.common.settings.Settings) ObjectCursor(com.carrotsearch.hppc.cursors.ObjectCursor) TransportService(org.opensearch.transport.TransportService) FailedToCommitClusterStateException(org.opensearch.cluster.coordination.FailedToCommitClusterStateException) ActionFilters(org.opensearch.action.support.ActionFilters) AbstractLifecycleComponent(org.opensearch.common.component.AbstractLifecycleComponent) ShardState(org.opensearch.cluster.SnapshotsInProgress.ShardState) Logger(org.apache.logging.log4j.Logger) Stream(java.util.stream.Stream) ClusterStateUpdateTask(org.opensearch.cluster.ClusterStateUpdateTask) StepListener(org.opensearch.action.StepListener) State(org.opensearch.cluster.SnapshotsInProgress.State) IndexNameExpressionResolver(org.opensearch.cluster.metadata.IndexNameExpressionResolver) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) RepositoriesService(org.opensearch.repositories.RepositoriesService) ThreadPool(org.opensearch.threadpool.ThreadPool) Priority(org.opensearch.common.Priority) TransportMasterNodeAction(org.opensearch.action.support.master.TransportMasterNodeAction) ArrayList(java.util.ArrayList) ClusterState(org.opensearch.cluster.ClusterState) LegacyESVersion(org.opensearch.LegacyESVersion) ClusterStateTaskConfig(org.opensearch.cluster.ClusterStateTaskConfig) RepositoryCleanupInProgress(org.opensearch.cluster.RepositoryCleanupInProgress) RepositoriesMetadata(org.opensearch.cluster.metadata.RepositoriesMetadata) Executor(java.util.concurrent.Executor) IOException(java.io.IOException) DeleteSnapshotRequest(org.opensearch.action.admin.cluster.snapshots.delete.DeleteSnapshotRequest) ClusterService(org.opensearch.cluster.service.ClusterService) RestoreInProgress(org.opensearch.cluster.RestoreInProgress) RoutingTable(org.opensearch.cluster.routing.RoutingTable) SnapshotsInProgress.completed(org.opensearch.cluster.SnapshotsInProgress.completed) ClusterChangedEvent(org.opensearch.cluster.ClusterChangedEvent) ShardGenerations(org.opensearch.repositories.ShardGenerations) AbstractRunnable(org.opensearch.common.util.concurrent.AbstractRunnable) ObjectObjectCursor(com.carrotsearch.hppc.cursors.ObjectObjectCursor) DiscoveryNode(org.opensearch.cluster.node.DiscoveryNode) IndexId(org.opensearch.repositories.IndexId) Locale(java.util.Locale) NotMasterException(org.opensearch.cluster.NotMasterException) ShardSnapshotStatus(org.opensearch.cluster.SnapshotsInProgress.ShardSnapshotStatus) RepositoryException(org.opensearch.repositories.RepositoryException) IndexShardRoutingTable(org.opensearch.cluster.routing.IndexShardRoutingTable) Collection(java.util.Collection) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) ClusterBlockException(org.opensearch.cluster.block.ClusterBlockException) Collectors(java.util.stream.Collectors) Nullable(org.opensearch.common.Nullable) Tuple(org.opensearch.common.collect.Tuple) Objects(java.util.Objects) List(java.util.List) Optional(java.util.Optional) ClusterStateTaskListener(org.opensearch.cluster.ClusterStateTaskListener) DiscoveryNodes(org.opensearch.cluster.node.DiscoveryNodes) IndexMetadata(org.opensearch.cluster.metadata.IndexMetadata) ActionRunnable(org.opensearch.action.ActionRunnable) SnapshotsInProgress(org.opensearch.cluster.SnapshotsInProgress) CloneSnapshotRequest(org.opensearch.action.admin.cluster.snapshots.clone.CloneSnapshotRequest) HashMap(java.util.HashMap) SnapshotDeletionsInProgress(org.opensearch.cluster.SnapshotDeletionsInProgress) Deque(java.util.Deque) ParameterizedMessage(org.apache.logging.log4j.message.ParameterizedMessage) Function(java.util.function.Function) HashSet(java.util.HashSet) IndexRoutingTable(org.opensearch.cluster.routing.IndexRoutingTable) UUIDs(org.opensearch.common.UUIDs) LinkedList(java.util.LinkedList) StreamInput(org.opensearch.common.io.stream.StreamInput) RepositoryData(org.opensearch.repositories.RepositoryData) Setting(org.opensearch.common.settings.Setting) Iterator(java.util.Iterator) Collections.emptySet(java.util.Collections.emptySet) RepositoryShardId(org.opensearch.repositories.RepositoryShardId) ShardRouting(org.opensearch.cluster.routing.ShardRouting) ShardId(org.opensearch.index.shard.ShardId) Consumer(java.util.function.Consumer) CreateSnapshotRequest(org.opensearch.action.admin.cluster.snapshots.create.CreateSnapshotRequest) LogManager(org.apache.logging.log4j.LogManager) Collections(java.util.Collections) IndexId(org.opensearch.repositories.IndexId) IndexRoutingTable(org.opensearch.cluster.routing.IndexRoutingTable) ShardSnapshotStatus(org.opensearch.cluster.SnapshotsInProgress.ShardSnapshotStatus) ShardGenerations(org.opensearch.repositories.ShardGenerations) ImmutableOpenMap(org.opensearch.common.collect.ImmutableOpenMap) RepositoryShardId(org.opensearch.repositories.RepositoryShardId) ShardId(org.opensearch.index.shard.ShardId) SnapshotsInProgress(org.opensearch.cluster.SnapshotsInProgress) ShardSnapshotStatus(org.opensearch.cluster.SnapshotsInProgress.ShardSnapshotStatus) IndexMetadata(org.opensearch.cluster.metadata.IndexMetadata) ShardRouting(org.opensearch.cluster.routing.ShardRouting)

Example 5 with Nullable

use of org.opensearch.common.Nullable in project OpenSearch by opensearch-project.

the class SourceFieldMapper method applyFilters.

@Nullable
public BytesReference applyFilters(@Nullable BytesReference originalSource, @Nullable XContentType contentType) throws IOException {
    if (enabled && originalSource != null) {
        // Percolate and tv APIs may not set the source and that is ok, because these APIs will not index any data
        if (filter != null) {
            // we don't update the context source if we filter, we want to keep it as is...
            Tuple<XContentType, Map<String, Object>> mapTuple = XContentHelper.convertToMap(originalSource, true, contentType);
            Map<String, Object> filteredSource = filter.apply(mapTuple.v2());
            BytesStreamOutput bStream = new BytesStreamOutput();
            XContentType actualContentType = mapTuple.v1();
            XContentBuilder builder = XContentFactory.contentBuilder(actualContentType, bStream).map(filteredSource);
            builder.close();
            return bStream.bytes();
        } else {
            return originalSource;
        }
    } else {
        return null;
    }
}
Also used : XContentType(org.opensearch.common.xcontent.XContentType) Map(java.util.Map) BytesStreamOutput(org.opensearch.common.io.stream.BytesStreamOutput) XContentBuilder(org.opensearch.common.xcontent.XContentBuilder) Nullable(org.opensearch.common.Nullable)

Aggregations

Nullable (org.opensearch.common.Nullable)16 IOException (java.io.IOException)7 ActionListener (org.opensearch.action.ActionListener)5 ClusterState (org.opensearch.cluster.ClusterState)5 ClusterService (org.opensearch.cluster.service.ClusterService)5 IndexSettings (org.opensearch.index.IndexSettings)5 ThreadPool (org.opensearch.threadpool.ThreadPool)5 Collections (java.util.Collections)4 Map (java.util.Map)4 Client (org.opensearch.client.Client)4 IndexMetadata (org.opensearch.cluster.metadata.IndexMetadata)4 TimeValue (org.opensearch.common.unit.TimeValue)4 ArrayList (java.util.ArrayList)3 List (java.util.List)3 FileNotFoundException (java.io.FileNotFoundException)2 FileAlreadyExistsException (java.nio.file.FileAlreadyExistsException)2 NoSuchFileException (java.nio.file.NoSuchFileException)2 Arrays (java.util.Arrays)2 Collection (java.util.Collection)2 EnumSet (java.util.EnumSet)2