Search in sources :

Example 1 with IndexMoveResult

use of org.graylog2.indexer.indices.IndexMoveResult in project graylog2-server by Graylog2.

the class IndicesAdapterES6 method move.

@Override
public void move(String source, String target, Consumer<IndexMoveResult> resultCallback) {
    // TODO: This method should use the Re-index API: https://www.elastic.co/guide/en/elasticsearch/reference/5.3/docs-reindex.html
    final String query = SearchSourceBuilder.searchSource().query(QueryBuilders.matchAllQuery()).size(350).sort(SortBuilders.fieldSort(FieldSortBuilder.DOC_FIELD_NAME)).toString();
    final Search request = new Search.Builder(query).setParameter(Parameters.SCROLL, "10s").addIndex(source).build();
    final SearchResult searchResult = JestUtils.execute(jestClient, request, () -> "Couldn't process search query response");
    final String scrollId = searchResult.getJsonObject().path("_scroll_id").asText(null);
    if (scrollId == null) {
        throw new ElasticsearchException("Couldn't find scroll ID in search query response");
    }
    while (true) {
        final SearchScroll scrollRequest = new SearchScroll.Builder(scrollId, "1m").build();
        final JestResult scrollResult = JestUtils.execute(jestClient, scrollRequest, () -> "Couldn't process result of scroll query");
        final JsonNode scrollHits = scrollResult.getJsonObject().path("hits").path("hits");
        // No more hits.
        if (scrollHits.size() == 0) {
            break;
        }
        final Bulk.Builder bulkRequestBuilder = new Bulk.Builder();
        for (JsonNode jsonElement : scrollHits) {
            Optional.ofNullable(jsonElement.path("_source")).map(sourceJson -> objectMapper.<Map<String, Object>>convertValue(sourceJson, TypeReferences.MAP_STRING_OBJECT)).ifPresent(doc -> {
                final String id = (String) doc.remove("_id");
                if (!Strings.isNullOrEmpty(id)) {
                    bulkRequestBuilder.addAction(indexingHelper.prepareIndexRequest(target, doc, id));
                }
            });
        }
        final BulkResult bulkResult = JestUtils.execute(jestClient, bulkRequestBuilder.build(), () -> "Couldn't bulk index messages into index " + target);
        final boolean hasFailedItems = !bulkResult.getFailedItems().isEmpty();
        final IndexMoveResult result = IndexMoveResult.create(bulkResult.getItems().size(), bulkResult.getJsonObject().path("took").asLong(), hasFailedItems);
        resultCallback.accept(result);
    }
}
Also used : TermsAggregation(io.searchbox.core.search.aggregation.TermsAggregation) DateTimeZone(org.joda.time.DateTimeZone) Arrays(java.util.Arrays) PutTemplate(io.searchbox.indices.template.PutTemplate) LoggerFactory(org.slf4j.LoggerFactory) ModifyAliases(io.searchbox.indices.aliases.ModifyAliases) RequestConfig(org.apache.http.client.config.RequestConfig) TypeReferences(org.graylog2.jackson.TypeReferences) UpdateSettings(io.searchbox.indices.settings.UpdateSettings) FieldSortBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.sort.FieldSortBuilder) MaxAggregation(io.searchbox.core.search.aggregation.MaxAggregation) IndicesAdapter(org.graylog2.indexer.indices.IndicesAdapter) HealthStatus(org.graylog2.indexer.indices.HealthStatus) JestUtils(org.graylog.storage.elasticsearch6.jest.JestUtils) Indices(org.graylog2.indexer.indices.Indices) Locale(java.util.Locale) Map(java.util.Map) JsonNode(com.fasterxml.jackson.databind.JsonNode) IndexRangeStats(org.graylog2.indexer.searches.IndexRangeStats) Bulk(io.searchbox.core.Bulk) Cat(io.searchbox.core.Cat) IndexMapping(org.graylog2.indexer.IndexMapping) FilterAggregation(io.searchbox.core.search.aggregation.FilterAggregation) QueryBuilders(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.QueryBuilders) SearchSourceBuilder.searchSource(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder.searchSource) ImmutableSet(com.google.common.collect.ImmutableSet) ImmutableMap(com.google.common.collect.ImmutableMap) Collection(java.util.Collection) Set(java.util.Set) Health(io.searchbox.cluster.Health) AddAliasMapping(io.searchbox.indices.aliases.AddAliasMapping) Collectors(java.util.stream.Collectors) StandardCharsets(java.nio.charset.StandardCharsets) GetAliases(io.searchbox.indices.aliases.GetAliases) RemoveAliasMapping(io.searchbox.indices.aliases.RemoveAliasMapping) PutMapping(io.searchbox.indices.mapping.PutMapping) DeleteIndex(io.searchbox.indices.DeleteIndex) Stats(io.searchbox.indices.Stats) List(java.util.List) Parameters(io.searchbox.params.Parameters) OpenIndex(io.searchbox.indices.OpenIndex) AggregationBuilders(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.AggregationBuilders) IndexMoveResult(org.graylog2.indexer.indices.IndexMoveResult) Optional(java.util.Optional) UnsupportedEncodingException(java.io.UnsupportedEncodingException) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder) SearchResult(io.searchbox.core.SearchResult) FilterAggregationBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder) HashMap(java.util.HashMap) ElasticsearchException(org.graylog2.indexer.ElasticsearchException) SearchType(io.searchbox.params.SearchType) Iterators(com.google.common.collect.Iterators) JestClient(io.searchbox.client.JestClient) Inject(javax.inject.Inject) Strings(com.google.common.base.Strings) SortBuilders(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.sort.SortBuilders) ImmutableList(com.google.common.collect.ImmutableList) Flush(io.searchbox.indices.Flush) DeleteTemplate(io.searchbox.indices.template.DeleteTemplate) IndexStatistics(org.graylog2.indexer.indices.stats.IndexStatistics) Duration(com.github.joschi.jadconfig.util.Duration) StreamSupport(java.util.stream.StreamSupport) Nonnull(javax.annotation.Nonnull) GetSettings(io.searchbox.indices.settings.GetSettings) Logger(org.slf4j.Logger) MinAggregation(io.searchbox.core.search.aggregation.MinAggregation) Iterator(java.util.Iterator) IndexSettings(org.graylog2.indexer.indices.IndexSettings) IndexNotFoundException(org.graylog2.indexer.IndexNotFoundException) SearchScroll(io.searchbox.core.SearchScroll) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) Search(io.searchbox.core.Search) AliasMapping(io.searchbox.indices.aliases.AliasMapping) DateTime(org.joda.time.DateTime) ForceMerge(io.searchbox.indices.ForceMerge) GetSingleAlias(org.graylog.storage.elasticsearch6.indices.GetSingleAlias) IOException(java.io.IOException) JestResult(io.searchbox.client.JestResult) Ints(com.google.common.primitives.Ints) CreateIndex(io.searchbox.indices.CreateIndex) BulkResult(io.searchbox.core.BulkResult) Consumer(java.util.function.Consumer) URLEncoder(java.net.URLEncoder) Collectors.toList(java.util.stream.Collectors.toList) CatResult(io.searchbox.core.CatResult) SerializationFeature(com.fasterxml.jackson.databind.SerializationFeature) CloseIndex(io.searchbox.indices.CloseIndex) GetTemplate(io.searchbox.indices.template.GetTemplate) Message(org.graylog2.plugin.Message) Collections(java.util.Collections) State(io.searchbox.cluster.State) FieldSortBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.sort.FieldSortBuilder) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder) FilterAggregationBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder) SearchResult(io.searchbox.core.SearchResult) JsonNode(com.fasterxml.jackson.databind.JsonNode) ElasticsearchException(org.graylog2.indexer.ElasticsearchException) IndexMoveResult(org.graylog2.indexer.indices.IndexMoveResult) Bulk(io.searchbox.core.Bulk) BulkResult(io.searchbox.core.BulkResult) SearchScroll(io.searchbox.core.SearchScroll) Search(io.searchbox.core.Search) Map(java.util.Map) ImmutableMap(com.google.common.collect.ImmutableMap) HashMap(java.util.HashMap) JestResult(io.searchbox.client.JestResult)

Example 2 with IndexMoveResult

use of org.graylog2.indexer.indices.IndexMoveResult in project graylog2-server by Graylog2.

the class IndicesAdapterES7 method move.

@Override
public void move(String source, String target, Consumer<IndexMoveResult> resultCallback) {
    final ReindexRequest request = new ReindexRequest();
    request.setSourceIndices(source);
    request.setDestIndex(target);
    final BulkByScrollResponse result = client.execute((c, requestOptions) -> c.reindex(request, requestOptions));
    final IndexMoveResult indexMoveResult = IndexMoveResult.create(Math.toIntExact(result.getTotal()), result.getTook().millis(), !result.getBulkFailures().isEmpty());
    resultCallback.accept(indexMoveResult);
}
Also used : ReindexRequest(org.graylog.shaded.elasticsearch7.org.elasticsearch.index.reindex.ReindexRequest) IndexMoveResult(org.graylog2.indexer.indices.IndexMoveResult) BulkByScrollResponse(org.graylog.shaded.elasticsearch7.org.elasticsearch.index.reindex.BulkByScrollResponse)

Aggregations

JsonNode (com.fasterxml.jackson.databind.JsonNode)1 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)1 SerializationFeature (com.fasterxml.jackson.databind.SerializationFeature)1 Duration (com.github.joschi.jadconfig.util.Duration)1 Strings (com.google.common.base.Strings)1 ImmutableList (com.google.common.collect.ImmutableList)1 ImmutableMap (com.google.common.collect.ImmutableMap)1 ImmutableSet (com.google.common.collect.ImmutableSet)1 Iterators (com.google.common.collect.Iterators)1 Ints (com.google.common.primitives.Ints)1 JestClient (io.searchbox.client.JestClient)1 JestResult (io.searchbox.client.JestResult)1 Health (io.searchbox.cluster.Health)1 State (io.searchbox.cluster.State)1 Bulk (io.searchbox.core.Bulk)1 BulkResult (io.searchbox.core.BulkResult)1 Cat (io.searchbox.core.Cat)1 CatResult (io.searchbox.core.CatResult)1 Search (io.searchbox.core.Search)1 SearchResult (io.searchbox.core.SearchResult)1