Search in sources :

Example 6 with SearchResult

use of org.graylog2.indexer.results.SearchResult in project graylog2-server by Graylog2.

the class QuerySuggestionsES6 method suggest.

@Override
public SuggestionResponse suggest(SuggestionRequest req) {
    final Set<String> affectedIndices = indexLookup.indexNamesForStreamsInTimeRange(req.streams(), req.timerange());
    final SearchSourceBuilder search = new SearchSourceBuilder().query(QueryBuilders.prefixQuery(req.field(), req.input())).size(0).aggregation(AggregationBuilders.terms("fieldvalues").field(req.field()).size(req.size())).suggest(new SuggestBuilder().addSuggestion("corrections", SuggestBuilders.termSuggestion(req.field()).text(req.input()).size(req.size())));
    final Search.Builder searchBuilder = new Search.Builder(search.toString()).addType(IndexMapping.TYPE_MESSAGE).addIndex(affectedIndices.isEmpty() ? Collections.singleton("") : affectedIndices).allowNoIndices(false).ignoreUnavailable(false);
    try {
        final SearchResult result = JestUtils.execute(jestClient, searchBuilder.build(), () -> "Unable to perform aggregation: ");
        final TermsAggregation aggregation = result.getAggregations().getTermsAggregation("fieldvalues");
        final List<SuggestionEntry> entries = aggregation.getBuckets().stream().map(b -> new SuggestionEntry(b.getKeyAsString(), b.getCount())).collect(Collectors.toList());
        if (!entries.isEmpty()) {
            return SuggestionResponse.forSuggestions(req.field(), req.input(), entries, aggregation.getSumOtherDocCount());
        } else {
            final List<SuggestionEntry> corrections = Optional.of(result.getJsonObject()).map(o -> o.get("suggest")).map(o -> o.get("corrections")).map(o -> o.get(0)).map(o -> o.get("options")).map(options -> StreamSupport.stream(Spliterators.spliteratorUnknownSize(options.elements(), Spliterator.ORDERED), false).map(option -> new SuggestionEntry(option.get("text").textValue(), option.get("freq").longValue())).collect(Collectors.toList())).orElseGet(Collections::emptyList);
            return SuggestionResponse.forSuggestions(req.field(), req.input(), corrections, null);
        }
    } catch (Exception e) {
        final SuggestionError err = SuggestionError.create(e.getClass().getSimpleName(), e.getMessage());
        return SuggestionResponse.forError(req.field(), req.input(), err);
    }
}
Also used : TermsAggregation(io.searchbox.core.search.aggregation.TermsAggregation) SuggestBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.suggest.SuggestBuilder) SuggestionEntry(org.graylog.plugins.views.search.engine.suggestions.SuggestionEntry) Spliterators(java.util.Spliterators) JestClient(io.searchbox.client.JestClient) Inject(javax.inject.Inject) JestUtils(org.graylog.storage.elasticsearch6.jest.JestUtils) StreamSupport(java.util.stream.StreamSupport) SuggestionResponse(org.graylog.plugins.views.search.engine.suggestions.SuggestionResponse) IndexMapping(org.graylog2.indexer.IndexMapping) SuggestionError(org.graylog.plugins.views.search.engine.suggestions.SuggestionError) QueryBuilders(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.QueryBuilders) Search(io.searchbox.core.Search) Set(java.util.Set) SuggestBuilders(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.suggest.SuggestBuilders) SuggestionRequest(org.graylog.plugins.views.search.engine.suggestions.SuggestionRequest) Collectors(java.util.stream.Collectors) List(java.util.List) QuerySuggestionsService(org.graylog.plugins.views.search.engine.QuerySuggestionsService) AggregationBuilders(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.AggregationBuilders) Optional(java.util.Optional) IndexLookup(org.graylog.plugins.views.search.elasticsearch.IndexLookup) Collections(java.util.Collections) Spliterator(java.util.Spliterator) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder) SearchResult(io.searchbox.core.SearchResult) TermsAggregation(io.searchbox.core.search.aggregation.TermsAggregation) SuggestBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.suggest.SuggestBuilder) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder) SearchResult(io.searchbox.core.SearchResult) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder) SuggestBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.suggest.SuggestBuilder) SuggestionError(org.graylog.plugins.views.search.engine.suggestions.SuggestionError) Search(io.searchbox.core.Search) Collections(java.util.Collections) SuggestionEntry(org.graylog.plugins.views.search.engine.suggestions.SuggestionEntry)

Example 7 with SearchResult

use of org.graylog2.indexer.results.SearchResult in project graylog2-server by Graylog2.

the class IndicesAdapterES6 method move.

@Override
public void move(String source, String target, Consumer<IndexMoveResult> resultCallback) {
    // TODO: This method should use the Re-index API: https://www.elastic.co/guide/en/elasticsearch/reference/5.3/docs-reindex.html
    final String query = SearchSourceBuilder.searchSource().query(QueryBuilders.matchAllQuery()).size(350).sort(SortBuilders.fieldSort(FieldSortBuilder.DOC_FIELD_NAME)).toString();
    final Search request = new Search.Builder(query).setParameter(Parameters.SCROLL, "10s").addIndex(source).build();
    final SearchResult searchResult = JestUtils.execute(jestClient, request, () -> "Couldn't process search query response");
    final String scrollId = searchResult.getJsonObject().path("_scroll_id").asText(null);
    if (scrollId == null) {
        throw new ElasticsearchException("Couldn't find scroll ID in search query response");
    }
    while (true) {
        final SearchScroll scrollRequest = new SearchScroll.Builder(scrollId, "1m").build();
        final JestResult scrollResult = JestUtils.execute(jestClient, scrollRequest, () -> "Couldn't process result of scroll query");
        final JsonNode scrollHits = scrollResult.getJsonObject().path("hits").path("hits");
        // No more hits.
        if (scrollHits.size() == 0) {
            break;
        }
        final Bulk.Builder bulkRequestBuilder = new Bulk.Builder();
        for (JsonNode jsonElement : scrollHits) {
            Optional.ofNullable(jsonElement.path("_source")).map(sourceJson -> objectMapper.<Map<String, Object>>convertValue(sourceJson, TypeReferences.MAP_STRING_OBJECT)).ifPresent(doc -> {
                final String id = (String) doc.remove("_id");
                if (!Strings.isNullOrEmpty(id)) {
                    bulkRequestBuilder.addAction(indexingHelper.prepareIndexRequest(target, doc, id));
                }
            });
        }
        final BulkResult bulkResult = JestUtils.execute(jestClient, bulkRequestBuilder.build(), () -> "Couldn't bulk index messages into index " + target);
        final boolean hasFailedItems = !bulkResult.getFailedItems().isEmpty();
        final IndexMoveResult result = IndexMoveResult.create(bulkResult.getItems().size(), bulkResult.getJsonObject().path("took").asLong(), hasFailedItems);
        resultCallback.accept(result);
    }
}
Also used : TermsAggregation(io.searchbox.core.search.aggregation.TermsAggregation) DateTimeZone(org.joda.time.DateTimeZone) Arrays(java.util.Arrays) PutTemplate(io.searchbox.indices.template.PutTemplate) LoggerFactory(org.slf4j.LoggerFactory) ModifyAliases(io.searchbox.indices.aliases.ModifyAliases) RequestConfig(org.apache.http.client.config.RequestConfig) TypeReferences(org.graylog2.jackson.TypeReferences) UpdateSettings(io.searchbox.indices.settings.UpdateSettings) FieldSortBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.sort.FieldSortBuilder) MaxAggregation(io.searchbox.core.search.aggregation.MaxAggregation) IndicesAdapter(org.graylog2.indexer.indices.IndicesAdapter) HealthStatus(org.graylog2.indexer.indices.HealthStatus) JestUtils(org.graylog.storage.elasticsearch6.jest.JestUtils) Indices(org.graylog2.indexer.indices.Indices) Locale(java.util.Locale) Map(java.util.Map) JsonNode(com.fasterxml.jackson.databind.JsonNode) IndexRangeStats(org.graylog2.indexer.searches.IndexRangeStats) Bulk(io.searchbox.core.Bulk) Cat(io.searchbox.core.Cat) IndexMapping(org.graylog2.indexer.IndexMapping) FilterAggregation(io.searchbox.core.search.aggregation.FilterAggregation) QueryBuilders(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.QueryBuilders) SearchSourceBuilder.searchSource(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder.searchSource) ImmutableSet(com.google.common.collect.ImmutableSet) ImmutableMap(com.google.common.collect.ImmutableMap) Collection(java.util.Collection) Set(java.util.Set) Health(io.searchbox.cluster.Health) AddAliasMapping(io.searchbox.indices.aliases.AddAliasMapping) Collectors(java.util.stream.Collectors) StandardCharsets(java.nio.charset.StandardCharsets) GetAliases(io.searchbox.indices.aliases.GetAliases) RemoveAliasMapping(io.searchbox.indices.aliases.RemoveAliasMapping) PutMapping(io.searchbox.indices.mapping.PutMapping) DeleteIndex(io.searchbox.indices.DeleteIndex) Stats(io.searchbox.indices.Stats) List(java.util.List) Parameters(io.searchbox.params.Parameters) OpenIndex(io.searchbox.indices.OpenIndex) AggregationBuilders(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.AggregationBuilders) IndexMoveResult(org.graylog2.indexer.indices.IndexMoveResult) Optional(java.util.Optional) UnsupportedEncodingException(java.io.UnsupportedEncodingException) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder) SearchResult(io.searchbox.core.SearchResult) FilterAggregationBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder) HashMap(java.util.HashMap) ElasticsearchException(org.graylog2.indexer.ElasticsearchException) SearchType(io.searchbox.params.SearchType) Iterators(com.google.common.collect.Iterators) JestClient(io.searchbox.client.JestClient) Inject(javax.inject.Inject) Strings(com.google.common.base.Strings) SortBuilders(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.sort.SortBuilders) ImmutableList(com.google.common.collect.ImmutableList) Flush(io.searchbox.indices.Flush) DeleteTemplate(io.searchbox.indices.template.DeleteTemplate) IndexStatistics(org.graylog2.indexer.indices.stats.IndexStatistics) Duration(com.github.joschi.jadconfig.util.Duration) StreamSupport(java.util.stream.StreamSupport) Nonnull(javax.annotation.Nonnull) GetSettings(io.searchbox.indices.settings.GetSettings) Logger(org.slf4j.Logger) MinAggregation(io.searchbox.core.search.aggregation.MinAggregation) Iterator(java.util.Iterator) IndexSettings(org.graylog2.indexer.indices.IndexSettings) IndexNotFoundException(org.graylog2.indexer.IndexNotFoundException) SearchScroll(io.searchbox.core.SearchScroll) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) Search(io.searchbox.core.Search) AliasMapping(io.searchbox.indices.aliases.AliasMapping) DateTime(org.joda.time.DateTime) ForceMerge(io.searchbox.indices.ForceMerge) GetSingleAlias(org.graylog.storage.elasticsearch6.indices.GetSingleAlias) IOException(java.io.IOException) JestResult(io.searchbox.client.JestResult) Ints(com.google.common.primitives.Ints) CreateIndex(io.searchbox.indices.CreateIndex) BulkResult(io.searchbox.core.BulkResult) Consumer(java.util.function.Consumer) URLEncoder(java.net.URLEncoder) Collectors.toList(java.util.stream.Collectors.toList) CatResult(io.searchbox.core.CatResult) SerializationFeature(com.fasterxml.jackson.databind.SerializationFeature) CloseIndex(io.searchbox.indices.CloseIndex) GetTemplate(io.searchbox.indices.template.GetTemplate) Message(org.graylog2.plugin.Message) Collections(java.util.Collections) State(io.searchbox.cluster.State) FieldSortBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.sort.FieldSortBuilder) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder) FilterAggregationBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder) SearchResult(io.searchbox.core.SearchResult) JsonNode(com.fasterxml.jackson.databind.JsonNode) ElasticsearchException(org.graylog2.indexer.ElasticsearchException) IndexMoveResult(org.graylog2.indexer.indices.IndexMoveResult) Bulk(io.searchbox.core.Bulk) BulkResult(io.searchbox.core.BulkResult) SearchScroll(io.searchbox.core.SearchScroll) Search(io.searchbox.core.Search) Map(java.util.Map) ImmutableMap(com.google.common.collect.ImmutableMap) HashMap(java.util.HashMap) JestResult(io.searchbox.client.JestResult)

Example 8 with SearchResult

use of org.graylog2.indexer.results.SearchResult in project graylog2-server by Graylog2.

the class ESPivot method processSeries.

private void processSeries(PivotResult.Row.Builder rowBuilder, SearchResult searchResult, ESGeneratedQueryContext queryContext, Pivot pivot, ArrayDeque<String> columnKeys, MetricAggregation aggregation, boolean rollup, String source) {
    pivot.series().forEach(seriesSpec -> {
        final ESPivotSeriesSpecHandler<? extends SeriesSpec, ? extends Aggregation> seriesHandler = seriesHandlers.get(seriesSpec.type());
        final Aggregation series = seriesHandler.extractAggregationFromResult(pivot, seriesSpec, aggregation, queryContext);
        seriesHandler.handleResult(pivot, seriesSpec, searchResult, series, this, queryContext).map(value -> {
            columnKeys.addLast(value.id());
            final PivotResult.Value v = PivotResult.Value.create(columnKeys, value.value(), rollup, source);
            columnKeys.removeLast();
            return v;
        }).forEach(rowBuilder::addValue);
    });
}
Also used : Aggregation(io.searchbox.core.search.aggregation.Aggregation) MetricAggregation(io.searchbox.core.search.aggregation.MetricAggregation) ESGeneratedQueryContext(org.graylog.storage.elasticsearch6.views.ESGeneratedQueryContext) ESSearchTypeHandler(org.graylog.storage.elasticsearch6.views.searchtypes.ESSearchTypeHandler) MinAggregationBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.metrics.min.MinAggregationBuilder) DateTimeZone(org.joda.time.DateTimeZone) InvalidRangeParametersException(org.graylog2.plugin.indexer.searches.timeranges.InvalidRangeParametersException) Query(org.graylog.plugins.views.search.Query) PivotResult(org.graylog.plugins.views.search.searchtypes.pivot.PivotResult) LoggerFactory(org.slf4j.LoggerFactory) RelativeRange(org.graylog2.plugin.indexer.searches.timeranges.RelativeRange) Inject(javax.inject.Inject) Tuple2(org.jooq.lambda.tuple.Tuple2) ImmutableList(com.google.common.collect.ImmutableList) SearchType(org.graylog.plugins.views.search.SearchType) BucketSpec(org.graylog.plugins.views.search.searchtypes.pivot.BucketSpec) SeriesSpec(org.graylog.plugins.views.search.searchtypes.pivot.SeriesSpec) Map(java.util.Map) AbsoluteRange(org.graylog2.plugin.indexer.searches.timeranges.AbsoluteRange) MaxAggregationBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.metrics.max.MaxAggregationBuilder) AggregationBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.AggregationBuilder) Pivot(org.graylog.plugins.views.search.searchtypes.pivot.Pivot) TimeRange(org.graylog2.plugin.indexer.searches.timeranges.TimeRange) PivotSpec(org.graylog.plugins.views.search.searchtypes.pivot.PivotSpec) SearchJob(org.graylog.plugins.views.search.SearchJob) Logger(org.slf4j.Logger) IdentityHashMap(java.util.IdentityHashMap) Iterator(java.util.Iterator) DateTime(org.joda.time.DateTime) Aggregation(io.searchbox.core.search.aggregation.Aggregation) EntryStream(one.util.streamex.EntryStream) List(java.util.List) Tuple(org.jooq.lambda.tuple.Tuple) Stream(java.util.stream.Stream) AggregationBuilders(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.AggregationBuilders) MetricAggregation(io.searchbox.core.search.aggregation.MetricAggregation) Optional(java.util.Optional) Preconditions(com.google.common.base.Preconditions) ArrayDeque(java.util.ArrayDeque) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder) SearchResult(io.searchbox.core.SearchResult)

Example 9 with SearchResult

use of org.graylog2.indexer.results.SearchResult in project graylog2-server by Graylog2.

the class FieldContentValueAlertConditionTest method testCorrectUsageOfRelativeRange.

@Test
public void testCorrectUsageOfRelativeRange() throws Exception {
    final Stream stream = mock(Stream.class);
    final Searches searches = mock(Searches.class);
    final Configuration configuration = mock(Configuration.class);
    final SearchResult searchResult = mock(SearchResult.class);
    final int alertCheckInterval = 42;
    final RelativeRange relativeRange = RelativeRange.create(alertCheckInterval);
    when(stream.getId()).thenReturn("stream-id");
    when(configuration.getAlertCheckInterval()).thenReturn(alertCheckInterval);
    when(searches.search(anyString(), anyString(), eq(relativeRange), anyInt(), anyInt(), any(Sorting.class))).thenReturn(searchResult);
    final FieldContentValueAlertCondition alertCondition = new FieldContentValueAlertCondition(searches, configuration, stream, null, DateTime.now(DateTimeZone.UTC), "mockuser", ImmutableMap.<String, Object>of("field", "test", "value", "test"), "Field Content Value Test COndition");
    final AbstractAlertCondition.CheckResult result = alertCondition.runCheck();
}
Also used : Searches(org.graylog2.indexer.searches.Searches) Configuration(org.graylog2.Configuration) RelativeRange(org.graylog2.plugin.indexer.searches.timeranges.RelativeRange) Stream(org.graylog2.plugin.streams.Stream) SearchResult(org.graylog2.indexer.results.SearchResult) AbstractAlertCondition(org.graylog2.alerts.AbstractAlertCondition) Sorting(org.graylog2.indexer.searches.Sorting) Test(org.junit.Test) AlertConditionTest(org.graylog2.alerts.AlertConditionTest)

Example 10 with SearchResult

use of org.graylog2.indexer.results.SearchResult in project graylog2-server by Graylog2.

the class SearchesIT method searchReturnsCorrectTotalHits.

@Test
public void searchReturnsCorrectTotalHits() throws Exception {
    importFixture("org/graylog2/indexer/searches/SearchesIT.json");
    final AbsoluteRange range = AbsoluteRange.create(new DateTime(2015, 1, 1, 0, 0, DateTimeZone.UTC).withZone(UTC), new DateTime(2015, 1, 2, 0, 0, DateTimeZone.UTC).withZone(UTC));
    final SearchResult searchResult = searches.search("*", range, 5, 0, Sorting.DEFAULT);
    assertThat(searchResult).isNotNull();
    assertThat(searchResult.getResults()).hasSize(5);
    assertThat(searchResult.getTotalResults()).isEqualTo(10L);
    assertThat(searchResult.getFields()).doesNotContain("es_metadata_id", "es_metadata_version");
}
Also used : AbsoluteRange(org.graylog2.plugin.indexer.searches.timeranges.AbsoluteRange) SearchResult(org.graylog2.indexer.results.SearchResult) ZonedDateTime(java.time.ZonedDateTime) DateTime(org.joda.time.DateTime) ElasticsearchBaseTest(org.graylog.testing.elasticsearch.ElasticsearchBaseTest) Test(org.junit.Test)

Aggregations

SearchResult (org.graylog2.indexer.results.SearchResult)11 DateTime (org.joda.time.DateTime)11 ResultMessage (org.graylog2.indexer.results.ResultMessage)10 Inject (javax.inject.Inject)8 AbsoluteRange (org.graylog2.plugin.indexer.searches.timeranges.AbsoluteRange)8 List (java.util.List)7 Optional (java.util.Optional)7 SearchSourceBuilder (org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder)7 Sorting (org.graylog2.indexer.searches.Sorting)7 Message (org.graylog2.plugin.Message)7 Test (org.junit.Test)7 Search (io.searchbox.core.Search)6 SearchResult (io.searchbox.core.SearchResult)6 Map (java.util.Map)6 Set (java.util.Set)6 RelativeRange (org.graylog2.plugin.indexer.searches.timeranges.RelativeRange)6 Collectors (java.util.stream.Collectors)5 Logger (org.slf4j.Logger)5 LoggerFactory (org.slf4j.LoggerFactory)5 ImmutableList (com.google.common.collect.ImmutableList)4