Search in sources :

Example 6 with SearchesConfig

use of org.graylog2.indexer.searches.SearchesConfig in project graylog2-server by Graylog2.

the class Searches method search.

public SearchResult search(SearchesConfig config) {
    final Set<IndexRange> indexRanges = determineAffectedIndicesWithRanges(config.range(), config.filter());
    final Set<String> indices = extractIndexNamesFromIndexRanges(indexRanges);
    final SearchResult result = searchesAdapter.search(indices, indexRanges, config);
    recordEsMetrics(result.tookMs(), config.range());
    return result;
}
Also used : IndexRange(org.graylog2.indexer.ranges.IndexRange) SearchResult(org.graylog2.indexer.results.SearchResult)

Example 7 with SearchesConfig

use of org.graylog2.indexer.searches.SearchesConfig in project graylog2-server by Graylog2.

the class SearchesAdapterES6 method search.

@Override
public SearchResult search(Set<String> indices, Set<IndexRange> indexRanges, SearchesConfig config) {
    final SearchSourceBuilder requestBuilder = searchRequest(config);
    if (indexRanges.isEmpty()) {
        return SearchResult.empty(config.query(), requestBuilder.toString());
    }
    final Search.Builder searchBuilder = new Search.Builder(requestBuilder.toString()).addType(IndexMapping.TYPE_MESSAGE).addIndex(indices);
    final io.searchbox.core.SearchResult searchResult = multiSearch.wrap(searchBuilder.build(), () -> "Unable to perform search query");
    final List<ResultMessage> hits = searchResult.getHits(Map.class, false).stream().map(hit -> ResultMessage.parseFromSource(hit.id, hit.index, (Map<String, Object>) hit.source, hit.highlight)).collect(Collectors.toList());
    return new SearchResult(hits, searchResult.getTotal(), indexRanges, config.query(), requestBuilder.toString(), multiSearch.tookMsFromSearchResult(searchResult));
}
Also used : FilterAggregationBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder) NO_LIMIT(org.graylog2.indexer.searches.ScrollCommand.NO_LIMIT) FieldStatsResult(org.graylog2.indexer.results.FieldStatsResult) SearchesConfig(org.graylog2.indexer.searches.SearchesConfig) Strings.isNullOrEmpty(com.google.common.base.Strings.isNullOrEmpty) ScrollResult(org.graylog2.indexer.results.ScrollResult) SearchesAdapter(org.graylog2.indexer.searches.SearchesAdapter) QueryBuilders.existsQuery(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.QueryBuilders.existsQuery) Inject(javax.inject.Inject) ScrollCommand(org.graylog2.indexer.searches.ScrollCommand) ResultMessage(org.graylog2.indexer.results.ResultMessage) Map(java.util.Map) CardinalityAggregation(io.searchbox.core.search.aggregation.CardinalityAggregation) IndexMapping(org.graylog2.indexer.IndexMapping) SearchResult(org.graylog2.indexer.results.SearchResult) TimeRange(org.graylog2.plugin.indexer.searches.timeranges.TimeRange) QueryBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.QueryBuilder) Nullable(javax.annotation.Nullable) QueryBuilders(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.QueryBuilders) HighlightBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder) Sorting(org.graylog2.indexer.searches.Sorting) NO_BATCHSIZE(org.graylog2.indexer.searches.ScrollCommand.NO_BATCHSIZE) Search(io.searchbox.core.Search) Set(java.util.Set) QueryBuilders.matchAllQuery(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.QueryBuilders.matchAllQuery) QueryBuilders.termsQuery(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.QueryBuilders.termsQuery) ValueCountAggregation(io.searchbox.core.search.aggregation.ValueCountAggregation) Collectors(java.util.stream.Collectors) ExtendedStatsAggregation(io.searchbox.core.search.aggregation.ExtendedStatsAggregation) Strings(org.graylog.shaded.elasticsearch6.org.elasticsearch.common.Strings) IndexRange(org.graylog2.indexer.ranges.IndexRange) CountResult(org.graylog2.indexer.results.CountResult) QueryBuilders.queryStringQuery(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.QueryBuilders.queryStringQuery) List(java.util.List) Parameters(io.searchbox.params.Parameters) Configuration(org.graylog2.Configuration) Stream(org.graylog2.plugin.streams.Stream) AggregationBuilders(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.AggregationBuilders) Optional(java.util.Optional) BoolQueryBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.BoolQueryBuilder) VisibleForTesting(com.google.common.annotations.VisibleForTesting) Message(org.graylog2.plugin.Message) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder) QueryBuilders.boolQuery(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.QueryBuilders.boolQuery) FilterAggregationBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder) QueryBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.QueryBuilder) HighlightBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder) BoolQueryBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.BoolQueryBuilder) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder) SearchResult(org.graylog2.indexer.results.SearchResult) ResultMessage(org.graylog2.indexer.results.ResultMessage) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder) Search(io.searchbox.core.Search) Map(java.util.Map)

Example 8 with SearchesConfig

use of org.graylog2.indexer.searches.SearchesConfig in project graylog2-server by Graylog2.

the class SearchesAdapterES7 method search.

@Override
public SearchResult search(Set<String> indices, Set<IndexRange> indexRanges, SearchesConfig config) {
    final SearchSourceBuilder searchSourceBuilder = searchRequestFactory.create(config);
    if (indexRanges.isEmpty()) {
        return SearchResult.empty(config.query(), searchSourceBuilder.toString());
    }
    final SearchRequest searchRequest = new SearchRequest(indices.toArray(new String[0])).source(searchSourceBuilder);
    final SearchResponse searchResult = client.search(searchRequest, "Unable to perform search query");
    final List<ResultMessage> resultMessages = extractResultMessages(searchResult);
    final long totalResults = searchResult.getHits().getTotalHits().value;
    final long tookMs = searchResult.getTook().getMillis();
    final String builtQuery = searchSourceBuilder.toString();
    return new SearchResult(resultMessages, totalResults, indexRanges, config.query(), builtQuery, tookMs);
}
Also used : SearchRequest(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.SearchRequest) SearchResult(org.graylog2.indexer.results.SearchResult) ResultMessage(org.graylog2.indexer.results.ResultMessage) SearchSourceBuilder(org.graylog.shaded.elasticsearch7.org.elasticsearch.search.builder.SearchSourceBuilder) SearchResponse(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.SearchResponse)

Example 9 with SearchesConfig

use of org.graylog2.indexer.searches.SearchesConfig in project graylog2-server by Graylog2.

the class SearchesAdapterES7 method fieldStats.

@Override
public FieldStatsResult fieldStats(String query, String filter, TimeRange range, Set<String> indices, String field, boolean includeCardinality, boolean includeStats, boolean includeCount) {
    final SearchesConfig config = SearchesConfig.builder().query(query).filter(filter).range(range).offset(0).limit(-1).build();
    final SearchSourceBuilder searchSourceBuilder = searchRequestFactory.create(config);
    if (includeCount) {
        searchSourceBuilder.aggregation(AggregationBuilders.count(AGG_VALUE_COUNT).field(field));
    }
    if (includeStats) {
        searchSourceBuilder.aggregation(AggregationBuilders.extendedStats(AGG_EXTENDED_STATS).field(field));
    }
    if (includeCardinality) {
        searchSourceBuilder.aggregation(AggregationBuilders.cardinality(AGG_CARDINALITY).field(field));
    }
    if (indices.isEmpty()) {
        return FieldStatsResult.empty(query, searchSourceBuilder.toString());
    }
    final SearchRequest searchRequest = new SearchRequest(indices.toArray(new String[0])).source(searchSourceBuilder);
    final SearchResponse searchResult = client.search(searchRequest, "Unable to retrieve fields stats");
    final List<ResultMessage> resultMessages = extractResultMessages(searchResult);
    final long tookMs = searchResult.getTook().getMillis();
    final ExtendedStats extendedStatsAggregation = searchResult.getAggregations().get(AGG_EXTENDED_STATS);
    final ValueCount valueCountAggregation = searchResult.getAggregations().get(AGG_VALUE_COUNT);
    final Cardinality cardinalityAggregation = searchResult.getAggregations().get(AGG_CARDINALITY);
    return createFieldStatsResult(extendedStatsAggregation, valueCountAggregation, cardinalityAggregation, resultMessages, query, searchSourceBuilder.toString(), tookMs);
}
Also used : SearchRequest(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.SearchRequest) ValueCount(org.graylog.shaded.elasticsearch7.org.elasticsearch.search.aggregations.metrics.ValueCount) SearchesConfig(org.graylog2.indexer.searches.SearchesConfig) Cardinality(org.graylog.shaded.elasticsearch7.org.elasticsearch.search.aggregations.metrics.Cardinality) ExtendedStats(org.graylog.shaded.elasticsearch7.org.elasticsearch.search.aggregations.metrics.ExtendedStats) ResultMessage(org.graylog2.indexer.results.ResultMessage) SearchSourceBuilder(org.graylog.shaded.elasticsearch7.org.elasticsearch.search.builder.SearchSourceBuilder) SearchResponse(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.SearchResponse)

Aggregations

SearchesConfig (org.graylog2.indexer.searches.SearchesConfig)6 SearchResult (org.graylog2.indexer.results.SearchResult)4 Sorting (org.graylog2.indexer.searches.Sorting)4 TimeRange (org.graylog2.plugin.indexer.searches.timeranges.TimeRange)4 Timed (com.codahale.metrics.annotation.Timed)3 ApiOperation (io.swagger.annotations.ApiOperation)3 ApiResponses (io.swagger.annotations.ApiResponses)3 GET (javax.ws.rs.GET)3 Produces (javax.ws.rs.Produces)3 SearchPhaseExecutionException (org.elasticsearch.action.search.SearchPhaseExecutionException)3 SearchRequest (org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.SearchRequest)3 SearchResponse (org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.SearchResponse)3 SearchSourceBuilder (org.graylog.shaded.elasticsearch7.org.elasticsearch.search.builder.SearchSourceBuilder)3 ResultMessage (org.graylog2.indexer.results.ResultMessage)3 IndexRange (org.graylog2.indexer.ranges.IndexRange)2 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 Strings.isNullOrEmpty (com.google.common.base.Strings.isNullOrEmpty)1 Search (io.searchbox.core.Search)1 CardinalityAggregation (io.searchbox.core.search.aggregation.CardinalityAggregation)1 ExtendedStatsAggregation (io.searchbox.core.search.aggregation.ExtendedStatsAggregation)1