Search in sources :

Example 86 with BoolQueryBuilder

use of org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.BoolQueryBuilder in project graylog2-server by Graylog2.

the class ElasticsearchBackend method generateFilterClause.

// TODO make pluggable
public Optional<QueryBuilder> generateFilterClause(Filter filter, SearchJob job, Query query) {
    if (filter == null) {
        return Optional.empty();
    }
    switch(filter.type()) {
        case AndFilter.NAME:
            final BoolQueryBuilder andBuilder = QueryBuilders.boolQuery();
            filter.filters().stream().map(filter1 -> generateFilterClause(filter1, job, query)).forEach(optQueryBuilder -> optQueryBuilder.ifPresent(andBuilder::must));
            return Optional.of(andBuilder);
        case OrFilter.NAME:
            final BoolQueryBuilder orBuilder = QueryBuilders.boolQuery();
            // TODO for the common case "any of these streams" we can optimize the filter into
            // a single "termsQuery" instead of "termQuery OR termQuery" if all direct children are "StreamFilter"
            filter.filters().stream().map(filter1 -> generateFilterClause(filter1, job, query)).forEach(optQueryBuilder -> optQueryBuilder.ifPresent(orBuilder::should));
            return Optional.of(orBuilder);
        case StreamFilter.NAME:
            // Skipping stream filter, will be extracted elsewhere
            return Optional.empty();
        case QueryStringFilter.NAME:
            return Optional.of(QueryBuilders.queryStringQuery(this.queryStringDecorators.decorate(((QueryStringFilter) filter).query(), job, query)));
    }
    return Optional.empty();
}
Also used : ESSearchTypeHandler(org.graylog.storage.elasticsearch7.views.searchtypes.ESSearchTypeHandler) AndFilter(org.graylog.plugins.views.search.filter.AndFilter) ElasticsearchClient(org.graylog.storage.elasticsearch7.ElasticsearchClient) Arrays(java.util.Arrays) BackendQuery(org.graylog.plugins.views.search.engine.BackendQuery) QueryBackend(org.graylog.plugins.views.search.engine.QueryBackend) Provider(javax.inject.Provider) LoggerFactory(org.slf4j.LoggerFactory) FieldTypeException(org.graylog2.indexer.FieldTypeException) MultiSearchResponse(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.MultiSearchResponse) StreamFilter(org.graylog.plugins.views.search.filter.StreamFilter) Map(java.util.Map) IndicesOptions(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.support.IndicesOptions) SearchConfig(org.graylog.plugins.views.search.engine.SearchConfig) Set(java.util.Set) Collectors(java.util.stream.Collectors) BoolQueryBuilder(org.graylog.shaded.elasticsearch7.org.elasticsearch.index.query.BoolQueryBuilder) Objects(java.util.Objects) List(java.util.List) Filter(org.graylog.plugins.views.search.Filter) Optional(java.util.Optional) SearchResponse(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.SearchResponse) Query(org.graylog.plugins.views.search.Query) SearchTypeErrorParser(org.graylog.plugins.views.search.errors.SearchTypeErrorParser) HashMap(java.util.HashMap) ElasticsearchException(org.graylog2.indexer.ElasticsearchException) ShardOperationFailedException(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.ShardOperationFailedException) ArrayList(java.util.ArrayList) GlobalOverride(org.graylog.plugins.views.search.GlobalOverride) Inject(javax.inject.Inject) HashSet(java.util.HashSet) OrFilter(org.graylog.plugins.views.search.filter.OrFilter) SearchType(org.graylog.plugins.views.search.SearchType) QueryStringFilter(org.graylog.plugins.views.search.filter.QueryStringFilter) SearchTypeError(org.graylog.plugins.views.search.errors.SearchTypeError) QueryResult(org.graylog.plugins.views.search.QueryResult) TimeRangeQueryFactory(org.graylog.storage.elasticsearch7.TimeRangeQueryFactory) SearchJob(org.graylog.plugins.views.search.SearchJob) QueryBuilder(org.graylog.shaded.elasticsearch7.org.elasticsearch.index.query.QueryBuilder) SearchSourceBuilder(org.graylog.shaded.elasticsearch7.org.elasticsearch.search.builder.SearchSourceBuilder) Logger(org.slf4j.Logger) SearchRequest(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.SearchRequest) Maps(com.google.common.collect.Maps) QueryStringDecorators(org.graylog.plugins.views.search.elasticsearch.QueryStringDecorators) Named(com.google.inject.name.Named) IndexLookup(org.graylog.plugins.views.search.elasticsearch.IndexLookup) QueryBuilders(org.graylog.shaded.elasticsearch7.org.elasticsearch.index.query.QueryBuilders) Message(org.graylog2.plugin.Message) Collections(java.util.Collections) BoolQueryBuilder(org.graylog.shaded.elasticsearch7.org.elasticsearch.index.query.BoolQueryBuilder)

Example 87 with BoolQueryBuilder

use of org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.BoolQueryBuilder in project graylog2-server by Graylog2.

the class SearchRequestFactory method buildStreamIdFilter.

private BoolQueryBuilder buildStreamIdFilter(Set<String> streams) {
    final BoolQueryBuilder filterBuilder = boolQuery();
    // exist in Graylog <2.2.0.
    if (streams.contains(Stream.DEFAULT_STREAM_ID)) {
        filterBuilder.should(boolQuery().mustNot(existsQuery(Message.FIELD_STREAMS)));
    }
    // Only select messages which are assigned to the given streams
    filterBuilder.should(termsQuery(Message.FIELD_STREAMS, streams));
    return filterBuilder;
}
Also used : BoolQueryBuilder(org.graylog.shaded.elasticsearch7.org.elasticsearch.index.query.BoolQueryBuilder)

Example 88 with BoolQueryBuilder

use of org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.BoolQueryBuilder in project graylog2-server by Graylog2.

the class IndexToolsAdapterES6 method fieldHistogram.

@Override
public Map<DateTime, Map<String, Long>> fieldHistogram(String fieldName, Set<String> indices, Optional<Set<String>> includedStreams, long interval) {
    final BoolQueryBuilder queryBuilder = buildStreamIdFilter(includedStreams);
    final FilterAggregationBuilder the_filter = AggregationBuilders.filter(AGG_FILTER, queryBuilder).subAggregation(AggregationBuilders.dateHistogram(AGG_DATE_HISTOGRAM).field("timestamp").subAggregation(AggregationBuilders.terms(AGG_MESSAGE_FIELD).field(fieldName)).interval(interval).minDocCount(1L));
    final SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder().query(QueryBuilders.matchAllQuery()).aggregation(the_filter);
    final Search.Builder searchBuilder = new Search.Builder(searchSourceBuilder.toString()).addIndex(indices).addType(IndexMapping.TYPE_MESSAGE);
    final SearchResult searchResult = JestUtils.execute(this.jestClient, searchBuilder.build(), () -> "Unable to retrieve field histogram.");
    final FilterAggregation filterAggregation = searchResult.getAggregations().getFilterAggregation(AGG_FILTER);
    final DateHistogramAggregation dateHistogram = filterAggregation.getDateHistogramAggregation(AGG_DATE_HISTOGRAM);
    final List<DateHistogramAggregation.DateHistogram> histogramBuckets = dateHistogram.getBuckets();
    final Map<DateTime, Map<String, Long>> result = Maps.newHashMapWithExpectedSize(histogramBuckets.size());
    for (HistogramAggregation.Histogram bucket : histogramBuckets) {
        final DateTime date = new DateTime(bucket.getKey()).toDateTime(DateTimeZone.UTC);
        final TermsAggregation sourceFieldAgg = bucket.getTermsAggregation(AGG_MESSAGE_FIELD);
        final List<TermsAggregation.Entry> termBuckets = sourceFieldAgg.getBuckets();
        final HashMap<String, Long> termCounts = Maps.newHashMapWithExpectedSize(termBuckets.size());
        for (TermsAggregation.Entry termBucket : termBuckets) {
            termCounts.put(termBucket.getKeyAsString(), termBucket.getCount());
        }
        result.put(date, termCounts);
    }
    return ImmutableMap.copyOf(result);
}
Also used : TermsAggregation(io.searchbox.core.search.aggregation.TermsAggregation) FilterAggregationBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder) FilterAggregationBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder) BoolQueryBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.BoolQueryBuilder) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder) SearchResult(io.searchbox.core.SearchResult) DateTime(org.joda.time.DateTime) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder) HistogramAggregation(io.searchbox.core.search.aggregation.HistogramAggregation) DateHistogramAggregation(io.searchbox.core.search.aggregation.DateHistogramAggregation) BoolQueryBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.BoolQueryBuilder) Search(io.searchbox.core.Search) FilterAggregation(io.searchbox.core.search.aggregation.FilterAggregation) HashMap(java.util.HashMap) Map(java.util.Map) ImmutableMap(com.google.common.collect.ImmutableMap) DateHistogramAggregation(io.searchbox.core.search.aggregation.DateHistogramAggregation)

Example 89 with BoolQueryBuilder

use of org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.BoolQueryBuilder in project graylog2-server by Graylog2.

the class SearchesAdapterES6 method buildSearchRequest.

private SearchSourceBuilder buildSearchRequest(ScrollCommand scrollCommand) {
    final String query = normalizeQuery(scrollCommand.query());
    final QueryBuilder queryBuilder = isWildcardQuery(query) ? matchAllQuery() : queryStringQuery(query).allowLeadingWildcard(configuration.isAllowLeadingWildcardSearches());
    final Optional<BoolQueryBuilder> rangeQueryBuilder = scrollCommand.range().map(range -> QueryBuilders.boolQuery().must(TimeRangeQueryFactory.create(range)));
    final Optional<BoolQueryBuilder> filterQueryBuilder = scrollCommand.filter().filter(filter -> !isWildcardQuery(filter)).map(QueryBuilders::queryStringQuery).map(filter -> rangeQueryBuilder.orElse(QueryBuilders.boolQuery()).must(filter));
    final BoolQueryBuilder filteredQueryBuilder = QueryBuilders.boolQuery().must(queryBuilder);
    filterQueryBuilder.ifPresent(filteredQueryBuilder::filter);
    applyStreamsFilter(filteredQueryBuilder, scrollCommand);
    final SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder().query(filteredQueryBuilder);
    applyPaginationIfPresent(searchSourceBuilder, scrollCommand);
    applySortingIfPresent(searchSourceBuilder, scrollCommand);
    applyHighlighting(searchSourceBuilder, scrollCommand);
    return searchSourceBuilder;
}
Also used : BoolQueryBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.BoolQueryBuilder) QueryBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.QueryBuilder) BoolQueryBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.BoolQueryBuilder) SearchSourceBuilder(org.graylog.shaded.elasticsearch6.org.elasticsearch.search.builder.SearchSourceBuilder)

Example 90 with BoolQueryBuilder

use of org.graylog.shaded.elasticsearch6.org.elasticsearch.index.query.BoolQueryBuilder in project yacy_grid_mcp by yacy.

the class YaCyQuery method preparse.

private QueryBuilder preparse(String q, int timezoneOffset) {
    // detect usage of OR connector usage.
    q = fixQueryMistakes(q);
    // OR binds stronger than AND
    List<String> terms = splitIntoORGroups(q);
    // special handling: we don't need a boolean query builder on top; just return one parse object
    if (terms.size() == 1)
        return parse(terms.get(0), timezoneOffset);
    // generic handling: all of those OR groups MUST match. That is done with the filter query
    BoolQueryBuilder aquery = QueryBuilders.boolQuery();
    for (String t : terms) {
        QueryBuilder partial = parse(t, timezoneOffset);
        aquery.filter(partial);
    }
    return aquery;
}
Also used : BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) QueryBuilder(org.elasticsearch.index.query.QueryBuilder) RangeQueryBuilder(org.elasticsearch.index.query.RangeQueryBuilder) TermQueryBuilder(org.elasticsearch.index.query.TermQueryBuilder) MultiMatchQueryBuilder(org.elasticsearch.index.query.MultiMatchQueryBuilder) MatchAllQueryBuilder(org.elasticsearch.index.query.MatchAllQueryBuilder) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder)

Aggregations

BoolQueryBuilder (org.elasticsearch.index.query.BoolQueryBuilder)201 SearchResponse (org.elasticsearch.action.search.SearchResponse)73 QueryBuilder (org.elasticsearch.index.query.QueryBuilder)58 SearchHit (org.elasticsearch.search.SearchHit)41 SearchRequestBuilder (org.elasticsearch.action.search.SearchRequestBuilder)38 SearchSourceBuilder (org.elasticsearch.search.builder.SearchSourceBuilder)30 Map (java.util.Map)29 ArrayList (java.util.ArrayList)27 HashMap (java.util.HashMap)25 List (java.util.List)23 SearchRequest (org.elasticsearch.action.search.SearchRequest)21 Test (org.junit.Test)18 RangeQueryBuilder (org.elasticsearch.index.query.RangeQueryBuilder)16 TermsAggregationBuilder (org.elasticsearch.search.aggregations.bucket.terms.TermsAggregationBuilder)16 QueryBuilders (org.elasticsearch.index.query.QueryBuilders)14 FilterAggregationBuilder (org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder)13 Logger (org.slf4j.Logger)12 LoggerFactory (org.slf4j.LoggerFactory)12 LinkedList (java.util.LinkedList)11 FieldSortBuilder (org.elasticsearch.search.sort.FieldSortBuilder)11