Search in sources :

Example 56 with SearchHits

use of org.graylog.shaded.elasticsearch7.org.elasticsearch.search.SearchHits in project Anserini by castorini.

the class SearchElastic method searchTweets.

public <K> ScoredDocuments searchTweets(String queryString, long t) {
    SearchHits results = null;
    String specials = "+-=&|><!(){}[]^\"~*?:\\/";
    for (int i = 0; i < specials.length(); i++) {
        char c = specials.charAt(i);
        queryString = queryString.replace(String.valueOf(c), " ");
    }
    // Do not consider the tweets with tweet ids that are beyond the queryTweetTime
    // <querytweettime> tag contains the timestamp of the query in terms of the
    // chronologically nearest tweet id within the corpus
    RangeQueryBuilder queryTweetTime = QueryBuilders.rangeQuery(TweetGenerator.TweetField.ID_LONG.name).from(0L).to(t);
    QueryStringQueryBuilder queryTerms = QueryBuilders.queryStringQuery(queryString).defaultField("contents").analyzer("english");
    BoolQueryBuilder query = QueryBuilders.boolQuery().filter(queryTweetTime).should(queryTerms);
    SearchRequest searchRequest = new SearchRequest(args.esIndex);
    SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
    sourceBuilder.query(query);
    sourceBuilder.size(args.hits);
    sourceBuilder.sort(new ScoreSortBuilder().order(SortOrder.DESC));
    sourceBuilder.sort(new FieldSortBuilder(TweetGenerator.TweetField.ID_LONG.name).order(SortOrder.DESC));
    searchRequest.source(sourceBuilder);
    try {
        SearchResponse searchResponse = client.search(searchRequest, COMMON_OPTIONS);
        results = searchResponse.getHits();
    } catch (Exception e) {
        LOG.error("Exception during ES query: ", e);
    }
    ScoreTiesAdjusterReranker reranker = new ScoreTiesAdjusterReranker();
    return reranker.rerank(ScoredDocuments.fromESDocs(results), null);
}
Also used : SearchRequest(org.elasticsearch.action.search.SearchRequest) FieldSortBuilder(org.elasticsearch.search.sort.FieldSortBuilder) RangeQueryBuilder(org.elasticsearch.index.query.RangeQueryBuilder) IOException(java.io.IOException) CmdLineException(org.kohsuke.args4j.CmdLineException) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) SearchResponse(org.elasticsearch.action.search.SearchResponse) ScoreSortBuilder(org.elasticsearch.search.sort.ScoreSortBuilder) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) ScoreTiesAdjusterReranker(io.anserini.rerank.lib.ScoreTiesAdjusterReranker) SearchHits(org.elasticsearch.search.SearchHits) QueryStringQueryBuilder(org.elasticsearch.index.query.QueryStringQueryBuilder)

Example 57 with SearchHits

use of org.graylog.shaded.elasticsearch7.org.elasticsearch.search.SearchHits in project titan by thinkaurelius.

the class ElasticSearchIndex method query.

@Override
public List<String> query(IndexQuery query, KeyInformation.IndexRetriever informations, TransactionHandle tx) throws StorageException {
    SearchRequestBuilder srb = client.prepareSearch(indexName);
    srb.setTypes(query.getStore());
    srb.setQuery(QueryBuilders.matchAllQuery());
    srb.setFilter(getFilter(query.getCondition(), informations.get(query.getStore())));
    if (!query.getOrder().isEmpty()) {
        List<IndexQuery.OrderEntry> orders = query.getOrder();
        for (int i = 0; i < orders.size(); i++) {
            srb.addSort(new FieldSortBuilder(orders.get(i).getKey()).order(orders.get(i).getOrder() == Order.ASC ? SortOrder.ASC : SortOrder.DESC).ignoreUnmapped(true));
        }
    }
    srb.setFrom(0);
    if (query.hasLimit())
        srb.setSize(query.getLimit());
    else
        srb.setSize(maxResultsSize);
    srb.setNoFields();
    // srb.setExplain(true);
    SearchResponse response = srb.execute().actionGet();
    log.debug("Executed query [{}] in {} ms", query.getCondition(), response.getTookInMillis());
    SearchHits hits = response.getHits();
    if (!query.hasLimit() && hits.totalHits() >= maxResultsSize)
        log.warn("Query result set truncated to first [{}] elements for query: {}", maxResultsSize, query);
    List<String> result = new ArrayList<String>(hits.hits().length);
    for (SearchHit hit : hits) {
        result.add(hit.id());
    }
    return result;
}
Also used : SearchRequestBuilder(org.elasticsearch.action.search.SearchRequestBuilder) SearchHit(org.elasticsearch.search.SearchHit) ArrayList(java.util.ArrayList) FieldSortBuilder(org.elasticsearch.search.sort.FieldSortBuilder) SearchHits(org.elasticsearch.search.SearchHits) SearchResponse(org.elasticsearch.action.search.SearchResponse)

Example 58 with SearchHits

use of org.graylog.shaded.elasticsearch7.org.elasticsearch.search.SearchHits in project snow-owl by b2ihealthcare.

the class EsDocumentSearcher method search.

@Override
public <T> Hits<T> search(Query<T> query) throws IOException {
    Stopwatch w = Stopwatch.createStarted();
    admin.log().trace("Executing query '{}'", query);
    final EsClient client = admin.client();
    final List<DocumentMapping> mappings = admin.mappings().getDocumentMapping(query);
    final DocumentMapping primaryMapping = Iterables.getFirst(mappings, null);
    // Restrict variables to the theoretical maximum
    final int limit = query.getLimit();
    final int toRead = Ints.min(limit, resultWindow);
    // TODO support multiple document mappings during query building
    final EsQueryBuilder esQueryBuilder = new EsQueryBuilder(primaryMapping, admin.settings(), admin.log());
    final QueryBuilder esQuery = esQueryBuilder.build(query.getWhere());
    final SearchRequest req = new SearchRequest(admin.getTypeIndexes(mappings).toArray(length -> new String[length]));
    // configure caching
    req.requestCache(query.isCached());
    final SearchSourceBuilder reqSource = req.source().size(toRead).query(esQuery).trackScores(esQueryBuilder.needsScoring()).trackTotalHitsUpTo(Integer.MAX_VALUE);
    // field selection
    final boolean fetchSource = applySourceFiltering(query.getFields(), primaryMapping, reqSource);
    // ES internals require loading the _id field when we require the _source
    if (fetchSource) {
        reqSource.storedFields(STORED_FIELDS_ID_ONLY);
    } else {
        reqSource.storedFields(STORED_FIELDS_NONE);
    }
    // paging config
    final boolean isLocalStreaming = limit > resultWindow;
    final boolean isLiveStreaming = !Strings.isNullOrEmpty(query.getSearchAfter());
    if (isLocalStreaming) {
        checkArgument(!isLiveStreaming, "Cannot use searchAfter when requesting more items (%s) than the configured result window (%s).", limit, resultWindow);
    } else if (isLiveStreaming) {
        reqSource.searchAfter(fromSearchAfterToken(query.getSearchAfter()));
    }
    // sorting config with a default sort field based on scroll config
    addSort(primaryMapping, reqSource, query.getSortBy());
    // disable explain explicitly, just in case
    reqSource.explain(false);
    // disable version field explicitly, just in case
    reqSource.version(false);
    // perform search
    SearchResponse response = null;
    try {
        response = client.search(req);
    } catch (Exception e) {
        if (e instanceof ElasticsearchStatusException && ((ElasticsearchStatusException) e).status() == RestStatus.BAD_REQUEST) {
            throw new IllegalArgumentException(e.getMessage(), e);
        }
        admin.log().error("Couldn't execute query", e);
        throw new IndexException("Couldn't execute query: " + e.getMessage(), null);
    }
    SearchHits responseHits = response.getHits();
    final TotalHits total = responseHits.getTotalHits();
    checkState(total.relation == Relation.EQUAL_TO, "Searches should always track total hits accurately");
    final int totalHitCount = (int) total.value;
    final SearchHit[] firstHits = responseHits.getHits();
    final int firstCount = firstHits.length;
    final int remainingCount = Math.min(limit, totalHitCount) - firstCount;
    // Add the first set of results
    final ImmutableList.Builder<SearchHit> allHits = ImmutableList.builder();
    allHits.addAll(responseHits);
    // If the client requested all data at once and there are more hits to retrieve, collect them all as part of the request
    if (isLocalStreaming && remainingCount > 0) {
        admin.log().warn("Returning all matches (totalHits: '{}') larger than the currently configured result_window ('{}') might not be the most " + "efficient way of getting the data. Consider using the index pagination API (searchAfter) instead.", totalHitCount, resultWindow);
        while (true) {
            // Extract searchAfter values for the next set of results
            final SearchHit lastHit = Iterables.getLast(responseHits, null);
            if (lastHit == null) {
                break;
            }
            reqSource.searchAfter(lastHit.getSortValues());
            // Request more search results, adding them to the list builder
            response = client.search(req);
            responseHits = response.getHits();
            allHits.addAll(responseHits);
        }
    }
    final Class<T> select = query.getSelection().getSelect();
    final List<Class<?>> from = query.getSelection().getFrom();
    final Hits<T> hits = toHits(select, from, query.getFields(), fetchSource, limit, totalHitCount, query.getSortBy(), allHits.build());
    admin.log().trace("Executed query '{}' in '{}'", query, w);
    return hits;
}
Also used : GetResponse(org.elasticsearch.action.get.GetResponse) SortBuilders(org.elasticsearch.search.sort.SortBuilders) Query(com.b2international.index.query.Query) SearchHits(org.elasticsearch.search.SearchHits) TopHitsAggregationBuilder(org.elasticsearch.search.aggregations.metrics.TopHitsAggregationBuilder) FormattedRuntimeException(com.b2international.commons.exceptions.FormattedRuntimeException) Preconditions.checkArgument(com.google.common.base.Preconditions.checkArgument) ByteArrayInputStream(java.io.ByteArrayInputStream) SearchResponse(org.elasticsearch.action.search.SearchResponse) ElasticsearchStatusException(org.elasticsearch.ElasticsearchStatusException) CompareUtils(com.b2international.commons.CompareUtils) com.google.common.collect(com.google.common.collect) SearchHit(org.elasticsearch.search.SearchHit) GetRequest(org.elasticsearch.action.get.GetRequest) Aggregations(org.elasticsearch.search.aggregations.Aggregations) SortBy(com.b2international.index.query.SortBy) Terms(org.elasticsearch.search.aggregations.bucket.terms.Terms) EsClient(com.b2international.index.es.client.EsClient) Preconditions.checkState(com.google.common.base.Preconditions.checkState) Lists.newArrayList(com.google.common.collect.Lists.newArrayList) Expressions(com.b2international.index.query.Expressions) RestStatus(org.elasticsearch.rest.RestStatus) AggregationBuilder(com.b2international.index.aggregations.AggregationBuilder) SortOrder(org.elasticsearch.search.sort.SortOrder) ReverseNested(org.elasticsearch.search.aggregations.bucket.nested.ReverseNested) EsQueryBuilder(com.b2international.index.es.query.EsQueryBuilder) JavaBinCodec(org.apache.solr.common.util.JavaBinCodec) TopHits(org.elasticsearch.search.aggregations.metrics.TopHits) FetchSourceContext(org.elasticsearch.search.fetch.subphase.FetchSourceContext) SortByScript(com.b2international.index.query.SortBy.SortByScript) DataInputStream(java.io.DataInputStream) SortByField(com.b2international.index.query.SortBy.SortByField) java.util(java.util) ByteArrayOutputStream(java.io.ByteArrayOutputStream) Stopwatch(com.google.common.base.Stopwatch) SearchRequest(org.elasticsearch.action.search.SearchRequest) Aggregation(com.b2international.index.aggregations.Aggregation) EsIndexAdmin(com.b2international.index.es.admin.EsIndexAdmin) Strings(com.google.common.base.Strings) com.b2international.index(com.b2international.index) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) MultiSortBy(com.b2international.index.query.SortBy.MultiSortBy) Nested(org.elasticsearch.search.aggregations.bucket.nested.Nested) TermsAggregationBuilder(org.elasticsearch.search.aggregations.bucket.terms.TermsAggregationBuilder) BadRequestException(com.b2international.commons.exceptions.BadRequestException) QueryBuilder(org.elasticsearch.index.query.QueryBuilder) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) IOException(java.io.IOException) AggregationBuilders(org.elasticsearch.search.aggregations.AggregationBuilders) Relation(org.apache.lucene.search.TotalHits.Relation) Bucket(com.b2international.index.aggregations.Bucket) Ints(com.google.common.primitives.Ints) DocumentMapping(com.b2international.index.mapping.DocumentMapping) TotalHits(org.apache.lucene.search.TotalHits) TotalHits(org.apache.lucene.search.TotalHits) SearchRequest(org.elasticsearch.action.search.SearchRequest) SearchHit(org.elasticsearch.search.SearchHit) Stopwatch(com.google.common.base.Stopwatch) EsQueryBuilder(com.b2international.index.es.query.EsQueryBuilder) QueryBuilder(org.elasticsearch.index.query.QueryBuilder) ElasticsearchStatusException(org.elasticsearch.ElasticsearchStatusException) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) SearchHits(org.elasticsearch.search.SearchHits) EsQueryBuilder(com.b2international.index.es.query.EsQueryBuilder) EsClient(com.b2international.index.es.client.EsClient) DocumentMapping(com.b2international.index.mapping.DocumentMapping) FormattedRuntimeException(com.b2international.commons.exceptions.FormattedRuntimeException) ElasticsearchStatusException(org.elasticsearch.ElasticsearchStatusException) BadRequestException(com.b2international.commons.exceptions.BadRequestException) IOException(java.io.IOException) SearchResponse(org.elasticsearch.action.search.SearchResponse)

Example 59 with SearchHits

use of org.graylog.shaded.elasticsearch7.org.elasticsearch.search.SearchHits in project sonarqube by SonarSource.

the class ComponentIndex method bucketToQualifier.

private static ComponentHitsPerQualifier bucketToQualifier(ParsedFilters.ParsedBucket bucket) {
    ParsedTopHits docs = bucket.getAggregations().get(DOCS_AGGREGATION_NAME);
    SearchHits hitList = docs.getHits();
    SearchHit[] hits = hitList.getHits();
    return new ComponentHitsPerQualifier(bucket.getKey(), ComponentHit.fromSearchHits(hits), getTotalHits(hitList.getTotalHits()).value);
}
Also used : SearchHit(org.elasticsearch.search.SearchHit) ParsedTopHits(org.elasticsearch.search.aggregations.metrics.ParsedTopHits) SearchHits(org.elasticsearch.search.SearchHits)

Example 60 with SearchHits

use of org.graylog.shaded.elasticsearch7.org.elasticsearch.search.SearchHits in project sonarqube by SonarSource.

the class EsUtilsTest method convertToDocs.

@Test
public void convertToDocs() {
    SearchHits hits = new SearchHits(new SearchHit[] { new SearchHit(16) }, new TotalHits(1, TotalHits.Relation.EQUAL_TO), 1);
    List<BaseDoc> docs = EsUtils.convertToDocs(hits, IssueDoc::new);
    assertThat(docs).hasSize(1);
}
Also used : TotalHits(org.apache.lucene.search.TotalHits) IssueDoc(org.sonar.server.issue.index.IssueDoc) SearchHit(org.elasticsearch.search.SearchHit) SearchHits(org.elasticsearch.search.SearchHits) Test(org.junit.Test)

Aggregations

SearchHits (org.elasticsearch.search.SearchHits)137 SearchResponse (org.elasticsearch.action.search.SearchResponse)93 SearchHit (org.elasticsearch.search.SearchHit)86 ArrayList (java.util.ArrayList)45 SearchRequestBuilder (org.elasticsearch.action.search.SearchRequestBuilder)27 IOException (java.io.IOException)25 Map (java.util.Map)23 ElasticsearchAssertions.assertSearchResponse (org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertSearchResponse)20 HashMap (java.util.HashMap)19 IndexRequestBuilder (org.elasticsearch.action.index.IndexRequestBuilder)17 SearchRequest (org.elasticsearch.action.search.SearchRequest)16 Terms (org.elasticsearch.search.aggregations.bucket.terms.Terms)16 TopHits (org.elasticsearch.search.aggregations.metrics.tophits.TopHits)15 SearchSourceBuilder (org.elasticsearch.search.builder.SearchSourceBuilder)14 TotalHits (org.apache.lucene.search.TotalHits)12 Test (org.junit.Test)12 ScoreDoc (org.apache.lucene.search.ScoreDoc)11 Text (org.elasticsearch.common.text.Text)11 InnerHitBuilder (org.elasticsearch.index.query.InnerHitBuilder)11 QueryBuilder (org.elasticsearch.index.query.QueryBuilder)10