Search in sources :

Example 1 with ScoredResult

use of io.zulia.message.ZuliaQuery.ScoredResult in project zuliasearch by zuliaio.

the class QueryCombiner method getQueryResponse.

public QueryResponse getQueryResponse() throws Exception {
    validate();
    long totalHits = 0;
    long returnedHits = 0;
    for (ShardQueryResponse sr : shardResponses) {
        totalHits += sr.getTotalHits();
        returnedHits += sr.getScoredResultList().size();
    }
    QueryResponse.Builder builder = QueryResponse.newBuilder();
    builder.setTotalHits(totalHits);
    int resultsSize = Math.min(amount, (int) returnedHits);
    Map<CountRequest, FacetCombiner> facetCombinerMap = new HashMap<>();
    Map<StatRequest, StatCombiner> statCombinerMap = new HashMap<>();
    Map<AnalysisRequest, Map<String, Term.Builder>> analysisRequestToTermMap = new HashMap<>();
    int shardIndex = 0;
    for (ShardQueryResponse sr : shardResponses) {
        for (FacetGroup fg : sr.getFacetGroupList()) {
            CountRequest countRequest = fg.getCountRequest();
            FacetCombiner facetCombiner = facetCombinerMap.computeIfAbsent(countRequest, countRequest1 -> new FacetCombiner(countRequest, shardResponses.size()));
            facetCombiner.handleFacetGroupForShard(fg, shardIndex);
        }
        for (ZuliaQuery.StatGroup sg : sr.getStatGroupList()) {
            StatRequest statRequest = sg.getStatRequest();
            StatCombiner statCombiner = statCombinerMap.computeIfAbsent(statRequest, statRequest1 -> new StatCombiner(statRequest, shardResponses.size()));
            statCombiner.handleStatGroupForShard(sg, shardIndex);
        }
        for (AnalysisResult analysisResult : sr.getAnalysisResultList()) {
            AnalysisRequest analysisRequest = analysisResult.getAnalysisRequest();
            if (!analysisRequestToTermMap.containsKey(analysisRequest)) {
                analysisRequestToTermMap.put(analysisRequest, new HashMap<>());
            }
            Map<String, Term.Builder> termMap = analysisRequestToTermMap.get(analysisRequest);
            for (Term term : analysisResult.getTermsList()) {
                String key = term.getValue();
                if (!termMap.containsKey(key)) {
                    termMap.put(key, Term.newBuilder().setValue(key).setDocFreq(0).setTermFreq(0));
                }
                Term.Builder termsBuilder = termMap.get(key);
                termsBuilder.setDocFreq(termsBuilder.getDocFreq() + term.getDocFreq());
                termsBuilder.setScore(termsBuilder.getScore() + term.getScore());
                termsBuilder.setTermFreq(termsBuilder.getTermFreq() + term.getTermFreq());
            }
        }
        shardIndex++;
    }
    for (AnalysisRequest analysisRequest : analysisRequestList) {
        Map<String, Term.Builder> termMap = analysisRequestToTermMap.get(analysisRequest);
        if (termMap != null) {
            List<Term.Builder> terms = new ArrayList<>(termMap.values());
            List<Term.Builder> topTerms = TermFreq.getTopTerms(terms, analysisRequest.getTopN(), analysisRequest.getTermSort());
            AnalysisResult.Builder analysisResultBuilder = AnalysisResult.newBuilder().setAnalysisRequest(analysisRequest);
            topTerms.forEach(analysisResultBuilder::addTerms);
            builder.addAnalysisResult(analysisResultBuilder);
        }
    }
    for (FacetCombiner facetCombiner : facetCombinerMap.values()) {
        builder.addFacetGroup(facetCombiner.getCombinedFacetGroup());
    }
    for (StatCombiner statCombiner : statCombinerMap.values()) {
        builder.addStatGroup(statCombiner.getCombinedStatGroup());
    }
    Map<String, ScoredResult[]> lastIndexResultMap = createLastIndexResultMapWithPreviousLastResults();
    List<ScoredResult> results;
    if (shardResponses.size() > 1) {
        results = mergeResults((int) returnedHits, resultsSize, lastIndexResultMap);
    } else {
        ShardQueryResponse shardQueryResponse = shardResponses.get(0);
        results = shardQueryResponse.getScoredResultList();
        if (!results.isEmpty()) {
            lastIndexResultMap.get(shardQueryResponse.getIndexName())[shardQueryResponse.getShardNumber()] = results.get(results.size() - 1);
        }
    }
    if (start == 0) {
        builder.addAllResults(results);
    } else {
        int i = 0;
        for (ScoredResult scoredResult : results) {
            if (i >= start) {
                builder.addResults(scoredResult);
            }
            i++;
        }
    }
    builder.setLastResult(createLastResult(lastIndexResultMap));
    return builder.build();
}
Also used : AnalysisRequest(io.zulia.message.ZuliaQuery.AnalysisRequest) HashMap(java.util.HashMap) ZuliaQuery(io.zulia.message.ZuliaQuery) ArrayList(java.util.ArrayList) CountRequest(io.zulia.message.ZuliaQuery.CountRequest) StatRequest(io.zulia.message.ZuliaQuery.StatRequest) FacetGroup(io.zulia.message.ZuliaQuery.FacetGroup) ShardQueryResponse(io.zulia.message.ZuliaQuery.ShardQueryResponse) Term(io.zulia.message.ZuliaBase.Term) AnalysisResult(io.zulia.message.ZuliaQuery.AnalysisResult) QueryResponse(io.zulia.message.ZuliaServiceOuterClass.QueryResponse) ShardQueryResponse(io.zulia.message.ZuliaQuery.ShardQueryResponse) InternalQueryResponse(io.zulia.message.ZuliaServiceOuterClass.InternalQueryResponse) ScoredResult(io.zulia.message.ZuliaQuery.ScoredResult) HashMap(java.util.HashMap) Map(java.util.Map)

Example 2 with ScoredResult

use of io.zulia.message.ZuliaQuery.ScoredResult in project zuliasearch by zuliaio.

the class CursorHelper method getUniqueSortedCursor.

public static String getUniqueSortedCursor(LastResult lastResult) {
    LastResult.Builder lastResultBuilder = LastResult.newBuilder();
    for (LastIndexResult lastIndexResult : lastResult.getLastIndexResultList()) {
        LastIndexResult.Builder lastIndexResultBuilder = LastIndexResult.newBuilder();
        lastIndexResultBuilder.setIndexName(lastIndexResult.getIndexName());
        for (ScoredResult scoredResult : lastIndexResult.getLastForShardList()) {
            ScoredResult.Builder scoredResultBuilder = ScoredResult.newBuilder(scoredResult).clearScore().clearResultDocument();
            lastIndexResultBuilder.addLastForShard(scoredResultBuilder);
        }
        lastResultBuilder.addLastIndexResult(lastIndexResultBuilder);
    }
    return new String(Base64.getEncoder().encode(lastResultBuilder.build().toByteArray()), StandardCharsets.UTF_8);
}
Also used : LastIndexResult(io.zulia.message.ZuliaQuery.LastIndexResult) LastResult(io.zulia.message.ZuliaQuery.LastResult) ScoredResult(io.zulia.message.ZuliaQuery.ScoredResult)

Example 3 with ScoredResult

use of io.zulia.message.ZuliaQuery.ScoredResult in project zuliasearch by zuliaio.

the class BatchDelete method deleteDocumentFromQueryResult.

public BatchDelete deleteDocumentFromQueryResult(SearchResult queryResult) {
    for (ScoredResult sr : queryResult.getResults()) {
        Delete delete = new DeleteDocument(sr.getUniqueId(), sr.getIndexName());
        deletes.add(delete);
    }
    return this;
}
Also used : ScoredResult(io.zulia.message.ZuliaQuery.ScoredResult)

Example 4 with ScoredResult

use of io.zulia.message.ZuliaQuery.ScoredResult in project zuliasearch by zuliaio.

the class BatchFetch method addFetchDocumentsFromResults.

public BatchFetch addFetchDocumentsFromResults(Collection<ZuliaQuery.ScoredResult> scoredResults) {
    for (ScoredResult scoredResult : scoredResults) {
        Fetch f = new Fetch(scoredResult.getUniqueId(), scoredResult.getIndexName());
        f.setResultFetchType(FetchType.FULL);
        f.setAssociatedFetchType(FetchType.NONE);
        fetchList.add(f);
    }
    return this;
}
Also used : ScoredResult(io.zulia.message.ZuliaQuery.ScoredResult)

Example 5 with ScoredResult

use of io.zulia.message.ZuliaQuery.ScoredResult in project zuliasearch by zuliaio.

the class QueryCombiner method mergeResults.

private List<ScoredResult> mergeResults(int returnedHits, int resultsSize, Map<String, ScoredResult[]> lastIndexResultMap) throws Exception {
    List<ScoredResult> results = Collections.emptyList();
    boolean sorting = (sortRequest != null && !sortRequest.getFieldSortList().isEmpty());
    List<ScoredResult> mergedResults = new ArrayList<>(returnedHits);
    for (ShardQueryResponse sr : shardResponses) {
        mergedResults.addAll(sr.getScoredResultList());
    }
    if (!mergedResults.isEmpty()) {
        List<FieldSort> fieldSortList = sortRequest != null ? sortRequest.getFieldSortList() : Collections.emptyList();
        HashMap<String, FieldConfig.FieldType> sortTypeMap = createSortTypeMap(fieldSortList);
        Comparator<ScoredResult> comparator = new ZuliaPostSortingComparator(fieldSortList, sortTypeMap);
        mergedResults.sort(comparator);
        results = mergedResults.subList(0, resultsSize);
        for (ScoredResult sr : results) {
            ScoredResult[] lastForShardArr = lastIndexResultMap.get(sr.getIndexName());
            lastForShardArr[sr.getShard()] = sr;
        }
        outside: for (ZuliaIndex index : indexes) {
            String indexName = index.getIndexName();
            ScoredResult[] lastForShardArr = lastIndexResultMap.get(indexName);
            ScoredResult lastForIndex = null;
            for (ScoredResult sr : lastForShardArr) {
                if (sr != null) {
                    if (lastForIndex == null) {
                        lastForIndex = sr;
                    } else {
                        if (comparator.compare(sr, lastForIndex) > 0) {
                            lastForIndex = sr;
                        }
                    }
                }
            }
            if (lastForIndex == null) {
                // this happen when amount from index is zero
                continue;
            }
            double shardTolerance = index.getShardTolerance();
            int numberOfShards = index.getNumberOfShards();
            Map<Integer, ShardQueryResponse> shardResponseMap = indexToShardQueryResponseMap.get(indexName);
            for (int shardNumber = 0; shardNumber < numberOfShards; shardNumber++) {
                ShardQueryResponse sr = shardResponseMap.get(shardNumber);
                if (sr.hasNext()) {
                    ScoredResult next = sr.getNext();
                    int compare = comparator.compare(lastForIndex, next);
                    if (compare > 0) {
                        if (sorting) {
                            String msg = "Result set did not return the most relevant sorted documents for index <" + indexName + ">\n";
                            msg += "    Last for index from shard <" + lastForIndex.getShard() + "> has sort values <" + lastForIndex.getSortValues() + ">\n";
                            msg += "    Next for shard <" + next.getShard() + ">  has sort values <" + next.getSortValues() + ">\n";
                            msg += "    Last for shards: \n";
                            msg += "      " + Arrays.toString(lastForShardArr) + "\n";
                            msg += "    Results: \n";
                            msg += "      " + results + "\n";
                            msg += "    If this happens frequently increase requestFactor or minShardRequest\n";
                            msg += "    Retrying with full request.\n";
                            log.severe(msg);
                            isShort = true;
                            break outside;
                        }
                        double diff = (Math.abs(lastForIndex.getScore() - next.getScore()));
                        if (diff > shardTolerance) {
                            String msg = "Result set did not return the most relevant documents for index <" + indexName + "> with shard tolerance <" + shardTolerance + ">\n";
                            msg += "    Last for index from shard <" + lastForIndex.getShard() + "> has score <" + lastForIndex.getScore() + ">\n";
                            msg += "    Next for shard <" + next.getShard() + "> has score <" + next.getScore() + ">\n";
                            msg += "    Last for shards: \n";
                            msg += "      " + Arrays.toString(lastForShardArr) + "\n";
                            msg += "    Results: \n";
                            msg += "      " + results + "\n";
                            msg += "    If this happens frequently increase requestFactor, minShardRequest, or shardTolerance\n";
                            msg += "    Retrying with full request.\n";
                            log.severe(msg);
                            isShort = true;
                            break outside;
                        }
                    }
                }
            }
        }
    }
    return results;
}
Also used : ShardQueryResponse(io.zulia.message.ZuliaQuery.ShardQueryResponse) FieldSort(io.zulia.message.ZuliaQuery.FieldSort) ArrayList(java.util.ArrayList) ZuliaIndex(io.zulia.server.index.ZuliaIndex) ScoredResult(io.zulia.message.ZuliaQuery.ScoredResult) HashMap(java.util.HashMap) Map(java.util.Map)

Aggregations

ScoredResult (io.zulia.message.ZuliaQuery.ScoredResult)7 LastIndexResult (io.zulia.message.ZuliaQuery.LastIndexResult)3 ArrayList (java.util.ArrayList)3 HashMap (java.util.HashMap)3 LastResult (io.zulia.message.ZuliaQuery.LastResult)2 ShardQueryResponse (io.zulia.message.ZuliaQuery.ShardQueryResponse)2 Map (java.util.Map)2 Term (io.zulia.message.ZuliaBase.Term)1 ZuliaQuery (io.zulia.message.ZuliaQuery)1 AnalysisRequest (io.zulia.message.ZuliaQuery.AnalysisRequest)1 AnalysisResult (io.zulia.message.ZuliaQuery.AnalysisResult)1 CountRequest (io.zulia.message.ZuliaQuery.CountRequest)1 FacetGroup (io.zulia.message.ZuliaQuery.FacetGroup)1 FieldSort (io.zulia.message.ZuliaQuery.FieldSort)1 StatRequest (io.zulia.message.ZuliaQuery.StatRequest)1 InternalQueryResponse (io.zulia.message.ZuliaServiceOuterClass.InternalQueryResponse)1 QueryResponse (io.zulia.message.ZuliaServiceOuterClass.QueryResponse)1 ZuliaIndex (io.zulia.server.index.ZuliaIndex)1