Search in sources :

Example 1 with Hash

use of org.apache.solr.common.util.Hash in project lucene-solr by apache.

the class UniqueSlotAcc method getShardHLL.

private Object getShardHLL(int slot) throws IOException {
    FixedBitSet ords = arr[slot];
    if (ords == null)
        return HLLAgg.NO_VALUES;
    HLL hll = factory.getHLL();
    long maxOrd = ords.length();
    Hash.LongPair hashResult = new Hash.LongPair();
    for (int ord = -1; ++ord < maxOrd; ) {
        ord = ords.nextSetBit(ord);
        if (ord == DocIdSetIterator.NO_MORE_DOCS)
            break;
        BytesRef val = lookupOrd(ord);
        // way to avoid recomputing hash across slots?  Prob not worth space
        Hash.murmurhash3_x64_128(val.bytes, val.offset, val.length, 0, hashResult);
        // idea: if the set is small enough, just send the hashes?  We can add at the top
        // level or even just do a hash table at the top level.
        hll.addRaw(hashResult.val1);
    }
    SimpleOrderedMap map = new SimpleOrderedMap();
    map.add("hll", hll.toBytes());
    return map;
}
Also used : FixedBitSet(org.apache.lucene.util.FixedBitSet) HLL(org.apache.solr.util.hll.HLL) Hash(org.apache.solr.common.util.Hash) SimpleOrderedMap(org.apache.solr.common.util.SimpleOrderedMap) BytesRef(org.apache.lucene.util.BytesRef)

Aggregations

BytesRef (org.apache.lucene.util.BytesRef)1 FixedBitSet (org.apache.lucene.util.FixedBitSet)1 Hash (org.apache.solr.common.util.Hash)1 SimpleOrderedMap (org.apache.solr.common.util.SimpleOrderedMap)1 HLL (org.apache.solr.util.hll.HLL)1