Examples with HllSketch org.apache.datasketches.hll.HllSketch used on opensource projects

Example 1 with HllSketch

use of org.apache.datasketches.hll.HllSketch in project druid by druid-io.

the class HllSketchAggregatorFactory method makeAggregateCombiner.

@Override
public AggregateCombiner makeAggregateCombiner() {
    return new ObjectAggregateCombiner<HllSketch>() {

        private final Union union = new Union(lgK);

        @Override
        public void reset(final ColumnValueSelector selector) {
            union.reset();
            fold(selector);
        }

        @Override
        public void fold(final ColumnValueSelector selector) {
            final HllSketch sketch = (HllSketch) selector.getObject();
            union.update(sketch);
        }

        @Nullable
        @Override
        public HllSketch getObject() {
            return union.getResult(tgtHllType);
        }

        @Override
        public Class<HllSketch> classOfObject() {
            return HllSketch.class;
        }
    };
}

Example 2 with HllSketch

use of org.apache.datasketches.hll.HllSketch in project druid by druid-io.

the class HllSketchBuildBufferAggregatorHelper method relocate.

/**
 * Helper for implementing {@link org.apache.druid.query.aggregation.BufferAggregator#relocate} and
 * {@link org.apache.druid.query.aggregation.VectorAggregator#relocate}.
 */
public void relocate(int oldPosition, int newPosition, ByteBuffer oldBuf, ByteBuffer newBuf) {
    HllSketch sketch = sketchCache.get(oldBuf).get(oldPosition);
    final WritableMemory oldMem = getMemory(oldBuf).writableRegion(oldPosition, size);
    if (sketch.isSameResource(oldMem)) {
        // sketch has not moved
        final WritableMemory newMem = getMemory(newBuf).writableRegion(newPosition, size);
        sketch = HllSketch.writableWrap(newMem);
    }
    putSketchIntoCache(newBuf, newPosition, sketch);
}

Example 3 with HllSketch

use of org.apache.datasketches.hll.HllSketch in project druid by druid-io.

the class GenerateTestData method generateSketches.

private static void generateSketches() throws Exception {
    int lgK = 12;
    String date = "20170101";
    Path rawPath = FileSystems.getDefault().getPath("hll_raw.tsv");
    Path sketchPath = FileSystems.getDefault().getPath("hll_sketches.tsv");
    try (BufferedWriter out1 = Files.newBufferedWriter(rawPath, StandardCharsets.UTF_8)) {
        try (BufferedWriter out2 = Files.newBufferedWriter(sketchPath, StandardCharsets.UTF_8)) {
            Random rand = ThreadLocalRandom.current();
            int key = 0;
            for (int i = 0; i < 100; i++) {
                HllSketch sketch = new HllSketch(lgK);
                String dimension = Integer.toString(rand.nextInt(10) + 1);
                writeRawRecord(out1, date, dimension, key);
                sketch.update(key++);
                writeRawRecord(out1, date, dimension, key);
                sketch.update(key++);
                writeSketchRecord(out2, date, dimension, sketch);
            }
        }
    }
}

Example 4 with HllSketch

use of org.apache.datasketches.hll.HllSketch in project druid by druid-io.

the class HllSketchAggregatorFactory method finalizeComputation.

@Nullable
@Override
public Object finalizeComputation(@Nullable final Object object) {
    if (object == null) {
        return null;
    }
    final HllSketch sketch = (HllSketch) object;
    final double estimate = sketch.getEstimate();
    if (round) {
        return Math.round(estimate);
    } else {
        return estimate;
    }
}

Example 5 with HllSketch

use of org.apache.datasketches.hll.HllSketch in project druid by druid-io.

the class HllSketchMergeBufferAggregator method aggregate.

@Override
public void aggregate(final ByteBuffer buf, final int position) {
    final HllSketch sketch = selector.getObject();
    if (sketch == null) {
        return;
    }
    final WritableMemory mem = WritableMemory.writableWrap(buf, ByteOrder.LITTLE_ENDIAN).writableRegion(position, helper.getSize());
    final Union union = Union.writableWrap(mem);
    union.update(sketch);
}