Examples with ArrayOfDoublesUnion org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUnion used on opensource projects

Example 1 with ArrayOfDoublesUnion

use of org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUnion in project sketches-core by DataSketches.

the class ReadOnlyMemoryTest method wrapAndTryUpdatingUnionEstimationMode.

@Test
public void wrapAndTryUpdatingUnionEstimationMode() {
    final int numUniques = 10000;
    int key = 0;
    final ArrayOfDoublesUpdatableSketch sketch1 = new ArrayOfDoublesUpdatableSketchBuilder().build();
    for (int i = 0; i < numUniques; i++) {
        sketch1.update(key++, new double[] { 1 });
    }
    final ArrayOfDoublesUnion union1 = new ArrayOfDoublesSetOperationBuilder().buildUnion();
    union1.union(sketch1);
    final ArrayOfDoublesUnion union2 = ArrayOfDoublesSketches.wrapUnion(Memory.wrap(union1.toByteArray()));
    final ArrayOfDoublesSketch resultSketch = union2.getResult();
    Assert.assertTrue(resultSketch.isEstimationMode());
    Assert.assertEquals(resultSketch.getEstimate(), numUniques, numUniques * 0.04);
    // make sure union update actually needs to modify the union
    final ArrayOfDoublesUpdatableSketch sketch2 = new ArrayOfDoublesUpdatableSketchBuilder().build();
    for (int i = 0; i < numUniques; i++) {
        sketch2.update(key++, new double[] { 1 });
    }
    boolean thrown = false;
    try {
        union2.union(sketch2);
    } catch (final SketchesReadOnlyException e) {
        thrown = true;
    }
    Assert.assertTrue(thrown);
}

Example 2 with ArrayOfDoublesUnion

use of org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUnion in project sketches-core by DataSketches.

the class ReadOnlyMemoryTest method heapifyAndUpdateUnion.

@Test
public void heapifyAndUpdateUnion() {
    final int numUniques = 10000;
    int key = 0;
    final ArrayOfDoublesUpdatableSketch sketch1 = new ArrayOfDoublesUpdatableSketchBuilder().build();
    for (int i = 0; i < numUniques; i++) {
        sketch1.update(key++, new double[] { 1 });
    }
    final ArrayOfDoublesUnion union1 = new ArrayOfDoublesSetOperationBuilder().buildUnion();
    union1.union(sketch1);
    final ArrayOfDoublesUnion union2 = ArrayOfDoublesSketches.heapifyUnion(Memory.wrap(union1.toByteArray()));
    final ArrayOfDoublesSketch resultSketch = union2.getResult();
    Assert.assertTrue(resultSketch.isEstimationMode());
    Assert.assertEquals(resultSketch.getEstimate(), numUniques, numUniques * 0.04);
    // make sure union update actually needs to modify the union
    final ArrayOfDoublesUpdatableSketch sketch2 = new ArrayOfDoublesUpdatableSketchBuilder().build();
    for (int i = 0; i < numUniques; i++) {
        sketch2.update(key++, new double[] { 1 });
    }
    union2.union(sketch2);
}

Example 3 with ArrayOfDoublesUnion

use of org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUnion in project druid by druid-io.

the class ArrayOfDoublesSketchAggregatorFactory method makeAggregateCombiner.

@Override
public AggregateCombiner makeAggregateCombiner() {
    return new ObjectAggregateCombiner<ArrayOfDoublesSketch>() {

        private final ArrayOfDoublesUnion union = new ArrayOfDoublesSetOperationBuilder().setNominalEntries(nominalEntries).setNumberOfValues(numberOfValues).buildUnion();

        @Override
        public void reset(final ColumnValueSelector selector) {
            union.reset();
            fold(selector);
        }

        @Override
        public void fold(final ColumnValueSelector selector) {
            final ArrayOfDoublesSketch sketch = (ArrayOfDoublesSketch) selector.getObject();
            union.union(sketch);
        }

        @Override
        public ArrayOfDoublesSketch getObject() {
            return union.getResult();
        }

        @Override
        public Class<ArrayOfDoublesSketch> classOfObject() {
            return ArrayOfDoublesSketch.class;
        }
    };
}

Example 4 with ArrayOfDoublesUnion

use of org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUnion in project druid by druid-io.

the class ArrayOfDoublesSketchMergeBufferAggregator method aggregate.

@Override
public void aggregate(final ByteBuffer buf, final int position) {
    final ArrayOfDoublesSketch update = selector.getObject();
    if (update == null) {
        return;
    }
    // Wrapping memory and ArrayOfDoublesUnion is inexpensive compared to union operations.
    // Maintaining a cache of wrapped objects per buffer position like in Theta sketch aggregator
    // might be considered, but it would increase complexity including relocate() support.
    final WritableMemory mem = WritableMemory.writableWrap(buf, ByteOrder.LITTLE_ENDIAN);
    final WritableMemory region = mem.writableRegion(position, maxIntermediateSize);
    final ArrayOfDoublesUnion union = ArrayOfDoublesSketches.wrapUnion(region);
    union.union(update);
}

Example 5 with ArrayOfDoublesUnion

use of org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUnion in project druid by druid-io.

the class ArrayOfDoublesSketchMergeBufferAggregator method get.

/**
 * This method uses locks because it can be used during indexing,
 * and Druid can call aggregate() and get() concurrently
 * https://github.com/apache/druid/pull/3956
 * The returned sketch is a separate instance of ArrayOfDoublesCompactSketch
 * representing the current state of the aggregation, and is not affected by consequent
 * aggregate() calls
 */
@Override
public Object get(final ByteBuffer buf, final int position) {
    final WritableMemory mem = WritableMemory.writableWrap(buf, ByteOrder.LITTLE_ENDIAN);
    final WritableMemory region = mem.writableRegion(position, maxIntermediateSize);
    final ArrayOfDoublesUnion union = ArrayOfDoublesSketches.wrapUnion(region);
    return union.getResult();
}