Search in sources :

Example 1 with ArrayOfDoublesUpdatableSketch

use of org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketch in project sketches-core by DataSketches.

the class ReadOnlyMemoryTest method wrapAndTryUpdatingUnionEstimationMode.

@Test
public void wrapAndTryUpdatingUnionEstimationMode() {
    final int numUniques = 10000;
    int key = 0;
    final ArrayOfDoublesUpdatableSketch sketch1 = new ArrayOfDoublesUpdatableSketchBuilder().build();
    for (int i = 0; i < numUniques; i++) {
        sketch1.update(key++, new double[] { 1 });
    }
    final ArrayOfDoublesUnion union1 = new ArrayOfDoublesSetOperationBuilder().buildUnion();
    union1.union(sketch1);
    final ArrayOfDoublesUnion union2 = ArrayOfDoublesSketches.wrapUnion(Memory.wrap(union1.toByteArray()));
    final ArrayOfDoublesSketch resultSketch = union2.getResult();
    Assert.assertTrue(resultSketch.isEstimationMode());
    Assert.assertEquals(resultSketch.getEstimate(), numUniques, numUniques * 0.04);
    // make sure union update actually needs to modify the union
    final ArrayOfDoublesUpdatableSketch sketch2 = new ArrayOfDoublesUpdatableSketchBuilder().build();
    for (int i = 0; i < numUniques; i++) {
        sketch2.update(key++, new double[] { 1 });
    }
    boolean thrown = false;
    try {
        union2.union(sketch2);
    } catch (final SketchesReadOnlyException e) {
        thrown = true;
    }
    Assert.assertTrue(thrown);
}
Also used : ArrayOfDoublesUpdatableSketchBuilder(org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketchBuilder) ArrayOfDoublesUpdatableSketch(org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketch) ArrayOfDoublesUnion(org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUnion) SketchesReadOnlyException(org.apache.datasketches.SketchesReadOnlyException) ArrayOfDoublesSetOperationBuilder(org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesSetOperationBuilder) ArrayOfDoublesSketch(org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesSketch) Test(org.testng.annotations.Test)

Example 2 with ArrayOfDoublesUpdatableSketch

use of org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketch in project sketches-core by DataSketches.

the class ReadOnlyMemoryTest method heapifyAndUpdateSketch.

@Test
public void heapifyAndUpdateSketch() {
    final ArrayOfDoublesUpdatableSketch sketch1 = new ArrayOfDoublesUpdatableSketchBuilder().build();
    sketch1.update(1, new double[] { 1 });
    // downcasting is not recommended, for testing only
    final ArrayOfDoublesUpdatableSketch sketch2 = (ArrayOfDoublesUpdatableSketch) ArrayOfDoublesSketches.heapifySketch(Memory.wrap(sketch1.toByteArray()));
    sketch2.update(2, new double[] { 1 });
    Assert.assertEquals(sketch2.getEstimate(), 2.0);
}
Also used : ArrayOfDoublesUpdatableSketchBuilder(org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketchBuilder) ArrayOfDoublesUpdatableSketch(org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketch) Test(org.testng.annotations.Test)

Example 3 with ArrayOfDoublesUpdatableSketch

use of org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketch in project druid by druid-io.

the class ArrayOfDoublesSketchBuildBufferAggregator method get.

/**
 * The returned sketch is a separate instance of ArrayOfDoublesCompactSketch
 * representing the current state of the aggregation, and is not affected by consequent
 * aggregate() calls
 */
@Override
public Object get(final ByteBuffer buf, final int position) {
    final WritableMemory mem = WritableMemory.writableWrap(buf, ByteOrder.LITTLE_ENDIAN);
    final WritableMemory region = mem.writableRegion(position, maxIntermediateSize);
    final ArrayOfDoublesUpdatableSketch sketch = (ArrayOfDoublesUpdatableSketch) ArrayOfDoublesSketches.wrapSketch(region);
    return sketch.compact();
}
Also used : ArrayOfDoublesUpdatableSketch(org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketch) WritableMemory(org.apache.datasketches.memory.WritableMemory)

Example 4 with ArrayOfDoublesUpdatableSketch

use of org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketch in project druid by druid-io.

the class ArrayOfDoublesSketchAggregatorFactoryTest method makeAggregateCombiner.

@Test
public void makeAggregateCombiner() {
    AggregatorFactory aggregatorFactory = new ArrayOfDoublesSketchAggregatorFactory("", "", null, null, null);
    AggregatorFactory combiningFactory = aggregatorFactory.getCombiningFactory();
    AggregateCombiner<ArrayOfDoublesSketch> combiner = combiningFactory.makeAggregateCombiner();
    ArrayOfDoublesUpdatableSketch sketch1 = new ArrayOfDoublesUpdatableSketchBuilder().build();
    sketch1.update("a", new double[] { 1 });
    ArrayOfDoublesUpdatableSketch sketch2 = new ArrayOfDoublesUpdatableSketchBuilder().build();
    sketch2.update("b", new double[] { 1 });
    sketch2.update("c", new double[] { 1 });
    TestObjectColumnSelector<ArrayOfDoublesSketch> selector = new TestObjectColumnSelector<ArrayOfDoublesSketch>(new ArrayOfDoublesSketch[] { sketch1, sketch2 });
    combiner.reset(selector);
    Assert.assertEquals(1, combiner.getObject().getEstimate(), 0);
    selector.increment();
    combiner.fold(selector);
    Assert.assertEquals(3, combiner.getObject().getEstimate(), 0);
}
Also used : ArrayOfDoublesUpdatableSketchBuilder(org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketchBuilder) ArrayOfDoublesUpdatableSketch(org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketch) TestObjectColumnSelector(org.apache.druid.query.aggregation.TestObjectColumnSelector) AggregatorFactory(org.apache.druid.query.aggregation.AggregatorFactory) CountAggregatorFactory(org.apache.druid.query.aggregation.CountAggregatorFactory) ArrayOfDoublesSketch(org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesSketch) Test(org.junit.Test)

Example 5 with ArrayOfDoublesUpdatableSketch

use of org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketch in project druid by druid-io.

the class ArrayOfDoublesSketchTTestPostAggregatorTest method testComputeMismatchedSketches.

@Test
public void testComputeMismatchedSketches() {
    expectedException.expect(IAE.class);
    expectedException.expectMessage("Sketches have different number of values: 2 and 100");
    ArrayOfDoublesUpdatableSketch s1 = new ArrayOfDoublesUpdatableSketchBuilder().setNominalEntries(16).setNumberOfValues(2).build();
    ArrayOfDoublesUpdatableSketch s2 = new ArrayOfDoublesUpdatableSketchBuilder().setNominalEntries(16).setNumberOfValues(100).build();
    PostAggregator field1 = EasyMock.createMock(PostAggregator.class);
    EasyMock.expect(field1.compute(EasyMock.anyObject(Map.class))).andReturn(s1).anyTimes();
    PostAggregator field2 = EasyMock.createMock(PostAggregator.class);
    EasyMock.expect(field2.compute(EasyMock.anyObject(Map.class))).andReturn(s2).anyTimes();
    EasyMock.replay(field1, field2);
    new ArrayOfDoublesSketchTTestPostAggregator("a", Arrays.asList(field1, field2)).compute(ImmutableMap.of());
}
Also used : ArrayOfDoublesUpdatableSketchBuilder(org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketchBuilder) ArrayOfDoublesUpdatableSketch(org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketch) PostAggregator(org.apache.druid.query.aggregation.PostAggregator) ConstantPostAggregator(org.apache.druid.query.aggregation.post.ConstantPostAggregator) Test(org.junit.Test)

Aggregations

ArrayOfDoublesUpdatableSketch (org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketch)13 ArrayOfDoublesUpdatableSketchBuilder (org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUpdatableSketchBuilder)11 Test (org.junit.Test)6 PostAggregator (org.apache.druid.query.aggregation.PostAggregator)5 ConstantPostAggregator (org.apache.druid.query.aggregation.post.ConstantPostAggregator)5 ArrayOfDoublesSketch (org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesSketch)4 Test (org.testng.annotations.Test)4 Comparator (java.util.Comparator)2 SketchesReadOnlyException (org.apache.datasketches.SketchesReadOnlyException)2 WritableMemory (org.apache.datasketches.memory.WritableMemory)2 ArrayOfDoublesSetOperationBuilder (org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesSetOperationBuilder)2 ArrayOfDoublesUnion (org.apache.datasketches.tuple.arrayofdoubles.ArrayOfDoublesUnion)2 BufferedWriter (java.io.BufferedWriter)1 ByteBuffer (java.nio.ByteBuffer)1 Path (java.nio.file.Path)1 Random (java.util.Random)1 ThreadLocalRandom (java.util.concurrent.ThreadLocalRandom)1 DoublesSketch (org.apache.datasketches.quantiles.DoublesSketch)1 AggregatorFactory (org.apache.druid.query.aggregation.AggregatorFactory)1 CountAggregatorFactory (org.apache.druid.query.aggregation.CountAggregatorFactory)1