Search in sources :

Example 31 with ArrayOfStringsSerDe

use of com.yahoo.sketches.ArrayOfStringsSerDe in project sketches-core by DataSketches.

the class ItemsSketchTest method checkPutMemory.

@Test
public void checkPutMemory() {
    ItemsSketch<String> sketch = ItemsSketch.getInstance(16, Comparator.naturalOrder());
    for (int i = 0; i < 40; i++) {
        sketch.update(Integer.toString(i));
    }
    byte[] byteArr = new byte[200];
    WritableMemory mem = WritableMemory.wrap(byteArr);
    sketch.putMemory(mem, new ArrayOfStringsSerDe());
}
Also used : ArrayOfStringsSerDe(com.yahoo.sketches.ArrayOfStringsSerDe) WritableMemory(com.yahoo.memory.WritableMemory) Test(org.testng.annotations.Test)

Example 32 with ArrayOfStringsSerDe

use of com.yahoo.sketches.ArrayOfStringsSerDe in project sketches-core by DataSketches.

the class ItemsSketchTest method checkPutMemoryException.

@Test(expectedExceptions = SketchesArgumentException.class)
public void checkPutMemoryException() {
    ItemsSketch<String> sketch = ItemsSketch.getInstance(16, Comparator.naturalOrder());
    for (int i = 0; i < 40; i++) {
        sketch.update(Integer.toString(i));
    }
    byte[] byteArr = new byte[100];
    WritableMemory mem = WritableMemory.wrap(byteArr);
    sketch.putMemory(mem, new ArrayOfStringsSerDe());
}
Also used : ArrayOfStringsSerDe(com.yahoo.sketches.ArrayOfStringsSerDe) WritableMemory(com.yahoo.memory.WritableMemory) Test(org.testng.annotations.Test)

Example 33 with ArrayOfStringsSerDe

use of com.yahoo.sketches.ArrayOfStringsSerDe in project sketches-core by DataSketches.

the class ItemsUnionTest method basedOnSketch.

@Test
public void basedOnSketch() {
    Comparator<String> comp = Comparator.naturalOrder();
    ArrayOfStringsSerDe serDe = new ArrayOfStringsSerDe();
    ItemsSketch<String> sketch = ItemsSketch.getInstance(comp);
    ItemsUnion<String> union = ItemsUnion.getInstance(sketch);
    union.reset();
    byte[] byteArr = sketch.toByteArray(serDe);
    Memory mem = Memory.wrap(byteArr);
    union = ItemsUnion.getInstance(mem, comp, serDe);
    Assert.assertEquals(byteArr.length, 8);
    union.reset();
}
Also used : ArrayOfStringsSerDe(com.yahoo.sketches.ArrayOfStringsSerDe) Memory(com.yahoo.memory.Memory) Test(org.testng.annotations.Test)

Example 34 with ArrayOfStringsSerDe

use of com.yahoo.sketches.ArrayOfStringsSerDe in project sketches-core by DataSketches.

the class ItemsSketchTest method oneItemUtf8.

@Test
public void oneItemUtf8() {
    ItemsSketch<String> sketch1 = new ItemsSketch<String>(1 << LG_MIN_MAP_SIZE);
    sketch1.update("徵");
    Assert.assertFalse(sketch1.isEmpty());
    Assert.assertEquals(sketch1.getNumActiveItems(), 1);
    Assert.assertEquals(sketch1.getStreamLength(), 1);
    Assert.assertEquals(sketch1.getEstimate("徵"), 1);
    byte[] bytes = sketch1.toByteArray(new ArrayOfStringsSerDe());
    ItemsSketch<String> sketch2 = ItemsSketch.getInstance(Memory.wrap(bytes), new ArrayOfStringsSerDe());
    Assert.assertFalse(sketch2.isEmpty());
    Assert.assertEquals(sketch2.getNumActiveItems(), 1);
    Assert.assertEquals(sketch2.getStreamLength(), 1);
    Assert.assertEquals(sketch2.getEstimate("徵"), 1);
}
Also used : ArrayOfStringsSerDe(com.yahoo.sketches.ArrayOfStringsSerDe) Test(org.testng.annotations.Test)

Example 35 with ArrayOfStringsSerDe

use of com.yahoo.sketches.ArrayOfStringsSerDe in project sketches-core by DataSketches.

the class ItemsSketchTest method serializeStringDeserializeEmpty.

@Test
public void serializeStringDeserializeEmpty() {
    ItemsSketch<String> sketch1 = new ItemsSketch<String>(1 << LG_MIN_MAP_SIZE);
    byte[] bytes = sketch1.toByteArray(new ArrayOfStringsSerDe());
    ItemsSketch<String> sketch2 = ItemsSketch.getInstance(Memory.wrap(bytes), new ArrayOfStringsSerDe());
    Assert.assertTrue(sketch2.isEmpty());
    Assert.assertEquals(sketch2.getNumActiveItems(), 0);
    Assert.assertEquals(sketch2.getStreamLength(), 0);
}
Also used : ArrayOfStringsSerDe(com.yahoo.sketches.ArrayOfStringsSerDe) Test(org.testng.annotations.Test)

Aggregations

ArrayOfStringsSerDe (com.yahoo.sketches.ArrayOfStringsSerDe)43 Test (org.testng.annotations.Test)37 DataByteArray (org.apache.pig.data.DataByteArray)23 Tuple (org.apache.pig.data.Tuple)19 WritableMemory (com.yahoo.memory.WritableMemory)15 DataBag (org.apache.pig.data.DataBag)12 Memory (com.yahoo.memory.Memory)10 ItemsSketch (com.yahoo.sketches.frequencies.ItemsSketch)9 SketchesArgumentException (com.yahoo.sketches.SketchesArgumentException)1