Search in sources :

Example 26 with ArrayOfDoublesSketch

use of com.yahoo.sketches.tuple.ArrayOfDoublesSketch in project sketches-pig by DataSketches.

the class ArrayOfDoublesSketchToVariances method exec.

@Override
public Tuple exec(final Tuple input) throws IOException {
    if ((input == null) || (input.size() == 0)) {
        return null;
    }
    final DataByteArray dba = (DataByteArray) input.get(0);
    final ArrayOfDoublesSketch sketch = ArrayOfDoublesSketches.wrapSketch(Memory.wrap(dba.get()));
    if (sketch.getRetainedEntries() < 1) {
        return null;
    }
    final SummaryStatistics[] summaries = ArrayOfDoublesSketchStats.sketchToSummaryStatistics(sketch);
    final Tuple variances = TupleFactory.getInstance().newTuple(sketch.getNumValues());
    for (int i = 0; i < sketch.getNumValues(); i++) {
        variances.set(i, summaries[i].getVariance());
    }
    return variances;
}
Also used : SummaryStatistics(org.apache.commons.math3.stat.descriptive.SummaryStatistics) DataByteArray(org.apache.pig.data.DataByteArray) ArrayOfDoublesSketch(com.yahoo.sketches.tuple.ArrayOfDoublesSketch) Tuple(org.apache.pig.data.Tuple)

Aggregations

ArrayOfDoublesSketch (com.yahoo.sketches.tuple.ArrayOfDoublesSketch)26 DataByteArray (org.apache.pig.data.DataByteArray)26 Tuple (org.apache.pig.data.Tuple)22 Test (org.testng.annotations.Test)19 DataBag (org.apache.pig.data.DataBag)12 ArrayOfDoublesUpdatableSketch (com.yahoo.sketches.tuple.ArrayOfDoublesUpdatableSketch)8 ArrayOfDoublesUpdatableSketchBuilder (com.yahoo.sketches.tuple.ArrayOfDoublesUpdatableSketchBuilder)8 SummaryStatistics (org.apache.commons.math3.stat.descriptive.SummaryStatistics)3 ArrayOfDoublesSketchIterator (com.yahoo.sketches.tuple.ArrayOfDoublesSketchIterator)2 DoublesSketchBuilder (com.yahoo.sketches.quantiles.DoublesSketchBuilder)1 UpdateDoublesSketch (com.yahoo.sketches.quantiles.UpdateDoublesSketch)1 Random (java.util.Random)1 TTest (org.apache.commons.math3.stat.inference.TTest)1