Search in sources :

Example 1 with Accumulator

use of com.facebook.presto.operator.aggregation.Accumulator in project presto by prestodb.

the class TestSpatialPartitioningInternalAggregation method test.

@Test(dataProvider = "partitionCount")
public void test(int partitionCount) {
    InternalAggregationFunction function = getFunction();
    List<OGCGeometry> geometries = makeGeometries();
    Block geometryBlock = makeGeometryBlock(geometries);
    Block partitionCountBlock = BlockAssertions.createRLEBlock(partitionCount, geometries.size());
    String expectedValue = getSpatialPartitioning(geometries, partitionCount);
    AccumulatorFactory accumulatorFactory = function.bind(Ints.asList(0, 1), Optional.empty());
    Page page = new Page(geometryBlock, partitionCountBlock);
    Accumulator accumulator = accumulatorFactory.createAccumulator(UpdateMemory.NOOP);
    accumulator.addInput(page);
    String aggregation = (String) BlockAssertions.getOnlyValue(accumulator.getFinalType(), getFinalBlock(accumulator));
    assertEquals(aggregation, expectedValue);
    GroupedAccumulator groupedAggregation = accumulatorFactory.createGroupedAccumulator(UpdateMemory.NOOP);
    groupedAggregation.addInput(createGroupByIdBlock(0, page.getPositionCount()), page);
    String groupValue = (String) getGroupValue(groupedAggregation, 0);
    assertEquals(groupValue, expectedValue);
}
Also used : OGCGeometry(com.esri.core.geometry.ogc.OGCGeometry) GroupedAccumulator(com.facebook.presto.operator.aggregation.GroupedAccumulator) Accumulator(com.facebook.presto.operator.aggregation.Accumulator) AccumulatorFactory(com.facebook.presto.operator.aggregation.AccumulatorFactory) AggregationTestUtils.createGroupByIdBlock(com.facebook.presto.operator.aggregation.AggregationTestUtils.createGroupByIdBlock) AggregationTestUtils.getFinalBlock(com.facebook.presto.operator.aggregation.AggregationTestUtils.getFinalBlock) Block(com.facebook.presto.common.block.Block) Page(com.facebook.presto.common.Page) InternalAggregationFunction(com.facebook.presto.operator.aggregation.InternalAggregationFunction) GroupedAccumulator(com.facebook.presto.operator.aggregation.GroupedAccumulator) Test(org.testng.annotations.Test)

Example 2 with Accumulator

use of com.facebook.presto.operator.aggregation.Accumulator in project presto by prestodb.

the class TestSpatialPartitioningInternalAggregation method testEmptyPartitionException.

@Test
public void testEmptyPartitionException() {
    InternalAggregationFunction function = getFunction();
    Block geometryBlock = GEOMETRY.createBlockBuilder(null, 0).build();
    Block partitionCountBlock = BlockAssertions.createRLEBlock(10, 0);
    Page page = new Page(geometryBlock, partitionCountBlock);
    AccumulatorFactory accumulatorFactory = function.bind(Ints.asList(0, 1), Optional.empty());
    Accumulator accumulator = accumulatorFactory.createAccumulator(UpdateMemory.NOOP);
    accumulator.addInput(page);
    try {
        getFinalBlock(accumulator);
        fail("Should fail creating spatial partition with no rows.");
    } catch (PrestoException e) {
        assertEquals(e.getErrorCode(), INVALID_FUNCTION_ARGUMENT.toErrorCode());
        assertEquals(e.getMessage(), "No rows supplied to spatial partition.");
    }
}
Also used : GroupedAccumulator(com.facebook.presto.operator.aggregation.GroupedAccumulator) Accumulator(com.facebook.presto.operator.aggregation.Accumulator) AccumulatorFactory(com.facebook.presto.operator.aggregation.AccumulatorFactory) AggregationTestUtils.createGroupByIdBlock(com.facebook.presto.operator.aggregation.AggregationTestUtils.createGroupByIdBlock) AggregationTestUtils.getFinalBlock(com.facebook.presto.operator.aggregation.AggregationTestUtils.getFinalBlock) Block(com.facebook.presto.common.block.Block) Page(com.facebook.presto.common.Page) PrestoException(com.facebook.presto.spi.PrestoException) InternalAggregationFunction(com.facebook.presto.operator.aggregation.InternalAggregationFunction) Test(org.testng.annotations.Test)

Example 3 with Accumulator

use of com.facebook.presto.operator.aggregation.Accumulator in project presto by prestodb.

the class TestEvaluateClassifierPredictions method testEvaluateClassifierPredictions.

@Test
public void testEvaluateClassifierPredictions() {
    metadata.registerBuiltInFunctions(extractFunctions(new MLPlugin().getFunctions()));
    InternalAggregationFunction aggregation = functionAndTypeManager.getAggregateFunctionImplementation(functionAndTypeManager.lookupFunction("evaluate_classifier_predictions", fromTypes(BIGINT, BIGINT)));
    Accumulator accumulator = aggregation.bind(ImmutableList.of(0, 1), Optional.empty()).createAccumulator(UpdateMemory.NOOP);
    accumulator.addInput(getPage());
    BlockBuilder finalOut = accumulator.getFinalType().createBlockBuilder(null, 1);
    accumulator.evaluateFinal(finalOut);
    Block block = finalOut.build();
    String output = VARCHAR.getSlice(block, 0).toStringUtf8();
    List<String> parts = ImmutableList.copyOf(Splitter.on('\n').omitEmptyStrings().split(output));
    assertEquals(parts.size(), 7, output);
    assertEquals(parts.get(0), "Accuracy: 1/2 (50.00%)");
}
Also used : Accumulator(com.facebook.presto.operator.aggregation.Accumulator) Block(com.facebook.presto.common.block.Block) InternalAggregationFunction(com.facebook.presto.operator.aggregation.InternalAggregationFunction) BlockBuilder(com.facebook.presto.common.block.BlockBuilder) Test(org.testng.annotations.Test)

Example 4 with Accumulator

use of com.facebook.presto.operator.aggregation.Accumulator in project presto by prestodb.

the class HashAggregationOperator method getGlobalAggregationOutput.

private Page getGlobalAggregationOutput() {
    List<Accumulator> accumulators = accumulatorFactories.stream().map(accumulatorFactory -> accumulatorFactory.createAccumulator(UpdateMemory.NOOP)).collect(Collectors.toList());
    // global aggregation output page will only be constructed once,
    // so a new PageBuilder is constructed (instead of using PageBuilder.reset)
    PageBuilder output = new PageBuilder(globalAggregationGroupIds.size(), types);
    for (int groupId : globalAggregationGroupIds) {
        output.declarePosition();
        int channel = 0;
        for (; channel < groupByTypes.size(); channel++) {
            if (channel == groupIdChannel.get()) {
                output.getBlockBuilder(channel).writeLong(groupId);
            } else {
                output.getBlockBuilder(channel).appendNull();
            }
        }
        if (hashChannel.isPresent()) {
            long hashValue = calculateDefaultOutputHash(groupByTypes, groupIdChannel.get(), groupId);
            output.getBlockBuilder(channel++).writeLong(hashValue);
        }
        for (int j = 0; j < accumulators.size(); channel++, j++) {
            if (step.isOutputPartial()) {
                accumulators.get(j).evaluateIntermediate(output.getBlockBuilder(channel));
            } else {
                accumulators.get(j).evaluateFinal(output.getBlockBuilder(channel));
            }
        }
    }
    if (output.isEmpty()) {
        return null;
    }
    return output.build();
}
Also used : Accumulator(com.facebook.presto.operator.aggregation.Accumulator) PlanNodeId(com.facebook.presto.spi.plan.PlanNodeId) Page(com.facebook.presto.common.Page) HashAggregationBuilder(com.facebook.presto.operator.aggregation.builder.HashAggregationBuilder) ListenableFuture(com.google.common.util.concurrent.ListenableFuture) InMemoryHashAggregationBuilder(com.facebook.presto.operator.aggregation.builder.InMemoryHashAggregationBuilder) MEGABYTE(io.airlift.units.DataSize.Unit.MEGABYTE) PageBuilder(com.facebook.presto.common.PageBuilder) CombineHashFunction(com.facebook.presto.operator.scalar.CombineHashFunction) ImmutableList(com.google.common.collect.ImmutableList) AccumulatorFactory(com.facebook.presto.operator.aggregation.AccumulatorFactory) NULL_HASH_CODE(com.facebook.presto.type.TypeUtils.NULL_HASH_CODE) Verify.verify(com.google.common.base.Verify.verify) Objects.requireNonNull(java.util.Objects.requireNonNull) Type(com.facebook.presto.common.type.Type) Step(com.facebook.presto.spi.plan.AggregationNode.Step) BigintType(com.facebook.presto.common.type.BigintType) InMemoryHashAggregationBuilder.toTypes(com.facebook.presto.operator.aggregation.builder.InMemoryHashAggregationBuilder.toTypes) SpillableHashAggregationBuilder(com.facebook.presto.operator.aggregation.builder.SpillableHashAggregationBuilder) Collectors(java.util.stream.Collectors) INITIAL_HASH_VALUE(com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer.INITIAL_HASH_VALUE) Preconditions.checkState(com.google.common.base.Preconditions.checkState) Accumulator(com.facebook.presto.operator.aggregation.Accumulator) DataSize(io.airlift.units.DataSize) List(java.util.List) Optional(java.util.Optional) SpillerFactory(com.facebook.presto.spiller.SpillerFactory) VisibleForTesting(com.google.common.annotations.VisibleForTesting) JoinCompiler(com.facebook.presto.sql.gen.JoinCompiler) PageBuilder(com.facebook.presto.common.PageBuilder)

Aggregations

Accumulator (com.facebook.presto.operator.aggregation.Accumulator)4 Page (com.facebook.presto.common.Page)3 Block (com.facebook.presto.common.block.Block)3 AccumulatorFactory (com.facebook.presto.operator.aggregation.AccumulatorFactory)3 InternalAggregationFunction (com.facebook.presto.operator.aggregation.InternalAggregationFunction)3 Test (org.testng.annotations.Test)3 AggregationTestUtils.createGroupByIdBlock (com.facebook.presto.operator.aggregation.AggregationTestUtils.createGroupByIdBlock)2 AggregationTestUtils.getFinalBlock (com.facebook.presto.operator.aggregation.AggregationTestUtils.getFinalBlock)2 GroupedAccumulator (com.facebook.presto.operator.aggregation.GroupedAccumulator)2 OGCGeometry (com.esri.core.geometry.ogc.OGCGeometry)1 PageBuilder (com.facebook.presto.common.PageBuilder)1 BlockBuilder (com.facebook.presto.common.block.BlockBuilder)1 BigintType (com.facebook.presto.common.type.BigintType)1 Type (com.facebook.presto.common.type.Type)1 HashAggregationBuilder (com.facebook.presto.operator.aggregation.builder.HashAggregationBuilder)1 InMemoryHashAggregationBuilder (com.facebook.presto.operator.aggregation.builder.InMemoryHashAggregationBuilder)1 InMemoryHashAggregationBuilder.toTypes (com.facebook.presto.operator.aggregation.builder.InMemoryHashAggregationBuilder.toTypes)1 SpillableHashAggregationBuilder (com.facebook.presto.operator.aggregation.builder.SpillableHashAggregationBuilder)1 CombineHashFunction (com.facebook.presto.operator.scalar.CombineHashFunction)1 PrestoException (com.facebook.presto.spi.PrestoException)1