Examples with HyperLogLog com.facebook.airlift.stats.cardinality.HyperLogLog used on opensource projects

Example 11 with HyperLogLog

use of com.facebook.airlift.stats.cardinality.HyperLogLog in project presto by prestodb.

the class ApproximateCountDistinctAggregation method input.

@InputFunction
@TypeParameter("T")
public static void input(@OperatorDependency(operator = XX_HASH_64, argumentTypes = { "T" }) MethodHandle methodHandle, @AggregationState HyperLogLogState state, @SqlType("T") double value, @SqlType(StandardTypes.DOUBLE) double maxStandardError) {
    HyperLogLog hll = HyperLogLogUtils.getOrCreateHyperLogLog(state, maxStandardError);
    state.addMemoryUsage(-hll.estimatedInMemorySize());
    long hash;
    try {
        hash = (long) methodHandle.invokeExact(value);
    } catch (Throwable t) {
        throw internalError(t);
    }
    hll.addHash(hash);
    state.addMemoryUsage(hll.estimatedInMemorySize());
}

Example 12 with HyperLogLog

use of com.facebook.airlift.stats.cardinality.HyperLogLog in project presto by prestodb.

the class ApproximateCountDistinctAggregation method input.

@InputFunction
@TypeParameter("T")
public static void input(@OperatorDependency(operator = XX_HASH_64, argumentTypes = { "T" }) MethodHandle methodHandle, @AggregationState HyperLogLogState state, @SqlType("T") long value, @SqlType(StandardTypes.DOUBLE) double maxStandardError) {
    HyperLogLog hll = HyperLogLogUtils.getOrCreateHyperLogLog(state, maxStandardError);
    state.addMemoryUsage(-hll.estimatedInMemorySize());
    long hash;
    try {
        hash = (long) methodHandle.invokeExact(value);
    } catch (Throwable t) {
        throw internalError(t);
    }
    hll.addHash(hash);
    state.addMemoryUsage(hll.estimatedInMemorySize());
}

Example 13 with HyperLogLog

use of com.facebook.airlift.stats.cardinality.HyperLogLog in project presto by prestodb.

the class ApproximateCountDistinctAggregation method input.

@InputFunction
@TypeParameter("T")
public static void input(@OperatorDependency(operator = XX_HASH_64, argumentTypes = { "T" }) MethodHandle methodHandle, @AggregationState HyperLogLogState state, @SqlType("T") Slice value, @SqlType(StandardTypes.DOUBLE) double maxStandardError) {
    HyperLogLog hll = HyperLogLogUtils.getOrCreateHyperLogLog(state, maxStandardError);
    state.addMemoryUsage(-hll.estimatedInMemorySize());
    long hash;
    try {
        hash = (long) methodHandle.invokeExact(value);
    } catch (Throwable t) {
        throw internalError(t);
    }
    hll.addHash(hash);
    state.addMemoryUsage(hll.estimatedInMemorySize());
}

Example 14 with HyperLogLog

use of com.facebook.airlift.stats.cardinality.HyperLogLog in project presto by prestodb.

the class HyperLogLogUtils method getOrCreateHyperLogLog.

public static HyperLogLog getOrCreateHyperLogLog(HyperLogLogState state, double maxStandardError) {
    HyperLogLog hll = state.getHyperLogLog();
    if (hll == null) {
        hll = HyperLogLog.newInstance(standardErrorToBuckets(maxStandardError));
        state.setHyperLogLog(hll);
        state.addMemoryUsage(hll.estimatedInMemorySize());
    }
    return hll;
}

Example 15 with HyperLogLog

use of com.facebook.airlift.stats.cardinality.HyperLogLog in project presto by prestodb.

the class HyperLogLogUtils method mergeState.

public static void mergeState(@AggregationState HyperLogLogState state, HyperLogLog input) {
    HyperLogLog previous = state.getHyperLogLog();
    if (previous == null) {
        state.setHyperLogLog(input);
        state.addMemoryUsage(input.estimatedInMemorySize());
    } else {
        state.addMemoryUsage(-previous.estimatedInMemorySize());
        try {
            // Throws if the bucket counts are different.
            // We currently cannot access these counts from HLL so we must
            // catch and rethrow a more useful exception
            previous.mergeWith(input);
        } catch (IllegalArgumentException e) {
            throw new PrestoException(INVALID_FUNCTION_ARGUMENT, e.getMessage(), e);
        }
        state.addMemoryUsage(previous.estimatedInMemorySize());
    }
}