Examples with IndexedInts org.apache.druid.segment.data.IndexedInts used on opensource projects

Example 11 with IndexedInts

use of org.apache.druid.segment.data.IndexedInts in project druid by druid-io.

the class Generic2AggPooledTopNScannerPrototype method scanAndAggregate.

/**
 * Any changes to this method should be coordinated with {@link TopNUtils}, {@link
 * PooledTopNAlgorithm#computeSpecializedScanAndAggregateImplementations} and downstream methods.
 *
 * It should be checked with a tool like https://github.com/AdoptOpenJDK/jitwatch that C2 compiler output for this
 * method doesn't have any method calls in the while loop, i. e. all method calls are inlined. To be able to see
 * assembly of this method in JITWatch and other similar tools, {@link
 * PooledTopNAlgorithm#SPECIALIZE_GENERIC_TWO_AGG_POOLED_TOPN} should be turned off. Note that in this case the benchmark
 * should be "naturally monomorphic", i. e. execute this method always with the same runtime shape.
 *
 * If the while loop contains not inlined method calls, it should be considered as a performance bug.
 */
@Override
public long scanAndAggregate(DimensionSelector dimensionSelector, BufferAggregator aggregator1, int aggregator1Size, BufferAggregator aggregator2, int aggregator2Size, Cursor cursor, int[] positions, ByteBuffer resultsBuffer) {
    int totalAggregatorsSize = aggregator1Size + aggregator2Size;
    long processedRows = 0;
    int positionToAllocate = 0;
    while (!cursor.isDoneOrInterrupted()) {
        final IndexedInts dimValues = dimensionSelector.getRow();
        final int dimSize = dimValues.size();
        for (int i = 0; i < dimSize; i++) {
            int dimIndex = dimValues.get(i);
            int position = positions[dimIndex];
            if (position >= 0) {
                aggregator1.aggregate(resultsBuffer, position);
                aggregator2.aggregate(resultsBuffer, position + aggregator1Size);
            } else if (position == TopNAlgorithm.INIT_POSITION_VALUE) {
                positions[dimIndex] = positionToAllocate;
                position = positionToAllocate;
                aggregator1.init(resultsBuffer, position);
                aggregator1.aggregate(resultsBuffer, position);
                position += aggregator1Size;
                aggregator2.init(resultsBuffer, position);
                aggregator2.aggregate(resultsBuffer, position);
                positionToAllocate += totalAggregatorsSize;
            }
        }
        processedRows++;
        cursor.advanceUninterruptibly();
    }
    return processedRows;
}

Example 12 with IndexedInts

use of org.apache.druid.segment.data.IndexedInts in project druid by druid-io.

the class StringTopNColumnAggregatesProcessor method scanAndAggregateWithCardinalityUnknown.

private long scanAndAggregateWithCardinalityUnknown(TopNQuery query, Cursor cursor, DimensionSelector selector) {
    long processedRows = 0;
    while (!cursor.isDone()) {
        final IndexedInts dimValues = selector.getRow();
        for (int i = 0, size = dimValues.size(); i < size; ++i) {
            final int dimIndex = dimValues.get(i);
            final Comparable<?> key = dimensionValueConverter.apply(selector.lookupName(dimIndex));
            Aggregator[] aggs = aggregatesStore.computeIfAbsent(key, k -> BaseTopNAlgorithm.makeAggregators(cursor, query.getAggregatorSpecs()));
            for (Aggregator aggregator : aggs) {
                aggregator.aggregate();
            }
        }
        cursor.advance();
        processedRows++;
    }
    return processedRows;
}

Example 13 with IndexedInts

use of org.apache.druid.segment.data.IndexedInts in project druid by druid-io.

the class FilteredAggregatorTest method makeColumnSelector.

private ColumnSelectorFactory makeColumnSelector(final TestFloatColumnSelector selector) {
    return new ColumnSelectorFactory() {

        @Override
        public DimensionSelector makeDimensionSelector(DimensionSpec dimensionSpec) {
            final String dimensionName = dimensionSpec.getDimension();
            if ("dim".equals(dimensionName)) {
                return dimensionSpec.decorate(new AbstractDimensionSelector() {

                    @Override
                    public IndexedInts getRow() {
                        SingleIndexedInt row = new SingleIndexedInt();
                        if (selector.getIndex() % 3 == 2) {
                            row.setValue(1);
                        } else {
                            row.setValue(0);
                        }
                        return row;
                    }

                    @Override
                    public ValueMatcher makeValueMatcher(String value) {
                        return DimensionSelectorUtils.makeValueMatcherGeneric(this, value);
                    }

                    @Override
                    public ValueMatcher makeValueMatcher(Predicate<String> predicate) {
                        return DimensionSelectorUtils.makeValueMatcherGeneric(this, predicate);
                    }

                    @Override
                    public int getValueCardinality() {
                        return 2;
                    }

                    @Override
                    public String lookupName(int id) {
                        switch(id) {
                            case 0:
                                return "a";
                            case 1:
                                return "b";
                            default:
                                throw new IllegalArgumentException();
                        }
                    }

                    @Override
                    public boolean nameLookupPossibleInAdvance() {
                        return true;
                    }

                    @Nullable
                    @Override
                    public IdLookup idLookup() {
                        return new IdLookup() {

                            @Override
                            public int lookupId(String name) {
                                switch(name) {
                                    case "a":
                                        return 0;
                                    case "b":
                                        return 1;
                                    default:
                                        throw new IllegalArgumentException();
                                }
                            }
                        };
                    }

                    @Override
                    public Class classOfObject() {
                        return Object.class;
                    }

                    @Override
                    public void inspectRuntimeShape(RuntimeShapeInspector inspector) {
                    // Don't care about runtime shape in tests
                    }
                });
            } else {
                throw new UnsupportedOperationException();
            }
        }

        @Override
        public ColumnValueSelector<?> makeColumnValueSelector(String columnName) {
            if ("value".equals(columnName)) {
                return selector;
            } else {
                throw new UnsupportedOperationException();
            }
        }

        @Override
        public ColumnCapabilities getColumnCapabilities(String columnName) {
            ColumnCapabilitiesImpl caps;
            if ("value".equals(columnName)) {
                caps = new ColumnCapabilitiesImpl();
                caps.setType(ColumnType.FLOAT);
                caps.setDictionaryEncoded(false);
                caps.setHasBitmapIndexes(false);
            } else {
                caps = new ColumnCapabilitiesImpl();
                caps.setType(ColumnType.STRING);
                caps.setDictionaryEncoded(true);
                caps.setHasBitmapIndexes(true);
            }
            return caps;
        }
    };
}

Example 14 with IndexedInts

use of org.apache.druid.segment.data.IndexedInts in project druid by druid-io.

the class MultiValueStringCardinalityVectorProcessor method aggregate.

@Override
public void aggregate(ByteBuffer buf, int position, int startRow, int endRow) {
    // Save position, limit and restore later instead of allocating a new ByteBuffer object
    final int oldPosition = buf.position();
    final int oldLimit = buf.limit();
    try {
        final IndexedInts[] vector = selector.getRowVector();
        buf.limit(position + HyperLogLogCollector.getLatestNumBytesForDenseStorage());
        buf.position(position);
        final HyperLogLogCollector collector = HyperLogLogCollector.makeCollector(buf);
        for (int i = startRow; i < endRow; i++) {
            final IndexedInts ids = vector[i];
            final int sz = ids.size();
            for (int j = 0; j < sz; j++) {
                final String value = selector.lookupName(ids.get(j));
                StringCardinalityAggregatorColumnSelectorStrategy.addStringToCollector(collector, value);
            }
        }
    } finally {
        buf.limit(oldLimit);
        buf.position(oldPosition);
    }
}

Example 15 with IndexedInts

use of org.apache.druid.segment.data.IndexedInts in project druid by druid-io.

the class StringCardinalityAggregatorColumnSelectorStrategy method hashValues.

@Override
public void hashValues(DimensionSelector dimSelector, HyperLogLogCollector collector) {
    IndexedInts row = dimSelector.getRow();
    for (int i = 0, rowSize = row.size(); i < rowSize; i++) {
        int index = row.get(i);
        final String value = dimSelector.lookupName(index);
        addStringToCollector(collector, value);
    }
}