Examples with RoaringBitmap org.roaringbitmap.RoaringBitmap used on opensource projects

Example 41 with RoaringBitmap

use of org.roaringbitmap.RoaringBitmap in project carbondata by apache.

the class LuceneIndexWriter method addToCache.

public static void addToCache(LuceneColumnKeys key, int rowId, int pageId, int blockletId, Map<LuceneColumnKeys, Map<Integer, RoaringBitmap>> cache, ByteBuffer intBuffer, boolean storeBlockletWise) {
    Map<Integer, RoaringBitmap> setMap = cache.get(key);
    if (setMap == null) {
        setMap = new HashMap<>();
        cache.put(key, setMap);
    }
    int combinKey;
    if (!storeBlockletWise) {
        intBuffer.clear();
        intBuffer.putShort((short) blockletId);
        intBuffer.putShort((short) pageId);
        intBuffer.rewind();
        combinKey = intBuffer.getInt();
    } else {
        combinKey = pageId;
    }
    RoaringBitmap bitSet = setMap.get(combinKey);
    if (bitSet == null) {
        bitSet = new RoaringBitmap();
        setMap.put(combinKey, bitSet);
    }
    bitSet.add(rowId);
}

Example 42 with RoaringBitmap

use of org.roaringbitmap.RoaringBitmap in project carbondata by apache.

the class CarbonBloomFilter method readFields.

@Override
public void readFields(DataInput in) throws IOException {
    this.blockletNo = in.readInt();
    this.nbHash = in.readInt();
    this.hashType = in.readByte();
    this.vectorSize = in.readInt();
    this.compress = in.readBoolean();
    if (!compress) {
        int len = in.readInt();
        byte[] bytes = new byte[len];
        in.readFully(bytes);
        setBitSet(BitSet.valueOf(bytes));
    } else {
        this.bitmap = new RoaringBitmap();
        bitmap.deserialize(in);
    }
    this.hash = new HashFunction(this.vectorSize, this.nbHash, this.hashType);
}

Example 43 with RoaringBitmap

use of org.roaringbitmap.RoaringBitmap in project Gaffer by gchq.

the class RoaringBitmapSerialiser method serialise.

@Override
public byte[] serialise(final Object object) throws SerialisationException {
    RoaringBitmap value = (RoaringBitmap) object;
    ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
    DataOutputStream out = new DataOutputStream(byteOut);
    try {
        value.serialize(out);
    } catch (final IOException e) {
        throw new SerialisationException(e.getMessage(), e);
    }
    return byteOut.toByteArray();
}

Example 44 with RoaringBitmap

use of org.roaringbitmap.RoaringBitmap in project Gaffer by gchq.

the class RoaringBitmapSerialiser method deserialise.

@Override
public Object deserialise(final byte[] bytes) throws SerialisationException {
    RoaringBitmap value = new RoaringBitmap();
    byte[] convertedBytes = RoaringBitmapUtils.upConvertSerialisedForm(bytes);
    ByteArrayInputStream byteIn = new ByteArrayInputStream(convertedBytes);
    DataInputStream in = new DataInputStream(byteIn);
    try {
        value.deserialize(in);
    } catch (final IOException e) {
        throw new SerialisationException(e.getMessage(), e);
    }
    return value;
}

Example 45 with RoaringBitmap

use of org.roaringbitmap.RoaringBitmap in project tez by apache.

the class ShuffleUtils method generateVMEvent.

public static VertexManagerEvent generateVMEvent(OutputContext context, long[] sizePerPartition, boolean reportDetailedPartitionStats, Deflater deflater) throws IOException {
    ShuffleUserPayloads.VertexManagerEventPayloadProto.Builder vmBuilder = ShuffleUserPayloads.VertexManagerEventPayloadProto.newBuilder();
    long outputSize = context.getCounters().findCounter(TaskCounter.OUTPUT_BYTES).getValue();
    // Set this information only when required.  In pipelined shuffle,
    // multiple events would end up adding up to final output size.
    // This is needed for auto-reduce parallelism to work properly.
    vmBuilder.setOutputSize(outputSize);
    vmBuilder.setNumRecord(context.getCounters().findCounter(TaskCounter.OUTPUT_RECORDS).getValue() + context.getCounters().findCounter(TaskCounter.OUTPUT_LARGE_RECORDS).getValue());
    // set partition stats
    if (sizePerPartition != null && sizePerPartition.length > 0) {
        if (reportDetailedPartitionStats) {
            vmBuilder.setDetailedPartitionStats(getDetailedPartitionStatsForPhysicalOutput(sizePerPartition));
        } else {
            RoaringBitmap stats = getPartitionStatsForPhysicalOutput(sizePerPartition);
            DataOutputBuffer dout = new DataOutputBuffer();
            stats.serialize(dout);
            ByteString partitionStatsBytes = TezCommonUtils.compressByteArrayToByteString(dout.getData(), deflater);
            vmBuilder.setPartitionStats(partitionStatsBytes);
        }
    }
    VertexManagerEvent vmEvent = VertexManagerEvent.create(context.getDestinationVertexName(), vmBuilder.build().toByteString().asReadOnlyByteBuffer());
    return vmEvent;
}