Examples with RoaringBitmap org.roaringbitmap.RoaringBitmap used on opensource projects

Example 31 with RoaringBitmap

use of org.roaringbitmap.RoaringBitmap in project narchy by automenta.

the class Conj method addIfValid.

protected boolean addIfValid(long at, int id) {
    Object what = event.getIfAbsentPut(at, () -> new byte[ROARING_UPGRADE_THRESH]);
    if (what instanceof RoaringBitmap) {
        RoaringBitmap r = (RoaringBitmap) what;
        if (!r.contains(-id)) {
            r.add(id);
            return true;
        }
    } else {
        byte[] ii = (byte[]) what;
        if (indexOfZeroTerminated(ii, ((byte) -id)) == -1) {
            int nextSlot = indexOfZeroTerminated(ii, (byte) 0);
            if (nextSlot != -1) {
                ii[nextSlot] = (byte) id;
            } else {
                // upgrade to roaring and add
                RoaringBitmap rb = new RoaringBitmap();
                for (byte b : ii) rb.add(b);
                rb.add(id);
                event.put(at, rb);
            }
            return true;
        }
    }
    return false;
}

Example 32 with RoaringBitmap

use of org.roaringbitmap.RoaringBitmap in project narchy by automenta.

the class SpaceKeys method accept.

@Override
public void accept(JoglWindow j) {
    RoaringBitmap queue = this.queue;
    if (!queue.isEmpty()) {
        float dt = j.dtMS() / 1000f;
        synchronized (on) {
            this.queue = new RoaringBitmap();
        }
        queue.forEach((int k) -> {
            // shouldnt ever be zero actually
            boolean s = k >= 0;
            FloatProcedure f = ((s) ? keyPressed : keyReleased).get(Math.abs(k));
            if (f != null)
                f.value(dt);
        });
    }
}

Example 33 with RoaringBitmap

use of org.roaringbitmap.RoaringBitmap in project tez by apache.

the class TestVertexImpl method getVertexManagerEvent.

private VertexManagerEvent getVertexManagerEvent(long[] sizes, long totalSize, Vertex vertex) throws IOException {
    ByteBuffer payload = null;
    if (sizes != null) {
        RoaringBitmap partitionStats = ShuffleUtils.getPartitionStatsForPhysicalOutput(sizes);
        DataOutputBuffer dout = new DataOutputBuffer();
        partitionStats.serialize(dout);
        ByteString partitionStatsBytes = TezCommonUtils.compressByteArrayToByteString(dout.getData());
        payload = ShuffleUserPayloads.VertexManagerEventPayloadProto.newBuilder().setOutputSize(totalSize).setPartitionStats(partitionStatsBytes).build().toByteString().asReadOnlyByteBuffer();
    } else {
        payload = ShuffleUserPayloads.VertexManagerEventPayloadProto.newBuilder().setOutputSize(totalSize).build().toByteString().asReadOnlyByteBuffer();
    }
    VertexManagerEvent vmEvent = VertexManagerEvent.create(vertex.getName(), payload);
    return vmEvent;
}

Example 34 with RoaringBitmap

use of org.roaringbitmap.RoaringBitmap in project tez by apache.

the class ShuffleUtils method getPartitionStatsForPhysicalOutput.

/**
 * Data size for the destinations
 *
 * @param sizes for physical outputs
 */
public static RoaringBitmap getPartitionStatsForPhysicalOutput(long[] sizes) {
    RoaringBitmap partitionStats = new RoaringBitmap();
    if (sizes == null || sizes.length == 0) {
        return partitionStats;
    }
    final int RANGE_LEN = DATA_RANGE_IN_MB.values().length;
    for (int i = 0; i < sizes.length; i++) {
        int bucket = DATA_RANGE_IN_MB.getRange(sizes[i]).ordinal();
        int index = i * (RANGE_LEN);
        partitionStats.add(index + bucket);
    }
    return partitionStats;
}

Example 35 with RoaringBitmap

use of org.roaringbitmap.RoaringBitmap in project presto by prestodb.

the class HiveManifestUtils method compressFileNamesUsingRoaringBitmap.

private static String compressFileNamesUsingRoaringBitmap(List<String> fileNames) {
    RoaringBitmap roaringBitmap = new RoaringBitmap();
    // Add file names to roaring bitmap
    fileNames.forEach(name -> roaringBitmap.add(Integer.parseInt(name)));
    // Serialize the compressed data into ByteBuffer
    ByteBuffer byteBuffer = ByteBuffer.allocate(roaringBitmap.serializedSizeInBytes());
    roaringBitmap.serialize(byteBuffer);
    byteBuffer.flip();
    return new String(byteBuffer.array(), ISO_8859_1);
}