Examples with NetworkBuffer org.apache.flink.runtime.io.network.buffer.NetworkBuffer used on opensource projects

Example 36 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class BoundedBlockingSubpartitionWriteReadTest method writeLongs.

// ------------------------------------------------------------------------
// utils
// ------------------------------------------------------------------------
private static void writeLongs(BoundedBlockingSubpartition partition, long nums) throws IOException {
    final MemorySegment memory = MemorySegmentFactory.allocateUnpooledSegment(BUFFER_SIZE);
    long l = 0;
    while (nums > 0) {
        int pos = 0;
        for (; nums > 0 && pos <= memory.size() - 8; pos += 8) {
            memory.putLongBigEndian(pos, l++);
            nums--;
        }
        partition.add(new BufferConsumer(new NetworkBuffer(memory, (ignored) -> {
        }, Buffer.DataType.DATA_BUFFER), pos));
        // we need to flush after every buffer as long as the add() contract is that
        // buffer are immediately added and can be filled further after that (for low latency
        // streaming data exchanges)
        partition.flush();
    }
}

Example 37 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class DataBufferTest method copyIntoSegment.

private BufferWithChannel copyIntoSegment(int bufferSize, DataBuffer dataBuffer) {
    if (useHashBuffer) {
        BufferWithChannel buffer = dataBuffer.getNextBuffer(null);
        if (buffer == null || !buffer.getBuffer().isBuffer()) {
            return buffer;
        }
        MemorySegment segment = MemorySegmentFactory.allocateUnpooledSegment(bufferSize);
        int numBytes = buffer.getBuffer().readableBytes();
        segment.put(0, buffer.getBuffer().getNioBufferReadable(), numBytes);
        buffer.getBuffer().recycleBuffer();
        return new BufferWithChannel(new NetworkBuffer(segment, MemorySegment::free, DataType.DATA_BUFFER, numBytes), buffer.getChannelIndex());
    } else {
        MemorySegment segment = MemorySegmentFactory.allocateUnpooledSegment(bufferSize);
        return dataBuffer.getNextBuffer(segment);
    }
}

Example 38 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class NettyMessageClientDecoderDelegateTest method createDataBuffer.

private Buffer createDataBuffer(int size, Buffer.DataType dataType) {
    MemorySegment segment = MemorySegmentFactory.allocateUnpooledSegment(size);
    NetworkBuffer buffer = new NetworkBuffer(segment, FreeingBufferRecycler.INSTANCE, dataType);
    for (int i = 0; i < size / 4; ++i) {
        buffer.writeInt(i);
    }
    return buffer;
}

Example 39 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class CompressedBlockChannelWriter method compressBuffer.

private void compressBuffer(ByteBuffer buffer, int len) throws IOException {
    MemorySegment compressedBuffer;
    try {
        compressedBuffer = compressedBuffers.take();
    } catch (InterruptedException e) {
        throw new IOException(e);
    }
    int compressedLen = compressedBuffer.processAsByteBuffer((dstBuffer) -> {
        return compressor.compress(buffer, 0, len, dstBuffer, 0);
    });
    NetworkBuffer networkBuffer = new NetworkBuffer(compressedBuffer, this);
    networkBuffer.setSize(compressedLen);
    writer.writeBlock(networkBuffer);
}