Examples with NetworkBuffer org.apache.flink.runtime.io.network.buffer.NetworkBuffer used on opensource projects

Example 11 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class EventSerializer method toBufferConsumer.

public static BufferConsumer toBufferConsumer(AbstractEvent event, boolean hasPriority) throws IOException {
    final ByteBuffer serializedEvent = EventSerializer.toSerializedEvent(event);
    MemorySegment data = MemorySegmentFactory.wrap(serializedEvent.array());
    return new BufferConsumer(new NetworkBuffer(data, FreeingBufferRecycler.INSTANCE, getDataType(event, hasPriority)), data.size());
}

Example 12 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class BufferReaderWriterUtil method sliceNextBuffer.

@Nullable
static Buffer sliceNextBuffer(ByteBuffer memory) {
    final int remaining = memory.remaining();
    // buffer underflow exceptions which will cause the read to fail.
    if (remaining == 0) {
        return null;
    }
    final boolean isEvent = memory.getShort() == HEADER_VALUE_IS_EVENT;
    final boolean isCompressed = memory.getShort() == BUFFER_IS_COMPRESSED;
    final int size = memory.getInt();
    memory.limit(memory.position() + size);
    ByteBuffer buf = memory.slice();
    memory.position(memory.limit());
    memory.limit(memory.capacity());
    MemorySegment memorySegment = MemorySegmentFactory.wrapOffHeapMemory(buf);
    Buffer.DataType dataType = isEvent ? Buffer.DataType.EVENT_BUFFER : Buffer.DataType.DATA_BUFFER;
    return new NetworkBuffer(memorySegment, FreeingBufferRecycler.INSTANCE, dataType, isCompressed, size);
}

Example 13 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class NetworkBufferAllocator method allocateUnPooledNetworkBuffer.

/**
 * Allocates an un-pooled network buffer with the specific size.
 *
 * @param size The requested buffer size.
 * @param dataType The data type this buffer represents.
 * @return The un-pooled network buffer.
 */
Buffer allocateUnPooledNetworkBuffer(int size, Buffer.DataType dataType) {
    checkArgument(size > 0, "Illegal buffer size, must be positive.");
    byte[] byteArray = new byte[size];
    MemorySegment memSeg = MemorySegmentFactory.wrap(byteArray);
    return new NetworkBuffer(memSeg, FreeingBufferRecycler.INSTANCE, dataType);
}

Example 14 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class AsynchronousBufferFileWriterTest method testAddWithFailingWriter.

@Test
public void testAddWithFailingWriter() throws Exception {
    AsynchronousBufferFileWriter writer = new AsynchronousBufferFileWriter(ioManager.createChannel(), new RequestQueue<>());
    writer.close();
    exception.expect(IOException.class);
    Buffer buffer = new NetworkBuffer(MemorySegmentFactory.allocateUnpooledSegment(4096), FreeingBufferRecycler.INSTANCE);
    try {
        writer.writeBlock(buffer);
    } finally {
        if (!buffer.isRecycled()) {
            buffer.recycleBuffer();
            Assert.fail("buffer not recycled");
        }
        assertEquals("Shouln't increment number of outstanding requests.", 0, writer.getNumberOfOutstandingRequests());
    }
}

Example 15 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class PartitionedFileWriteReadTest method testNotWriteDataOfTheSameSubpartitionTogether.

@Test(expected = IllegalStateException.class)
public void testNotWriteDataOfTheSameSubpartitionTogether() throws Exception {
    PartitionedFileWriter partitionedFileWriter = createPartitionedFileWriter(2);
    try {
        MemorySegment segment = MemorySegmentFactory.allocateUnpooledSegment(1024);
        NetworkBuffer buffer1 = new NetworkBuffer(segment, (buf) -> {
        });
        partitionedFileWriter.writeBuffers(getBufferWithChannels(buffer1, 1));
        NetworkBuffer buffer2 = new NetworkBuffer(segment, (buf) -> {
        });
        partitionedFileWriter.writeBuffers(getBufferWithChannels(buffer2, 0));
        NetworkBuffer buffer3 = new NetworkBuffer(segment, (buf) -> {
        });
        partitionedFileWriter.writeBuffers(getBufferWithChannels(buffer3, 1));
    } finally {
        partitionedFileWriter.finish();
    }
}