Examples with NetworkBuffer org.apache.flink.runtime.io.network.buffer.NetworkBuffer used on opensource projects

Example 31 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class BufferReaderWriterUtil method readFromByteChannel.

@Nullable
static Buffer readFromByteChannel(FileChannel channel, ByteBuffer headerBuffer, MemorySegment memorySegment, BufferRecycler bufferRecycler) throws IOException {
    headerBuffer.clear();
    if (!tryReadByteBuffer(channel, headerBuffer)) {
        return null;
    }
    headerBuffer.flip();
    final ByteBuffer targetBuf;
    final boolean isEvent;
    final boolean isCompressed;
    final int size;
    try {
        isEvent = headerBuffer.getShort() == HEADER_VALUE_IS_EVENT;
        isCompressed = headerBuffer.getShort() == BUFFER_IS_COMPRESSED;
        size = headerBuffer.getInt();
        targetBuf = memorySegment.wrap(0, size);
    } catch (BufferUnderflowException | IllegalArgumentException e) {
        // buffer underflow if header buffer is undersized
        // IllegalArgumentException if size is outside memory segment size
        throwCorruptDataException();
        // silence compiler
        return null;
    }
    readByteBufferFully(channel, targetBuf);
    Buffer.DataType dataType = isEvent ? Buffer.DataType.EVENT_BUFFER : Buffer.DataType.DATA_BUFFER;
    return new NetworkBuffer(memorySegment, bufferRecycler, dataType, isCompressed, size);
}

Example 32 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class ChannelStateWriteRequestDispatcherImplTest method testBuffersRecycled.

private void testBuffersRecycled(Function<NetworkBuffer[], ChannelStateWriteRequest> requestBuilder) throws Exception {
    ChannelStateWriteRequestDispatcher dispatcher = new ChannelStateWriteRequestDispatcherImpl("dummy task", 0, new MemoryBackendCheckpointStorageAccess(new JobID(), null, null, 1), new ChannelStateSerializerImpl());
    ChannelStateWriteResult result = new ChannelStateWriteResult();
    dispatcher.dispatch(ChannelStateWriteRequest.start(1L, result, CheckpointStorageLocationReference.getDefault()));
    result.getResultSubpartitionStateHandles().completeExceptionally(new TestException());
    result.getInputChannelStateHandles().completeExceptionally(new TestException());
    NetworkBuffer[] buffers = new NetworkBuffer[] { buffer(), buffer() };
    dispatcher.dispatch(requestBuilder.apply(buffers));
    for (NetworkBuffer buffer : buffers) {
        assertTrue(buffer.isRecycled());
    }
}

Example 33 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class ChannelPersistenceITCase method wrapWithBuffer.

private static Buffer wrapWithBuffer(byte[] data) {
    NetworkBuffer buffer = new NetworkBuffer(MemorySegmentFactory.allocateUnpooledSegment(data.length, null), FreeingBufferRecycler.INSTANCE);
    buffer.writeBytes(data);
    return buffer;
}

Example 34 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class EventSerializer method toBuffer.

// ------------------------------------------------------------------------
// Buffer helpers
// ------------------------------------------------------------------------
public static Buffer toBuffer(AbstractEvent event, boolean hasPriority) throws IOException {
    final ByteBuffer serializedEvent = EventSerializer.toSerializedEvent(event);
    MemorySegment data = MemorySegmentFactory.wrap(serializedEvent.array());
    final Buffer buffer = new NetworkBuffer(data, FreeingBufferRecycler.INSTANCE, getDataType(event, hasPriority));
    buffer.setSize(serializedEvent.remaining());
    return buffer;
}

Example 35 with NetworkBuffer

use of org.apache.flink.runtime.io.network.buffer.NetworkBuffer in project flink by apache.

the class BufferManager method requestExclusiveBuffers.

/**
 * Requests exclusive buffers from the provider.
 */
void requestExclusiveBuffers(int numExclusiveBuffers) throws IOException {
    checkArgument(numExclusiveBuffers >= 0, "Num exclusive buffers must be non-negative.");
    if (numExclusiveBuffers == 0) {
        return;
    }
    Collection<MemorySegment> segments = globalPool.requestUnpooledMemorySegments(numExclusiveBuffers);
    synchronized (bufferQueue) {
        // AvailableBufferQueue::addExclusiveBuffer may release the previously allocated
        // floating buffer, which requires the caller to recycle these released floating
        // buffers. There should be no floating buffers that have been allocated before the
        // exclusive buffers are initialized, so here only a simple assertion is required
        checkState(unsynchronizedGetFloatingBuffersAvailable() == 0, "Bug in buffer allocation logic: floating buffer is allocated before exclusive buffers are initialized.");
        for (MemorySegment segment : segments) {
            bufferQueue.addExclusiveBuffer(new NetworkBuffer(segment, this), numRequiredBuffers);
        }
    }
}