Examples with ThrowingRunnable org.apache.flink.util.function.ThrowingRunnable used on opensource projects

Example 1 with ThrowingRunnable

use of org.apache.flink.util.function.ThrowingRunnable in project flink by apache.

the class MemoryManager method getSharedMemoryResourceForManagedMemory.

// ------------------------------------------------------------------------
// Shared opaque memory resources
// ------------------------------------------------------------------------
/**
 * Acquires a shared memory resource, identified by a type string. If the resource already
 * exists, this returns a descriptor to the resource. If the resource does not yet exist, the
 * given memory fraction is reserved and the resource is initialized with that size.
 *
 * <p>The memory for the resource is reserved from the memory budget of this memory manager
 * (thus determining the size of the resource), but resource itself is opaque, meaning the
 * memory manager does not understand its structure.
 *
 * <p>The OpaqueMemoryResource object returned from this method must be closed once not used any
 * further. Once all acquisitions have closed the object, the resource itself is closed.
 *
 * <p><b>Important:</b> The failure semantics are as follows: If the memory manager fails to
 * reserve the memory, the external resource initializer will not be called. If an exception is
 * thrown when the opaque resource is closed (last lease is released), the memory manager will
 * still un-reserve the memory to make sure its own accounting is clean. The exception will need
 * to be handled by the caller of {@link OpaqueMemoryResource#close()}. For example, if this
 * indicates that native memory was not released and the process might thus have a memory leak,
 * the caller can decide to kill the process as a result.
 */
public <T extends AutoCloseable> OpaqueMemoryResource<T> getSharedMemoryResourceForManagedMemory(String type, LongFunctionWithException<T, Exception> initializer, double fractionToInitializeWith) throws Exception {
    // if we need to allocate the resource (no shared resource allocated, yet), this would be
    // the size to use
    final long numBytes = computeMemorySize(fractionToInitializeWith);
    // initializer and releaser as functions that are pushed into the SharedResources,
    // so that the SharedResources can decide in (thread-safely execute) when initialization
    // and release should happen
    final LongFunctionWithException<T, Exception> reserveAndInitialize = (size) -> {
        try {
            reserveMemory(type, size);
        } catch (MemoryReservationException e) {
            throw new MemoryAllocationException("Could not created the shared memory resource of size " + size + ". Not enough memory left to reserve from the slot's managed memory.", e);
        }
        try {
            return initializer.apply(size);
        } catch (Throwable t) {
            releaseMemory(type, size);
            throw t;
        }
    };
    final LongConsumer releaser = (size) -> releaseMemory(type, size);
    // This object identifies the lease in this request. It is used only to identify the release
    // operation.
    // Using the object to represent the lease is a bit nicer safer than just using a reference
    // counter.
    final Object leaseHolder = new Object();
    final SharedResources.ResourceAndSize<T> resource = sharedResources.getOrAllocateSharedResource(type, leaseHolder, reserveAndInitialize, numBytes);
    // the actual size may theoretically be different from what we requested, if allocated it
    // was by
    // someone else before with a different value for fraction (should not happen in practice,
    // though).
    final long size = resource.size();
    final ThrowingRunnable<Exception> disposer = () -> sharedResources.release(type, leaseHolder, releaser);
    return new OpaqueMemoryResource<>(resource.resourceHandle(), size, disposer);
}

Example 2 with ThrowingRunnable

use of org.apache.flink.util.function.ThrowingRunnable in project flink by apache.

the class SourceCoordinatorTest method testBlockOnClose.

@Test
public void testBlockOnClose() throws Exception {
    // It is possible that the split enumerator submits some heavy-duty work to the
    // coordinator executor which blocks the coordinator closure.
    final CountDownLatch latch = new CountDownLatch(1);
    try (final MockSplitEnumeratorContext<MockSourceSplit> enumeratorContext = new MockSplitEnumeratorContext<>(1);
        final MockSplitEnumerator splitEnumerator = new MockSplitEnumerator(1, enumeratorContext) {

            @Override
            public void handleSourceEvent(int subtaskId, SourceEvent sourceEvent) {
                context.callAsync(() -> 1L, (ignored, t) -> {
                    latch.countDown();
                    // Submit a callable that will never return.
                    try {
                        Thread.sleep(Long.MAX_VALUE);
                    } catch (InterruptedException e) {
                        throw new RuntimeException(e);
                    }
                });
            }
        };
        final SourceCoordinator<?, ?> coordinator = new SourceCoordinator<>(OPERATOR_NAME, new EnumeratorCreatingSource<>(() -> splitEnumerator), context, new CoordinatorStoreImpl())) {
        coordinator.start();
        coordinator.handleEventFromOperator(1, new SourceEventWrapper(new SourceEvent() {
        }));
        // Wait until the coordinator executor blocks.
        latch.await();
        CompletableFuture<?> future = ComponentClosingUtils.closeAsyncWithTimeout("testBlockOnClose", (ThrowingRunnable<Exception>) coordinator::close, Duration.ofMillis(1));
        future.exceptionally(e -> {
            assertTrue(e instanceof TimeoutException);
            return null;
        }).get();
        waitUtil(splitEnumerator::closed, Duration.ofSeconds(5), "Split enumerator was not closed in 5 seconds.");
    }
}