Examples with LongFunctionWithException org.apache.flink.util.function.LongFunctionWithException used on opensource projects

Example 1 with LongFunctionWithException

use of org.apache.flink.util.function.LongFunctionWithException in project flink by apache.

the class MemoryManager method getSharedMemoryResourceForManagedMemory.

// ------------------------------------------------------------------------
// Shared opaque memory resources
// ------------------------------------------------------------------------
/**
 * Acquires a shared memory resource, identified by a type string. If the resource already
 * exists, this returns a descriptor to the resource. If the resource does not yet exist, the
 * given memory fraction is reserved and the resource is initialized with that size.
 *
 * <p>The memory for the resource is reserved from the memory budget of this memory manager
 * (thus determining the size of the resource), but resource itself is opaque, meaning the
 * memory manager does not understand its structure.
 *
 * <p>The OpaqueMemoryResource object returned from this method must be closed once not used any
 * further. Once all acquisitions have closed the object, the resource itself is closed.
 *
 * <p><b>Important:</b> The failure semantics are as follows: If the memory manager fails to
 * reserve the memory, the external resource initializer will not be called. If an exception is
 * thrown when the opaque resource is closed (last lease is released), the memory manager will
 * still un-reserve the memory to make sure its own accounting is clean. The exception will need
 * to be handled by the caller of {@link OpaqueMemoryResource#close()}. For example, if this
 * indicates that native memory was not released and the process might thus have a memory leak,
 * the caller can decide to kill the process as a result.
 */
public <T extends AutoCloseable> OpaqueMemoryResource<T> getSharedMemoryResourceForManagedMemory(String type, LongFunctionWithException<T, Exception> initializer, double fractionToInitializeWith) throws Exception {
    // if we need to allocate the resource (no shared resource allocated, yet), this would be
    // the size to use
    final long numBytes = computeMemorySize(fractionToInitializeWith);
    // initializer and releaser as functions that are pushed into the SharedResources,
    // so that the SharedResources can decide in (thread-safely execute) when initialization
    // and release should happen
    final LongFunctionWithException<T, Exception> reserveAndInitialize = (size) -> {
        try {
            reserveMemory(type, size);
        } catch (MemoryReservationException e) {
            throw new MemoryAllocationException("Could not created the shared memory resource of size " + size + ". Not enough memory left to reserve from the slot's managed memory.", e);
        }
        try {
            return initializer.apply(size);
        } catch (Throwable t) {
            releaseMemory(type, size);
            throw t;
        }
    };
    final LongConsumer releaser = (size) -> releaseMemory(type, size);
    // This object identifies the lease in this request. It is used only to identify the release
    // operation.
    // Using the object to represent the lease is a bit nicer safer than just using a reference
    // counter.
    final Object leaseHolder = new Object();
    final SharedResources.ResourceAndSize<T> resource = sharedResources.getOrAllocateSharedResource(type, leaseHolder, reserveAndInitialize, numBytes);
    // the actual size may theoretically be different from what we requested, if allocated it
    // was by
    // someone else before with a different value for fraction (should not happen in practice,
    // though).
    final long size = resource.size();
    final ThrowingRunnable<Exception> disposer = () -> sharedResources.release(type, leaseHolder, releaser);
    return new OpaqueMemoryResource<>(resource.resourceHandle(), size, disposer);
}

Example 2 with LongFunctionWithException

use of org.apache.flink.util.function.LongFunctionWithException in project flink by apache.

the class RocksDBOperationUtils method allocateSharedCachesIfConfigured.

@Nullable
public static OpaqueMemoryResource<RocksDBSharedResources> allocateSharedCachesIfConfigured(RocksDBMemoryConfiguration memoryConfig, MemoryManager memoryManager, double memoryFraction, Logger logger) throws IOException {
    if (!memoryConfig.isUsingFixedMemoryPerSlot() && !memoryConfig.isUsingManagedMemory()) {
        return null;
    }
    final double highPriorityPoolRatio = memoryConfig.getHighPriorityPoolRatio();
    final double writeBufferRatio = memoryConfig.getWriteBufferRatio();
    final boolean usingPartitionedIndexFilters = memoryConfig.isUsingPartitionedIndexFilters();
    final LongFunctionWithException<RocksDBSharedResources, Exception> allocator = (size) -> RocksDBMemoryControllerUtils.allocateRocksDBSharedResources(size, writeBufferRatio, highPriorityPoolRatio, usingPartitionedIndexFilters);
    try {
        if (memoryConfig.isUsingFixedMemoryPerSlot()) {
            assert memoryConfig.getFixedMemoryPerSlot() != null;
            logger.info("Getting fixed-size shared cache for RocksDB.");
            return memoryManager.getExternalSharedMemoryResource(FIXED_SLOT_MEMORY_RESOURCE_ID, allocator, memoryConfig.getFixedMemoryPerSlot().getBytes());
        } else {
            logger.info("Getting managed memory shared cache for RocksDB.");
            return memoryManager.getSharedMemoryResourceForManagedMemory(MANAGED_MEMORY_RESOURCE_ID, allocator, memoryFraction);
        }
    } catch (Exception e) {
        throw new IOException("Failed to acquire shared cache resource for RocksDB", e);
    }
}

Example 3 with LongFunctionWithException

use of org.apache.flink.util.function.LongFunctionWithException in project flink by apache.

the class BeamPythonFunctionRunner method open.

// ------------------------------------------------------------------------
@Override
public void open(PythonConfig config) throws Exception {
    this.bundleStarted = false;
    this.resultBuffer = new LinkedBlockingQueue<>();
    this.reusableResultTuple = new Tuple2<>();
    // The creation of stageBundleFactory depends on the initialized environment manager.
    environmentManager.open();
    PortablePipelineOptions portableOptions = PipelineOptionsFactory.as(PortablePipelineOptions.class);
    if (jobOptions.containsKey(PythonOptions.STATE_CACHE_SIZE.key())) {
        portableOptions.as(ExperimentalOptions.class).setExperiments(Collections.singletonList(ExperimentalOptions.STATE_CACHE_SIZE + "=" + jobOptions.get(PythonOptions.STATE_CACHE_SIZE.key())));
    }
    Struct pipelineOptions = PipelineOptionsTranslation.toProto(portableOptions);
    if (memoryManager != null && config.isUsingManagedMemory()) {
        Preconditions.checkArgument(managedMemoryFraction > 0 && managedMemoryFraction <= 1.0, "The configured managed memory fraction for Python worker process must be within (0, 1], was: %s. " + "It may be because the consumer type \"Python\" was missing or set to 0 for the config option \"taskmanager.memory.managed.consumer-weights\"." + managedMemoryFraction);
        final LongFunctionWithException<PythonSharedResources, Exception> initializer = (size) -> new PythonSharedResources(createJobBundleFactory(pipelineOptions), createPythonExecutionEnvironment(size));
        sharedResources = memoryManager.getSharedMemoryResourceForManagedMemory(MANAGED_MEMORY_RESOURCE_ID, initializer, managedMemoryFraction);
        LOG.info("Obtained shared Python process of size {} bytes", sharedResources.getSize());
        sharedResources.getResourceHandle().addPythonEnvironmentManager(environmentManager);
        JobBundleFactory jobBundleFactory = sharedResources.getResourceHandle().getJobBundleFactory();
        RunnerApi.Environment environment = sharedResources.getResourceHandle().getEnvironment();
        stageBundleFactory = createStageBundleFactory(jobBundleFactory, environment);
    } else {
        // there is no way to access the MemoryManager for the batch job of old planner,
        // fallback to the way that spawning a Python process for each Python operator
        jobBundleFactory = createJobBundleFactory(pipelineOptions);
        stageBundleFactory = createStageBundleFactory(jobBundleFactory, createPythonExecutionEnvironment(-1));
    }
    progressHandler = getProgressHandler(flinkMetricContainer);
}