Examples with MemorySize org.apache.flink.configuration.MemorySize used on opensource projects

Example 11 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class TaskExecutorProcessUtilsTest method testConfigNetworkMemoryRangeFailure.

@Test
public void testConfigNetworkMemoryRangeFailure() {
    final MemorySize networkMin = MemorySize.parse("200m");
    final MemorySize networkMax = MemorySize.parse("50m");
    Configuration conf = new Configuration();
    conf.set(TaskManagerOptions.NETWORK_MEMORY_MAX, networkMax);
    conf.set(TaskManagerOptions.NETWORK_MEMORY_MIN, networkMin);
    validateFailInAllConfigurations(conf);
}

Example 12 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class TaskExecutorProcessUtilsTest method configWithFineGrainedOptions.

@Override
protected void configWithFineGrainedOptions(Configuration configuration, MemorySize totalFlinkMemorySize) {
    MemorySize componentSize = new MemorySize(totalFlinkMemorySize.getBytes() / 6);
    configuration.set(TaskManagerOptions.TASK_HEAP_MEMORY, componentSize);
    configuration.set(TaskManagerOptions.TASK_OFF_HEAP_MEMORY, componentSize);
    configuration.set(TaskManagerOptions.FRAMEWORK_HEAP_MEMORY, componentSize);
    configuration.set(TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY, componentSize);
    configuration.set(TaskManagerOptions.MANAGED_MEMORY_SIZE, componentSize);
// network is the 6th component, fixed implicitly
}

Example 13 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class TaskExecutorProcessUtilsTest method testConfigManagedMemorySize.

@Test
public void testConfigManagedMemorySize() {
    final MemorySize managedMemSize = MemorySize.parse("100m");
    Configuration conf = new Configuration();
    conf.set(TaskManagerOptions.MANAGED_MEMORY_SIZE, managedMemSize);
    // validate in configurations without explicit managed memory size,
    // to avoid checking against overwritten managed memory size
    validateInConfigurationsWithoutExplicitManagedMem(conf, taskExecutorProcessSpec -> assertThat(taskExecutorProcessSpec.getManagedMemorySize(), is(managedMemSize)));
}

Example 14 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class TaskExecutorProcessUtils method processSpecFromWorkerResourceSpec.

public static TaskExecutorProcessSpec processSpecFromWorkerResourceSpec(final Configuration config, final WorkerResourceSpec workerResourceSpec) {
    final MemorySize frameworkHeapMemorySize = TaskExecutorFlinkMemoryUtils.getFrameworkHeapMemorySize(config);
    final MemorySize frameworkOffHeapMemorySize = TaskExecutorFlinkMemoryUtils.getFrameworkOffHeapMemorySize(config);
    final TaskExecutorFlinkMemory flinkMemory = new TaskExecutorFlinkMemory(frameworkHeapMemorySize, frameworkOffHeapMemorySize, workerResourceSpec.getTaskHeapSize(), workerResourceSpec.getTaskOffHeapSize(), workerResourceSpec.getNetworkMemSize(), workerResourceSpec.getManagedMemSize());
    final JvmMetaspaceAndOverhead jvmMetaspaceAndOverhead = PROCESS_MEMORY_UTILS.deriveJvmMetaspaceAndOverheadFromTotalFlinkMemory(config, flinkMemory.getTotalFlinkMemorySize());
    return new TaskExecutorProcessSpec(workerResourceSpec.getCpuCores(), flinkMemory, jvmMetaspaceAndOverhead, workerResourceSpec.getNumSlots(), workerResourceSpec.getExtendedResources().values());
}

Example 15 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class ProcessMemoryUtils method deriveJvmMetaspaceAndOverheadWithTotalProcessMemory.

private JvmMetaspaceAndOverhead deriveJvmMetaspaceAndOverheadWithTotalProcessMemory(Configuration config, MemorySize totalProcessMemorySize) {
    MemorySize jvmMetaspaceSize = getMemorySizeFromConfig(config, options.getJvmOptions().getJvmMetaspaceOption());
    MemorySize jvmOverheadSize = deriveWithFraction("jvm overhead memory", totalProcessMemorySize, getJvmOverheadRangeFraction(config));
    JvmMetaspaceAndOverhead jvmMetaspaceAndOverhead = new JvmMetaspaceAndOverhead(jvmMetaspaceSize, jvmOverheadSize);
    if (jvmMetaspaceAndOverhead.getTotalJvmMetaspaceAndOverheadSize().getBytes() > totalProcessMemorySize.getBytes()) {
        throw new IllegalConfigurationException("Sum of configured JVM Metaspace (" + jvmMetaspaceAndOverhead.getMetaspace().toHumanReadableString() + ") and JVM Overhead (" + jvmMetaspaceAndOverhead.getOverhead().toHumanReadableString() + ") exceed configured Total Process Memory (" + totalProcessMemorySize.toHumanReadableString() + ").");
    }
    return jvmMetaspaceAndOverhead;
}