Examples with MemorySize org.apache.flink.configuration.MemorySize used on opensource projects

Example 51 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class TaskExecutorFlinkMemoryUtils method deriveFromRequiredFineGrainedOptions.

@Override
public TaskExecutorFlinkMemory deriveFromRequiredFineGrainedOptions(Configuration config) {
    final MemorySize taskHeapMemorySize = getTaskHeapMemorySize(config);
    final MemorySize managedMemorySize = getManagedMemorySize(config);
    final MemorySize frameworkHeapMemorySize = getFrameworkHeapMemorySize(config);
    final MemorySize frameworkOffHeapMemorySize = getFrameworkOffHeapMemorySize(config);
    final MemorySize taskOffHeapMemorySize = getTaskOffHeapMemorySize(config);
    final MemorySize networkMemorySize;
    final MemorySize totalFlinkExcludeNetworkMemorySize = frameworkHeapMemorySize.add(frameworkOffHeapMemorySize).add(taskHeapMemorySize).add(taskOffHeapMemorySize).add(managedMemorySize);
    if (isTotalFlinkMemorySizeExplicitlyConfigured(config)) {
        // derive network memory from total flink memory, and check against network min/max
        final MemorySize totalFlinkMemorySize = getTotalFlinkMemorySize(config);
        if (totalFlinkExcludeNetworkMemorySize.getBytes() > totalFlinkMemorySize.getBytes()) {
            throw new IllegalConfigurationException("Sum of configured Framework Heap Memory (" + frameworkHeapMemorySize.toHumanReadableString() + "), Framework Off-Heap Memory (" + frameworkOffHeapMemorySize.toHumanReadableString() + "), Task Heap Memory (" + taskHeapMemorySize.toHumanReadableString() + "), Task Off-Heap Memory (" + taskOffHeapMemorySize.toHumanReadableString() + ") and Managed Memory (" + managedMemorySize.toHumanReadableString() + ") exceed configured Total Flink Memory (" + totalFlinkMemorySize.toHumanReadableString() + ").");
        }
        networkMemorySize = totalFlinkMemorySize.subtract(totalFlinkExcludeNetworkMemorySize);
        sanityCheckNetworkMemoryWithExplicitlySetTotalFlinkAndHeapMemory(config, networkMemorySize, totalFlinkMemorySize);
    } else {
        // derive network memory from network configs
        networkMemorySize = isUsingLegacyNetworkConfigs(config) ? getNetworkMemorySizeWithLegacyConfig(config) : deriveNetworkMemoryWithInverseFraction(config, totalFlinkExcludeNetworkMemorySize);
    }
    final TaskExecutorFlinkMemory flinkInternalMemory = new TaskExecutorFlinkMemory(frameworkHeapMemorySize, frameworkOffHeapMemorySize, taskHeapMemorySize, taskOffHeapMemorySize, networkMemorySize, managedMemorySize);
    sanityCheckTotalFlinkMemory(config, flinkInternalMemory);
    return flinkInternalMemory;
}

Example 52 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class BashJavaUtils method logTaskExecutorConfiguration.

private static void logTaskExecutorConfiguration(TaskExecutorProcessSpec spec) {
    TaskExecutorFlinkMemory flinkMemory = spec.getFlinkMemory();
    MemorySize totalOffHeapMemory = flinkMemory.getManaged().add(flinkMemory.getJvmDirectMemorySize());
    LOG.info("Final TaskExecutor Memory configuration:");
    LOG.info("  Total Process Memory:          {}", spec.getTotalProcessMemorySize().toHumanReadableString());
    LOG.info("    Total Flink Memory:          {}", flinkMemory.getTotalFlinkMemorySize().toHumanReadableString());
    LOG.info("      Total JVM Heap Memory:     {}", flinkMemory.getJvmHeapMemorySize().toHumanReadableString());
    LOG.info("        Framework:               {}", flinkMemory.getFrameworkHeap().toHumanReadableString());
    LOG.info("        Task:                    {}", flinkMemory.getTaskHeap().toHumanReadableString());
    LOG.info("      Total Off-heap Memory:     {}", totalOffHeapMemory.toHumanReadableString());
    LOG.info("        Managed:                 {}", flinkMemory.getManaged().toHumanReadableString());
    LOG.info("        Total JVM Direct Memory: {}", flinkMemory.getJvmDirectMemorySize().toHumanReadableString());
    LOG.info("          Framework:             {}", flinkMemory.getFrameworkOffHeap().toHumanReadableString());
    LOG.info("          Task:                  {}", flinkMemory.getTaskOffHeap().toHumanReadableString());
    LOG.info("          Network:               {}", flinkMemory.getNetwork().toHumanReadableString());
    LOG.info("    JVM Metaspace:               {}", spec.getJvmMetaspaceSize().toHumanReadableString());
    LOG.info("    JVM Overhead:                {}", spec.getJvmOverheadSize().toHumanReadableString());
}

Example 53 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class JobManagerProcessUtilsTest method testDeriveFromRequiredFineGrainedOptions.

@Test
public void testDeriveFromRequiredFineGrainedOptions() {
    MemorySize jvmHeap = MemorySize.ofMebiBytes(150);
    MemorySize offHeap = MemorySize.ofMebiBytes(50);
    MemorySize totalFlinkMemory = MemorySize.ofMebiBytes(200);
    MemorySize expectedOffHeap = MemorySize.ofMebiBytes(50);
    Configuration conf = new Configuration();
    conf.set(JobManagerOptions.TOTAL_FLINK_MEMORY, totalFlinkMemory);
    conf.set(JobManagerOptions.OFF_HEAP_MEMORY, offHeap);
    conf.set(JobManagerOptions.JVM_HEAP_MEMORY, jvmHeap);
    JobManagerProcessSpec jobManagerProcessSpec = JobManagerProcessUtils.processSpecFromConfig(conf);
    assertThat(jobManagerProcessSpec.getJvmDirectMemorySize(), is(expectedOffHeap));
}

Example 54 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class JobManagerProcessUtilsTest method testConfigJvmHeapMemory.

@Test
public void testConfigJvmHeapMemory() {
    MemorySize jvmHeapSize = MemorySize.parse("50m");
    Configuration conf = new Configuration();
    conf.set(JobManagerOptions.JVM_HEAP_MEMORY, jvmHeapSize);
    JobManagerProcessSpec jobManagerProcessSpec = JobManagerProcessUtils.processSpecFromConfig(conf);
    assertThat(jobManagerProcessSpec.getJvmHeapMemorySize(), is(jvmHeapSize));
}

Example 55 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class JobManagerProcessUtilsTest method testLogFailureOfJvmHeapSizeMinSizeVerification.

@Test
public void testLogFailureOfJvmHeapSizeMinSizeVerification() {
    MemorySize jvmHeapMemory = MemorySize.parse("50m");
    Configuration conf = new Configuration();
    conf.set(JobManagerOptions.JVM_HEAP_MEMORY, jvmHeapMemory);
    JobManagerProcessUtils.processSpecFromConfig(conf);
    MatcherAssert.assertThat(testLoggerResource.getMessages(), hasItem(containsString(String.format("The configured or derived JVM heap memory size (%s) is less than its recommended minimum value (%s)", jvmHeapMemory.toHumanReadableString(), JobManagerOptions.MIN_JVM_HEAP_SIZE.toHumanReadableString()))));
}