Examples with MemorySize org.apache.flink.configuration.MemorySize used on opensource projects

Example 61 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class TaskExecutorProcessUtilsTest method testConfigFrameworkHeapMemory.

@Test
public void testConfigFrameworkHeapMemory() {
    final MemorySize frameworkHeapSize = MemorySize.parse("100m");
    Configuration conf = new Configuration();
    conf.set(TaskManagerOptions.FRAMEWORK_HEAP_MEMORY, frameworkHeapSize);
    validateInAllConfigurations(conf, taskExecutorProcessSpec -> assertThat(taskExecutorProcessSpec.getFrameworkHeapSize(), is(frameworkHeapSize)));
}

Example 62 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class TaskExecutorProcessUtilsTest method testConfigNetworkMemoryLegacyNumOfBuffers.

@Test
public void testConfigNetworkMemoryLegacyNumOfBuffers() {
    final MemorySize pageSize = MemorySize.parse("32k");
    final int numOfBuffers = 1024;
    final MemorySize networkSize = pageSize.multiply(numOfBuffers);
    @SuppressWarnings("deprecation") final ConfigOption<Integer> legacyOption = NettyShuffleEnvironmentOptions.NETWORK_NUM_BUFFERS;
    Configuration conf = new Configuration();
    conf.set(TaskManagerOptions.MEMORY_SEGMENT_SIZE, pageSize);
    conf.setInteger(legacyOption, numOfBuffers);
    // validate in configurations without explicit total flink/process memory, otherwise
    // explicit configured
    // network memory size might conflict with total flink/process memory minus other memory
    // sizes
    validateInConfigWithExplicitTaskHeapAndManagedMem(conf, taskExecutorProcessSpec -> assertThat(taskExecutorProcessSpec.getNetworkMemSize(), is(networkSize)));
    validateInConfigurationsWithoutExplicitTaskHeapMem(conf, taskExecutorProcessSpec -> assertThat(taskExecutorProcessSpec.getNetworkMemSize(), is(networkSize)));
}

Example 63 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class BucketsRollingPolicyTest method testDefaultRollingPolicy.

@Test
public void testDefaultRollingPolicy() throws Exception {
    final File outDir = TEMP_FOLDER.newFolder();
    final Path path = new Path(outDir.toURI());
    final RollingPolicy<String, String> originalRollingPolicy = DefaultRollingPolicy.builder().withMaxPartSize(new MemorySize(10L)).withInactivityInterval(Duration.ofMillis(4L)).withRolloverInterval(Duration.ofMillis(11L)).build();
    final MethodCallCountingPolicyWrapper<String, String> rollingPolicy = new MethodCallCountingPolicyWrapper<>(originalRollingPolicy);
    final Buckets<String, String> buckets = createBuckets(path, rollingPolicy);
    rollingPolicy.verifyCallCounters(0L, 0L, 0L, 0L, 0L, 0L);
    // these two will fill up the first in-progress file and at the third it will roll ...
    buckets.onElement("test1", new TestUtils.MockSinkContext(1L, 1L, 1L));
    buckets.onElement("test1", new TestUtils.MockSinkContext(2L, 1L, 2L));
    rollingPolicy.verifyCallCounters(0L, 0L, 1L, 0L, 0L, 0L);
    buckets.onElement("test1", new TestUtils.MockSinkContext(3L, 1L, 3L));
    rollingPolicy.verifyCallCounters(0L, 0L, 2L, 1L, 0L, 0L);
    // still no time to roll
    buckets.onProcessingTime(5L);
    rollingPolicy.verifyCallCounters(0L, 0L, 2L, 1L, 1L, 0L);
    // roll due to inactivity
    buckets.onProcessingTime(7L);
    rollingPolicy.verifyCallCounters(0L, 0L, 2L, 1L, 2L, 1L);
    buckets.onElement("test1", new TestUtils.MockSinkContext(3L, 1L, 3L));
    // roll due to rollover interval
    buckets.onProcessingTime(20L);
    rollingPolicy.verifyCallCounters(0L, 0L, 2L, 1L, 3L, 2L);
    // we take a checkpoint but we should not roll.
    buckets.snapshotState(1L, new TestUtils.MockListState<>(), new TestUtils.MockListState<>());
    rollingPolicy.verifyCallCounters(0L, 0L, 2L, 1L, 3L, 2L);
}

Example 64 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class BucketsRollingPolicyTest method testDefaultRollingPolicyDeprecatedCreate.

@Test
public void testDefaultRollingPolicyDeprecatedCreate() throws Exception {
    DefaultRollingPolicy policy = DefaultRollingPolicy.builder().withInactivityInterval(Duration.ofMillis(10)).withMaxPartSize(new MemorySize(20)).withRolloverInterval(Duration.ofMillis(30)).build();
    Assert.assertEquals(10, policy.getInactivityInterval());
    Assert.assertEquals(20, policy.getMaxPartSize());
    Assert.assertEquals(30, policy.getRolloverInterval());
}

Example 65 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class ProcessMemoryUtilsTestBase method testDerivedTotalProcessMemoryGreaterThanConfiguredFailureWithFineGrainedOptions.

@Test
public void testDerivedTotalProcessMemoryGreaterThanConfiguredFailureWithFineGrainedOptions() {
    Configuration conf = getConfigurationWithJvmMetaspaceAndTotalFlinkMemory(100, 200);
    // Total Flink memory + JVM Metaspace > Total Process Memory (no space for JVM overhead)
    MemorySize totalFlinkMemorySize = MemorySize.ofMebiBytes(150);
    configWithFineGrainedOptions(conf, totalFlinkMemorySize);
    validateFail(conf);
}