Examples with MemorySize org.apache.flink.configuration.MemorySize used on opensource projects

Example 16 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class ProcessMemoryUtils method getJvmOverheadRangeFraction.

private RangeFraction getJvmOverheadRangeFraction(Configuration config) {
    MemorySize minSize = getMemorySizeFromConfig(config, options.getJvmOptions().getJvmOverheadMin());
    MemorySize maxSize = getMemorySizeFromConfig(config, options.getJvmOptions().getJvmOverheadMax());
    return getRangeFraction(minSize, maxSize, options.getJvmOptions().getJvmOverheadFraction(), config);
}

Example 17 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class ProcessMemoryUtils method deriveProcessSpecWithTotalFlinkMemory.

private CommonProcessMemorySpec<FM> deriveProcessSpecWithTotalFlinkMemory(Configuration config) {
    MemorySize totalFlinkMemorySize = getMemorySizeFromConfig(config, options.getTotalFlinkMemoryOption());
    FM flinkInternalMemory = flinkMemoryUtils.deriveFromTotalFlinkMemory(config, totalFlinkMemorySize);
    JvmMetaspaceAndOverhead jvmMetaspaceAndOverhead = deriveJvmMetaspaceAndOverheadFromTotalFlinkMemory(config, totalFlinkMemorySize);
    return new CommonProcessMemorySpec<>(flinkInternalMemory, jvmMetaspaceAndOverhead);
}

Example 18 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class ProcessMemoryUtils method deriveProcessSpecWithTotalProcessMemory.

private CommonProcessMemorySpec<FM> deriveProcessSpecWithTotalProcessMemory(Configuration config) {
    MemorySize totalProcessMemorySize = getMemorySizeFromConfig(config, options.getTotalProcessMemoryOption());
    JvmMetaspaceAndOverhead jvmMetaspaceAndOverhead = deriveJvmMetaspaceAndOverheadWithTotalProcessMemory(config, totalProcessMemorySize);
    MemorySize totalFlinkMemorySize = totalProcessMemorySize.subtract(jvmMetaspaceAndOverhead.getTotalJvmMetaspaceAndOverheadSize());
    FM flinkInternalMemory = flinkMemoryUtils.deriveFromTotalFlinkMemory(config, totalFlinkMemorySize);
    return new CommonProcessMemorySpec<>(flinkInternalMemory, jvmMetaspaceAndOverhead);
}

Example 19 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class ProcessMemoryUtils method deriveWithInverseFraction.

public static MemorySize deriveWithInverseFraction(String memoryDescription, MemorySize base, RangeFraction rangeFraction) {
    checkArgument(rangeFraction.getFraction() < 1);
    MemorySize relative = base.multiply(rangeFraction.getFraction() / (1 - rangeFraction.getFraction()));
    return capToMinMax(memoryDescription, relative, rangeFraction);
}

Example 20 with MemorySize

use of org.apache.flink.configuration.MemorySize in project flink by apache.

the class AdapterTestBase method testReading.

private void testReading(FormatT format, int numSplits, int... recoverAfterRecords) throws IOException {
    // add the end boundary for recovery
    final int[] boundaries = Arrays.copyOf(recoverAfterRecords, recoverAfterRecords.length + 1);
    boundaries[boundaries.length - 1] = NUM_NUMBERS;
    // set a fetch size so that we get three records per fetch
    final Configuration config = new Configuration();
    config.set(StreamFormat.FETCH_IO_SIZE, new MemorySize(10));
    final BulkFormat<Integer, FileSourceSplit> adapter = wrapWithAdapter(format);
    final Queue<FileSourceSplit> splits = buildSplits(numSplits);
    final List<Integer> result = new ArrayList<>();
    FileSourceSplit currentSplit = null;
    BulkFormat.Reader<Integer> currentReader = null;
    for (int nextRecordToRecover : boundaries) {
        final FileSourceSplit toRecoverFrom = readNumbers(currentReader, currentSplit, adapter, splits, config, result, nextRecordToRecover - result.size());
        currentSplit = toRecoverFrom;
        currentReader = toRecoverFrom == null ? null : adapter.restoreReader(config, toRecoverFrom);
    }
    verifyIntListResult(result);
}