Examples with MiniClusterConfiguration org.apache.flink.runtime.minicluster.MiniClusterConfiguration used on opensource projects

Example 1 with MiniClusterConfiguration

use of org.apache.flink.runtime.minicluster.MiniClusterConfiguration in project flink by apache.

the class JobManagerWatermarkTrackerTest method setUp.

@BeforeClass
public static void setUp() throws Exception {
    final Configuration config = new Configuration();
    config.setInteger(RestOptions.PORT, 0);
    final MiniClusterConfiguration miniClusterConfiguration = new MiniClusterConfiguration.Builder().setConfiguration(config).setNumTaskManagers(1).setNumSlotsPerTaskManager(1).build();
    flink = new MiniCluster(miniClusterConfiguration);
    flink.start();
}

Example 2 with MiniClusterConfiguration

use of org.apache.flink.runtime.minicluster.MiniClusterConfiguration in project flink by apache.

the class FileSinkITBase method testFileSink.

@Test
public void testFileSink() throws Exception {
    String path = TEMPORARY_FOLDER.newFolder().getAbsolutePath();
    JobGraph jobGraph = createJobGraph(path);
    final Configuration config = new Configuration();
    config.setString(RestOptions.BIND_PORT, "18081-19000");
    final MiniClusterConfiguration cfg = new MiniClusterConfiguration.Builder().setNumTaskManagers(1).setNumSlotsPerTaskManager(4).setConfiguration(config).build();
    try (MiniCluster miniCluster = new MiniCluster(cfg)) {
        miniCluster.start();
        miniCluster.executeJobBlocking(jobGraph);
    }
    IntegerFileSinkTestDataUtils.checkIntegerSequenceSinkOutput(path, NUM_RECORDS, NUM_BUCKETS, NUM_SOURCES);
}

Example 3 with MiniClusterConfiguration

use of org.apache.flink.runtime.minicluster.MiniClusterConfiguration in project flink by apache.

the class FileBufferReaderITCase method testSequentialReading.

@Test
public void testSequentialReading() throws Exception {
    // setup
    final Configuration configuration;
    if (sslEnabled) {
        configuration = SSLUtilsTest.createInternalSslConfigWithKeyAndTrustStores("JDK");
    } else {
        configuration = new Configuration();
    }
    // Increases the handshake timeout to avoid connection reset/close issues
    // if the netty server thread could not response in time, like when it is
    // busy reading the files.
    configuration.setInteger(SecurityOptions.SSL_INTERNAL_HANDSHAKE_TIMEOUT, 100000);
    configuration.setString(RestOptions.BIND_PORT, "0");
    configuration.setString(NettyShuffleEnvironmentOptions.NETWORK_BLOCKING_SHUFFLE_TYPE, "file");
    configuration.set(TaskManagerOptions.TOTAL_FLINK_MEMORY, MemorySize.parse("1g"));
    configuration.set(TaskManagerOptions.MEMORY_SEGMENT_SIZE, MemorySize.parse(bufferSize + "b"));
    final MiniClusterConfiguration miniClusterConfiguration = new MiniClusterConfiguration.Builder().setConfiguration(configuration).setNumTaskManagers(parallelism).setNumSlotsPerTaskManager(1).build();
    try (final MiniCluster miniCluster = new MiniCluster(miniClusterConfiguration)) {
        miniCluster.start();
        final JobGraph jobGraph = createJobGraph();
        // the job needs to complete without throwing an exception
        miniCluster.executeJobBlocking(jobGraph);
    }
}

Example 4 with MiniClusterConfiguration

use of org.apache.flink.runtime.minicluster.MiniClusterConfiguration in project flink by apache.

the class DefaultSchedulerLocalRecoveryITCase method executeSchedulingTest.

private ArchivedExecutionGraph executeSchedulingTest(Configuration configuration, int parallelism) throws Exception {
    configuration.setString(RestOptions.BIND_PORT, "0");
    final long slotIdleTimeout = TIMEOUT;
    configuration.setLong(JobManagerOptions.SLOT_IDLE_TIMEOUT, slotIdleTimeout);
    configuration.set(TaskManagerOptions.TOTAL_FLINK_MEMORY, MemorySize.parse("64mb"));
    configuration.set(TaskManagerOptions.FRAMEWORK_HEAP_MEMORY, MemorySize.parse("16mb"));
    configuration.set(TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY, MemorySize.parse("16mb"));
    final MiniClusterConfiguration miniClusterConfiguration = new MiniClusterConfiguration.Builder().setConfiguration(configuration).setNumTaskManagers(parallelism).setNumSlotsPerTaskManager(1).build();
    try (MiniCluster miniCluster = new MiniCluster(miniClusterConfiguration)) {
        miniCluster.start();
        MiniClusterClient miniClusterClient = new MiniClusterClient(configuration, miniCluster);
        JobGraph jobGraph = createJobGraph(parallelism);
        // wait for the submission to succeed
        JobID jobId = miniClusterClient.submitJob(jobGraph).get(TIMEOUT, TimeUnit.SECONDS);
        // wait until all tasks running before triggering task failures
        waitUntilAllVerticesRunning(jobId, miniCluster);
        // kill one TM to trigger task failure and remove one existing slot
        CompletableFuture<Void> terminationFuture = miniCluster.terminateTaskManager(0);
        terminationFuture.get();
        // restart a taskmanager as a replacement for the killed one
        miniCluster.startTaskManager();
        // wait until all tasks running again
        waitUntilAllVerticesRunning(jobId, miniCluster);
        ArchivedExecutionGraph graph = miniCluster.getArchivedExecutionGraph(jobGraph.getJobID()).get();
        miniCluster.cancelJob(jobId).get();
        return graph;
    }
}

Example 5 with MiniClusterConfiguration

use of org.apache.flink.runtime.minicluster.MiniClusterConfiguration in project flink by apache.

the class CheckpointRestoreWithUidHashITCase method createMiniClusterConfig.

private MiniClusterConfiguration createMiniClusterConfig() {
    final Configuration config = new Configuration();
    config.setString(RestOptions.BIND_PORT, "18081-19000");
    return new MiniClusterConfiguration.Builder().setNumTaskManagers(1).setNumSlotsPerTaskManager(1).setConfiguration(config).build();
}