Examples with MiniCluster org.apache.flink.runtime.minicluster.MiniCluster used on opensource projects

Example 41 with MiniCluster

use of org.apache.flink.runtime.minicluster.MiniCluster in project flink by apache.

the class JobGraphRunningUtil method execute.

public static void execute(JobGraph jobGraph, Configuration configuration, int numTaskManagers, int numSlotsPerTaskManager) throws Exception {
    configuration.set(TaskManagerOptions.TOTAL_FLINK_MEMORY, MemorySize.parse("1g"));
    configuration.setString(RestOptions.BIND_PORT, "0");
    final MiniClusterConfiguration miniClusterConfiguration = new MiniClusterConfiguration.Builder().setConfiguration(configuration).setNumTaskManagers(numTaskManagers).setNumSlotsPerTaskManager(numSlotsPerTaskManager).build();
    try (MiniCluster miniCluster = new MiniCluster(miniClusterConfiguration)) {
        miniCluster.start();
        MiniClusterClient miniClusterClient = new MiniClusterClient(configuration, miniCluster);
        // wait for the submission to succeed
        JobID jobID = miniClusterClient.submitJob(jobGraph).get();
        JobResult jobResult = miniClusterClient.requestJobResult(jobID).get();
        if (jobResult.getSerializedThrowable().isPresent()) {
            throw new AssertionError(jobResult.getSerializedThrowable().get());
        }
    }
}

Example 42 with MiniCluster

use of org.apache.flink.runtime.minicluster.MiniCluster in project flink by apache.

the class PipelinedRegionSchedulingITCase method executeSchedulingTest.

private JobResult executeSchedulingTest(JobGraph jobGraph, int numSlots, Configuration configuration) throws Exception {
    configuration.setString(RestOptions.BIND_PORT, "0");
    configuration.setLong(JobManagerOptions.SLOT_REQUEST_TIMEOUT, 5000L);
    final MiniClusterConfiguration miniClusterConfiguration = new MiniClusterConfiguration.Builder().setConfiguration(configuration).setNumTaskManagers(1).setNumSlotsPerTaskManager(numSlots).build();
    try (MiniCluster miniCluster = new MiniCluster(miniClusterConfiguration)) {
        miniCluster.start();
        final MiniClusterClient miniClusterClient = new MiniClusterClient(configuration, miniCluster);
        // wait for the submission to succeed
        final JobID jobID = miniClusterClient.submitJob(jobGraph).get();
        final CompletableFuture<JobResult> resultFuture = miniClusterClient.requestJobResult(jobID);
        final JobResult jobResult = resultFuture.get();
        return jobResult;
    }
}

Example 43 with MiniCluster

use of org.apache.flink.runtime.minicluster.MiniCluster in project flink by apache.

the class SchedulingITCase method executeSchedulingTest.

private void executeSchedulingTest(Configuration configuration) throws Exception {
    configuration.setString(RestOptions.BIND_PORT, "0");
    final long slotIdleTimeout = 50L;
    configuration.setLong(JobManagerOptions.SLOT_IDLE_TIMEOUT, slotIdleTimeout);
    configuration.set(TaskManagerOptions.TOTAL_FLINK_MEMORY, MemorySize.parse("1g"));
    final int parallelism = 4;
    final MiniClusterConfiguration miniClusterConfiguration = new MiniClusterConfiguration.Builder().setConfiguration(configuration).setNumTaskManagers(parallelism).setNumSlotsPerTaskManager(1).build();
    try (MiniCluster miniCluster = new MiniCluster(miniClusterConfiguration)) {
        miniCluster.start();
        MiniClusterClient miniClusterClient = new MiniClusterClient(configuration, miniCluster);
        JobGraph jobGraph = createJobGraph(slotIdleTimeout << 1, parallelism);
        // wait for the submission to succeed
        JobID jobID = miniClusterClient.submitJob(jobGraph).get();
        CompletableFuture<JobResult> resultFuture = miniClusterClient.requestJobResult(jobID);
        JobResult jobResult = resultFuture.get();
        assertThat(jobResult.getSerializedThrowable().isPresent(), is(false));
    }
}

Example 44 with MiniCluster

use of org.apache.flink.runtime.minicluster.MiniCluster in project flink by apache.

the class TestStreamEnvironment method setAsContext.

/**
 * Sets the streaming context environment to a TestStreamEnvironment that runs its programs on
 * the given cluster with the given default parallelism and the specified jar files and class
 * paths.
 *
 * @param miniCluster The MiniCluster to execute jobs on.
 * @param parallelism The default parallelism for the test programs.
 * @param jarFiles Additional jar files to execute the job with
 * @param classpaths Additional class paths to execute the job with
 */
public static void setAsContext(final MiniCluster miniCluster, final int parallelism, final Collection<Path> jarFiles, final Collection<URL> classpaths) {
    StreamExecutionEnvironmentFactory factory = conf -> {
        TestStreamEnvironment env = new TestStreamEnvironment(miniCluster, conf, parallelism, jarFiles, classpaths);
        randomizeConfiguration(miniCluster, conf);
        env.configure(conf, env.getUserClassloader());
        return env;
    };
    initializeContextEnvironment(factory);
}