Search in sources :

Example 6 with FlinkYarnSessionCli

use of org.apache.flink.yarn.cli.FlinkYarnSessionCli in project flink by apache.

the class FlinkYarnSessionCliTest method testResumeFromYarnIDZookeeperNamespace.

@Test
public void testResumeFromYarnIDZookeeperNamespace() throws Exception {
    final FlinkYarnSessionCli flinkYarnSessionCli = createFlinkYarnSessionCli();
    final CommandLine commandLine = flinkYarnSessionCli.parseCommandLineOptions(new String[] { "-yid", TEST_YARN_APPLICATION_ID.toString() }, true);
    final Configuration executorConfig = flinkYarnSessionCli.toConfiguration(commandLine);
    final ClusterClientFactory<ApplicationId> clientFactory = getClusterClientFactory(executorConfig);
    final YarnClusterDescriptor clusterDescriptor = (YarnClusterDescriptor) clientFactory.createClusterDescriptor(executorConfig);
    final Configuration clusterDescriptorConfiguration = clusterDescriptor.getFlinkConfiguration();
    String zkNs = clusterDescriptorConfiguration.getValue(HighAvailabilityOptions.HA_CLUSTER_ID);
    assertTrue(zkNs.matches("application_\\d+_0042"));
}
Also used : CustomCommandLine(org.apache.flink.client.cli.CustomCommandLine) CommandLine(org.apache.commons.cli.CommandLine) Configuration(org.apache.flink.configuration.Configuration) ApplicationId(org.apache.hadoop.yarn.api.records.ApplicationId) FlinkYarnSessionCli(org.apache.flink.yarn.cli.FlinkYarnSessionCli) Test(org.junit.Test)

Example 7 with FlinkYarnSessionCli

use of org.apache.flink.yarn.cli.FlinkYarnSessionCli in project flink by apache.

the class FlinkYarnSessionCliTest method testMissingShipFiles.

@Test
public void testMissingShipFiles() throws Exception {
    File tmpFile = tmp.newFile();
    final String[] args = new String[] { "run", "--yarnship", tmpFile.toString(), "--yarnship", "missing.file" };
    final FlinkYarnSessionCli flinkYarnSessionCli = createFlinkYarnSessionCli();
    final CommandLine commandLine = flinkYarnSessionCli.parseCommandLineOptions(args, false);
    try {
        flinkYarnSessionCli.toConfiguration(commandLine);
        fail("Expected error for missing file");
    } catch (ConfigurationException ce) {
        assertEquals("Ship file missing.file does not exist", ce.getMessage());
    }
}
Also used : CustomCommandLine(org.apache.flink.client.cli.CustomCommandLine) CommandLine(org.apache.commons.cli.CommandLine) ConfigurationException(org.apache.flink.util.ConfigurationException) File(java.io.File) FlinkYarnSessionCli(org.apache.flink.yarn.cli.FlinkYarnSessionCli) Test(org.junit.Test)

Example 8 with FlinkYarnSessionCli

use of org.apache.flink.yarn.cli.FlinkYarnSessionCli in project flink by apache.

the class FlinkYarnSessionCliTest method testShipFiles.

@Test
public void testShipFiles() throws Exception {
    File tmpFile = tmp.newFile();
    final String[] args = new String[] { "run", "--yarnship", tmpFile.toString() };
    final FlinkYarnSessionCli flinkYarnSessionCli = createFlinkYarnSessionCli();
    final CommandLine commandLine = flinkYarnSessionCli.parseCommandLineOptions(args, false);
    final Configuration executorConfig = flinkYarnSessionCli.toConfiguration(commandLine);
    final ClusterClientFactory<ApplicationId> clientFactory = getClusterClientFactory(executorConfig);
    YarnClusterDescriptor flinkYarnDescriptor = (YarnClusterDescriptor) clientFactory.createClusterDescriptor(executorConfig);
    assertEquals(Lists.newArrayList(tmpFile), flinkYarnDescriptor.getShipFiles());
}
Also used : CustomCommandLine(org.apache.flink.client.cli.CustomCommandLine) CommandLine(org.apache.commons.cli.CommandLine) Configuration(org.apache.flink.configuration.Configuration) ApplicationId(org.apache.hadoop.yarn.api.records.ApplicationId) File(java.io.File) FlinkYarnSessionCli(org.apache.flink.yarn.cli.FlinkYarnSessionCli) Test(org.junit.Test)

Example 9 with FlinkYarnSessionCli

use of org.apache.flink.yarn.cli.FlinkYarnSessionCli in project flink by apache.

the class FlinkYarnSessionCliTest method testHeapMemoryPropertyWithOldConfigKey.

/**
 * Tests the specifying heap memory with old config key for job manager and task manager.
 */
@Test
public void testHeapMemoryPropertyWithOldConfigKey() throws Exception {
    Configuration configuration = new Configuration();
    configuration.setInteger(JobManagerOptions.JOB_MANAGER_HEAP_MEMORY_MB, 2048);
    configuration.setInteger(TaskManagerOptions.TASK_MANAGER_HEAP_MEMORY_MB, 4096);
    final FlinkYarnSessionCli flinkYarnSessionCli = createFlinkYarnSessionCli(configuration);
    final CommandLine commandLine = flinkYarnSessionCli.parseCommandLineOptions(new String[0], false);
    configuration.addAll(flinkYarnSessionCli.toConfiguration(commandLine));
    final ClusterClientFactory<ApplicationId> clientFactory = getClusterClientFactory(configuration);
    final ClusterSpecification clusterSpecification = clientFactory.getClusterSpecification(configuration);
    assertThat(clusterSpecification.getMasterMemoryMB(), is(2048));
    assertThat(clusterSpecification.getTaskManagerMemoryMB(), is(4096));
}
Also used : CustomCommandLine(org.apache.flink.client.cli.CustomCommandLine) CommandLine(org.apache.commons.cli.CommandLine) Configuration(org.apache.flink.configuration.Configuration) ClusterSpecification(org.apache.flink.client.deployment.ClusterSpecification) ApplicationId(org.apache.hadoop.yarn.api.records.ApplicationId) FlinkYarnSessionCli(org.apache.flink.yarn.cli.FlinkYarnSessionCli) Test(org.junit.Test)

Example 10 with FlinkYarnSessionCli

use of org.apache.flink.yarn.cli.FlinkYarnSessionCli in project flink by apache.

the class FlinkYarnSessionCliTest method testCorrectSettingOfMaxSlots.

@Test
public void testCorrectSettingOfMaxSlots() throws Exception {
    String[] params = new String[] { "-ys", "3" };
    final Configuration configuration = createConfigurationWithJmAndTmTotalMemory(2048);
    final FlinkYarnSessionCli yarnCLI = createFlinkYarnSessionCli(configuration);
    final CommandLine commandLine = yarnCLI.parseCommandLineOptions(params, true);
    configuration.addAll(yarnCLI.toConfiguration(commandLine));
    final ClusterClientFactory<ApplicationId> clientFactory = getClusterClientFactory(configuration);
    final ClusterSpecification clusterSpecification = clientFactory.getClusterSpecification(configuration);
    // each task manager has 3 slots but the parallelism is 7. Thus the slots should be
    // increased.
    assertEquals(3, clusterSpecification.getSlotsPerTaskManager());
}
Also used : CustomCommandLine(org.apache.flink.client.cli.CustomCommandLine) CommandLine(org.apache.commons.cli.CommandLine) Configuration(org.apache.flink.configuration.Configuration) ClusterSpecification(org.apache.flink.client.deployment.ClusterSpecification) ApplicationId(org.apache.hadoop.yarn.api.records.ApplicationId) FlinkYarnSessionCli(org.apache.flink.yarn.cli.FlinkYarnSessionCli) Test(org.junit.Test)

Aggregations

FlinkYarnSessionCli (org.apache.flink.yarn.cli.FlinkYarnSessionCli)23 Test (org.junit.Test)22 CommandLine (org.apache.commons.cli.CommandLine)21 CustomCommandLine (org.apache.flink.client.cli.CustomCommandLine)21 Configuration (org.apache.flink.configuration.Configuration)20 ApplicationId (org.apache.hadoop.yarn.api.records.ApplicationId)16 ClusterSpecification (org.apache.flink.client.deployment.ClusterSpecification)8 File (java.io.File)5 CommandLineParser (org.apache.commons.cli.CommandLineParser)1 DefaultParser (org.apache.commons.cli.DefaultParser)1 Options (org.apache.commons.cli.Options)1 CliFrontend (org.apache.flink.client.CliFrontend)1 RunOptions (org.apache.flink.client.cli.RunOptions)1 ClusterClientServiceLoader (org.apache.flink.client.deployment.ClusterClientServiceLoader)1 DefaultClusterClientServiceLoader (org.apache.flink.client.deployment.DefaultClusterClientServiceLoader)1 AkkaOptions (org.apache.flink.configuration.AkkaOptions)1 CoreOptions (org.apache.flink.configuration.CoreOptions)1 DeploymentOptions (org.apache.flink.configuration.DeploymentOptions)1 HighAvailabilityOptions (org.apache.flink.configuration.HighAvailabilityOptions)1 JobManagerOptions (org.apache.flink.configuration.JobManagerOptions)1