Search in sources :

Example 11 with FlinkYarnSessionCli

use of org.apache.flink.yarn.cli.FlinkYarnSessionCli in project flink by apache.

the class FlinkYarnSessionCliTest method testMemoryPropertyWithUnitMB.

/**
 * Tests the specifying total process memory with unit (MB) for job manager and task manager.
 */
@Test
public void testMemoryPropertyWithUnitMB() throws Exception {
    final String[] args = new String[] { "-yjm", "1024m", "-ytm", "2048m" };
    final FlinkYarnSessionCli flinkYarnSessionCli = createFlinkYarnSessionCli();
    final CommandLine commandLine = flinkYarnSessionCli.parseCommandLineOptions(args, false);
    final Configuration executorConfig = flinkYarnSessionCli.toConfiguration(commandLine);
    final ClusterClientFactory<ApplicationId> clientFactory = getClusterClientFactory(executorConfig);
    final ClusterSpecification clusterSpecification = clientFactory.getClusterSpecification(executorConfig);
    assertThat(clusterSpecification.getMasterMemoryMB(), is(1024));
    assertThat(clusterSpecification.getTaskManagerMemoryMB(), is(2048));
}
Also used : CustomCommandLine(org.apache.flink.client.cli.CustomCommandLine) CommandLine(org.apache.commons.cli.CommandLine) Configuration(org.apache.flink.configuration.Configuration) ClusterSpecification(org.apache.flink.client.deployment.ClusterSpecification) ApplicationId(org.apache.hadoop.yarn.api.records.ApplicationId) FlinkYarnSessionCli(org.apache.flink.yarn.cli.FlinkYarnSessionCli) Test(org.junit.Test)

Example 12 with FlinkYarnSessionCli

use of org.apache.flink.yarn.cli.FlinkYarnSessionCli in project flink by apache.

the class CliFrontendRunWithYarnTest method testRun.

@Test
public void testRun() throws Exception {
    String testJarPath = getTestJarPath("BatchWordCount.jar").getAbsolutePath();
    Configuration configuration = new Configuration();
    configuration.setString(JobManagerOptions.ADDRESS, "localhost");
    configuration.setInteger(JobManagerOptions.PORT, 8081);
    configuration.set(TaskManagerOptions.TOTAL_FLINK_MEMORY, MemorySize.parse("1g"));
    final ClusterClientServiceLoader testServiceLoader = new DefaultClusterClientServiceLoader();
    final FlinkYarnSessionCli yarnCLI = new FlinkYarnSessionCli(configuration, testServiceLoader, tmp.getRoot().getAbsolutePath(), "y", "yarn", true);
    // test detached mode
    {
        String[] parameters = { "-m", "yarn-cluster", "-p", "2", "-d", testJarPath };
        verifyCliFrontend(configuration, testServiceLoader, yarnCLI, parameters, 2, true);
    }
    // test detached mode
    {
        String[] parameters = { "-m", "yarn-cluster", "-p", "2", "-yd", testJarPath };
        verifyCliFrontend(configuration, testServiceLoader, yarnCLI, parameters, 2, true);
    }
}
Also used : Configuration(org.apache.flink.configuration.Configuration) DefaultClusterClientServiceLoader(org.apache.flink.client.deployment.DefaultClusterClientServiceLoader) DefaultClusterClientServiceLoader(org.apache.flink.client.deployment.DefaultClusterClientServiceLoader) ClusterClientServiceLoader(org.apache.flink.client.deployment.ClusterClientServiceLoader) FlinkYarnSessionCli(org.apache.flink.yarn.cli.FlinkYarnSessionCli) Test(org.junit.Test)

Example 13 with FlinkYarnSessionCli

use of org.apache.flink.yarn.cli.FlinkYarnSessionCli in project flink by apache.

the class FlinkYarnSessionCliTest method testNotEnoughTaskSlots.

@Test
public void testNotEnoughTaskSlots() throws Exception {
    File confFile = tmp.newFile("flink-conf.yaml");
    File jarFile = tmp.newFile("test.jar");
    new CliFrontend(tmp.getRoot().getAbsolutePath());
    String[] params = new String[] { "-yn", "2", "-ys", "3", "-p", "7", jarFile.getAbsolutePath() };
    RunOptions runOptions = CliFrontendParser.parseRunCommand(params);
    FlinkYarnSessionCli yarnCLI = new TestCLI("y", "yarn");
    AbstractYarnClusterDescriptor descriptor = yarnCLI.createDescriptor("", runOptions.getCommandLine());
    // each task manager has 3 slots but the parallelism is 7. Thus the slots should be increased.
    Assert.assertEquals(4, descriptor.getTaskManagerSlots());
    Assert.assertEquals(2, descriptor.getTaskManagerCount());
}
Also used : CliFrontend(org.apache.flink.client.CliFrontend) File(java.io.File) RunOptions(org.apache.flink.client.cli.RunOptions) FlinkYarnSessionCli(org.apache.flink.yarn.cli.FlinkYarnSessionCli) Test(org.junit.Test)

Example 14 with FlinkYarnSessionCli

use of org.apache.flink.yarn.cli.FlinkYarnSessionCli in project flink by apache.

the class FlinkYarnSessionCliTest method testMemoryPropertyWithArbitraryUnit.

/**
 * Tests the specifying total process memory with arbitrary unit for job manager and task
 * manager.
 */
@Test
public void testMemoryPropertyWithArbitraryUnit() throws Exception {
    final String[] args = new String[] { "-yjm", "1g", "-ytm", "2g" };
    final FlinkYarnSessionCli flinkYarnSessionCli = createFlinkYarnSessionCli();
    final CommandLine commandLine = flinkYarnSessionCli.parseCommandLineOptions(args, false);
    final Configuration executorConfig = flinkYarnSessionCli.toConfiguration(commandLine);
    final ClusterClientFactory<ApplicationId> clientFactory = getClusterClientFactory(executorConfig);
    final ClusterSpecification clusterSpecification = clientFactory.getClusterSpecification(executorConfig);
    assertThat(clusterSpecification.getMasterMemoryMB(), is(1024));
    assertThat(clusterSpecification.getTaskManagerMemoryMB(), is(2048));
}
Also used : CustomCommandLine(org.apache.flink.client.cli.CustomCommandLine) CommandLine(org.apache.commons.cli.CommandLine) Configuration(org.apache.flink.configuration.Configuration) ClusterSpecification(org.apache.flink.client.deployment.ClusterSpecification) ApplicationId(org.apache.hadoop.yarn.api.records.ApplicationId) FlinkYarnSessionCli(org.apache.flink.yarn.cli.FlinkYarnSessionCli) Test(org.junit.Test)

Example 15 with FlinkYarnSessionCli

use of org.apache.flink.yarn.cli.FlinkYarnSessionCli in project flink by apache.

the class FlinkYarnSessionCliTest method testNodeLabelProperty.

@Test
public void testNodeLabelProperty() throws Exception {
    String nodeLabelCliInput = "flink_test_nodelabel";
    String[] params = new String[] { "-ynl", nodeLabelCliInput };
    FlinkYarnSessionCli yarnCLI = createFlinkYarnSessionCli();
    CommandLine commandLine = yarnCLI.parseCommandLineOptions(params, true);
    Configuration executorConfig = yarnCLI.toConfiguration(commandLine);
    ClusterClientFactory<ApplicationId> clientFactory = getClusterClientFactory(executorConfig);
    YarnClusterDescriptor descriptor = (YarnClusterDescriptor) clientFactory.createClusterDescriptor(executorConfig);
    assertEquals(nodeLabelCliInput, descriptor.getNodeLabel());
}
Also used : CustomCommandLine(org.apache.flink.client.cli.CustomCommandLine) CommandLine(org.apache.commons.cli.CommandLine) Configuration(org.apache.flink.configuration.Configuration) ApplicationId(org.apache.hadoop.yarn.api.records.ApplicationId) FlinkYarnSessionCli(org.apache.flink.yarn.cli.FlinkYarnSessionCli) Test(org.junit.Test)

Aggregations

FlinkYarnSessionCli (org.apache.flink.yarn.cli.FlinkYarnSessionCli)23 Test (org.junit.Test)22 CommandLine (org.apache.commons.cli.CommandLine)21 CustomCommandLine (org.apache.flink.client.cli.CustomCommandLine)21 Configuration (org.apache.flink.configuration.Configuration)20 ApplicationId (org.apache.hadoop.yarn.api.records.ApplicationId)16 ClusterSpecification (org.apache.flink.client.deployment.ClusterSpecification)8 File (java.io.File)5 CommandLineParser (org.apache.commons.cli.CommandLineParser)1 DefaultParser (org.apache.commons.cli.DefaultParser)1 Options (org.apache.commons.cli.Options)1 CliFrontend (org.apache.flink.client.CliFrontend)1 RunOptions (org.apache.flink.client.cli.RunOptions)1 ClusterClientServiceLoader (org.apache.flink.client.deployment.ClusterClientServiceLoader)1 DefaultClusterClientServiceLoader (org.apache.flink.client.deployment.DefaultClusterClientServiceLoader)1 AkkaOptions (org.apache.flink.configuration.AkkaOptions)1 CoreOptions (org.apache.flink.configuration.CoreOptions)1 DeploymentOptions (org.apache.flink.configuration.DeploymentOptions)1 HighAvailabilityOptions (org.apache.flink.configuration.HighAvailabilityOptions)1 JobManagerOptions (org.apache.flink.configuration.JobManagerOptions)1