Search in sources :

Example 16 with PackagedProgram

use of org.apache.flink.client.program.PackagedProgram in project flink by apache.

the class ClassLoaderITCase method testStreamingClassloaderJobWithCustomClassLoader.

@Test
public void testStreamingClassloaderJobWithCustomClassLoader() throws ProgramInvocationException {
    // regular streaming job
    PackagedProgram streamingProg = PackagedProgram.newBuilder().setJarFile(new File(STREAMING_PROG_JAR_FILE)).build();
    TestStreamEnvironment.setAsContext(miniClusterResource.getMiniCluster(), parallelism, Collections.singleton(new Path(STREAMING_PROG_JAR_FILE)), Collections.emptyList());
    streamingProg.invokeInteractiveModeForExecution();
}
Also used : Path(org.apache.flink.core.fs.Path) PackagedProgram(org.apache.flink.client.program.PackagedProgram) File(java.io.File) Test(org.junit.Test)

Example 17 with PackagedProgram

use of org.apache.flink.client.program.PackagedProgram in project flink by apache.

the class ClassLoaderITCase method testKMeansJobWithCustomClassLoader.

@Test
public void testKMeansJobWithCustomClassLoader() throws ProgramInvocationException {
    PackagedProgram kMeansProg = PackagedProgram.newBuilder().setJarFile(new File(KMEANS_JAR_PATH)).setArguments(new String[] { KMeansData.DATAPOINTS, KMeansData.INITIAL_CENTERS, "25" }).build();
    TestEnvironment.setAsContext(miniClusterResource.getMiniCluster(), parallelism, Collections.singleton(new Path(KMEANS_JAR_PATH)), Collections.emptyList());
    kMeansProg.invokeInteractiveModeForExecution();
}
Also used : Path(org.apache.flink.core.fs.Path) PackagedProgram(org.apache.flink.client.program.PackagedProgram) File(java.io.File) Test(org.junit.Test)

Example 18 with PackagedProgram

use of org.apache.flink.client.program.PackagedProgram in project beam by apache.

the class FlinkRunnerTest method testEnsureStdoutStdErrIsRestored.

@Test
public void testEnsureStdoutStdErrIsRestored() throws Exception {
    PackagedProgram packagedProgram = getPackagedProgram();
    OptimizerPlanEnvironment env = getOptimizerPlanEnvironment();
    try {
        // Flink will throw an error because no job graph will be generated by the main method
        getPipeline(env, packagedProgram);
        Assert.fail("This should have failed to create the Flink Plan.");
    } catch (ProgramInvocationException e) {
        // Test that Flink wasn't able to intercept the stdout/stderr and we printed to the regular
        // output instead
        MatcherAssert.assertThat(e.getMessage(), allOf(StringContains.containsString("System.out: (none)"), StringContains.containsString("System.err: (none)")));
    }
}
Also used : PackagedProgram(org.apache.flink.client.program.PackagedProgram) OptimizerPlanEnvironment(org.apache.flink.client.program.OptimizerPlanEnvironment) ProgramInvocationException(org.apache.flink.client.program.ProgramInvocationException) Test(org.junit.Test)

Example 19 with PackagedProgram

use of org.apache.flink.client.program.PackagedProgram in project flink by apache.

the class JarActionHandler method getJobGraphAndClassLoader.

protected Tuple2<JobGraph, ClassLoader> getJobGraphAndClassLoader(JarActionHandlerConfig config) throws Exception {
    // generate the graph
    JobGraph graph = null;
    PackagedProgram program = new PackagedProgram(new File(jarDir, config.getJarFile()), config.getEntryClass(), config.getProgramArgs());
    ClassLoader classLoader = program.getUserCodeClassLoader();
    Optimizer optimizer = new Optimizer(new DataStatistics(), new DefaultCostEstimator(), new Configuration());
    FlinkPlan plan = ClusterClient.getOptimizedPlan(optimizer, program, config.getParallelism());
    if (plan instanceof StreamingPlan) {
        graph = ((StreamingPlan) plan).getJobGraph();
    } else if (plan instanceof OptimizedPlan) {
        graph = new JobGraphGenerator().compileJobGraph((OptimizedPlan) plan);
    }
    if (graph == null) {
        throw new CompilerException("A valid job graph couldn't be generated for the jar.");
    }
    // Set the savepoint settings
    graph.setSavepointRestoreSettings(config.getSavepointRestoreSettings());
    for (URL jar : program.getAllLibraries()) {
        try {
            graph.addJar(new Path(jar.toURI()));
        } catch (URISyntaxException e) {
            throw new ProgramInvocationException("Invalid jar path. Unexpected error. :(");
        }
    }
    return Tuple2.of(graph, classLoader);
}
Also used : Path(org.apache.flink.core.fs.Path) Configuration(org.apache.flink.configuration.Configuration) StreamingPlan(org.apache.flink.optimizer.plan.StreamingPlan) Optimizer(org.apache.flink.optimizer.Optimizer) DataStatistics(org.apache.flink.optimizer.DataStatistics) FlinkPlan(org.apache.flink.optimizer.plan.FlinkPlan) URISyntaxException(java.net.URISyntaxException) URL(java.net.URL) OptimizedPlan(org.apache.flink.optimizer.plan.OptimizedPlan) PackagedProgram(org.apache.flink.client.program.PackagedProgram) JobGraph(org.apache.flink.runtime.jobgraph.JobGraph) JobGraphGenerator(org.apache.flink.optimizer.plantranslate.JobGraphGenerator) CompilerException(org.apache.flink.optimizer.CompilerException) ProgramInvocationException(org.apache.flink.client.program.ProgramInvocationException) DefaultCostEstimator(org.apache.flink.optimizer.costs.DefaultCostEstimator) File(java.io.File)

Example 20 with PackagedProgram

use of org.apache.flink.client.program.PackagedProgram in project flink by apache.

the class CliFrontend method buildProgram.

/**
	 * Creates a Packaged program from the given command line options.
	 *
	 * @return A PackagedProgram (upon success)
	 * @throws java.io.FileNotFoundException
	 * @throws org.apache.flink.client.program.ProgramInvocationException
	 */
protected PackagedProgram buildProgram(ProgramOptions options) throws FileNotFoundException, ProgramInvocationException {
    String[] programArgs = options.getProgramArgs();
    String jarFilePath = options.getJarFilePath();
    List<URL> classpaths = options.getClasspaths();
    if (jarFilePath == null) {
        throw new IllegalArgumentException("The program JAR file was not specified.");
    }
    File jarFile = new File(jarFilePath);
    // Check if JAR file exists
    if (!jarFile.exists()) {
        throw new FileNotFoundException("JAR file does not exist: " + jarFile);
    } else if (!jarFile.isFile()) {
        throw new FileNotFoundException("JAR file is not a file: " + jarFile);
    }
    // Get assembler class
    String entryPointClass = options.getEntryPointClassName();
    PackagedProgram program = entryPointClass == null ? new PackagedProgram(jarFile, classpaths, programArgs) : new PackagedProgram(jarFile, classpaths, entryPointClass, programArgs);
    program.setSavepointRestoreSettings(options.getSavepointRestoreSettings());
    return program;
}
Also used : PackagedProgram(org.apache.flink.client.program.PackagedProgram) FileNotFoundException(java.io.FileNotFoundException) File(java.io.File) URL(java.net.URL)

Aggregations

PackagedProgram (org.apache.flink.client.program.PackagedProgram)42 Test (org.junit.Test)25 File (java.io.File)20 Configuration (org.apache.flink.configuration.Configuration)19 URL (java.net.URL)13 ProgramInvocationException (org.apache.flink.client.program.ProgramInvocationException)12 Path (org.apache.flink.core.fs.Path)12 FileNotFoundException (java.io.FileNotFoundException)7 IOException (java.io.IOException)7 CommandLine (org.apache.commons.cli.CommandLine)6 RunOptions (org.apache.flink.client.cli.RunOptions)5 CompilerException (org.apache.flink.optimizer.CompilerException)5 FlinkException (org.apache.flink.util.FlinkException)5 Pipeline (org.apache.flink.api.dag.Pipeline)4 DataStatistics (org.apache.flink.optimizer.DataStatistics)4 Optimizer (org.apache.flink.optimizer.Optimizer)4 DefaultCostEstimator (org.apache.flink.optimizer.costs.DefaultCostEstimator)4 JobID (org.apache.flink.api.common.JobID)3 ApplicationConfiguration (org.apache.flink.client.deployment.application.ApplicationConfiguration)3 MiniClusterResourceConfiguration (org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration)3