Search in sources :

Example 1 with JobListener

use of org.apache.flink.core.execution.JobListener in project flink by apache.

the class StreamContextEnvironment method execute.

@Override
public JobExecutionResult execute(StreamGraph streamGraph) throws Exception {
    checkNotAllowedConfigurations();
    final JobClient jobClient = executeAsync(streamGraph);
    final List<JobListener> jobListeners = getJobListeners();
    try {
        final JobExecutionResult jobExecutionResult = getJobExecutionResult(jobClient);
        jobListeners.forEach(jobListener -> jobListener.onJobExecuted(jobExecutionResult, null));
        return jobExecutionResult;
    } catch (Throwable t) {
        jobListeners.forEach(jobListener -> jobListener.onJobExecuted(null, ExceptionUtils.stripExecutionException(t)));
        ExceptionUtils.rethrowException(t);
        // never reached, only make javac happy
        return null;
    }
}
Also used : Arrays(java.util.Arrays) ByteArrayOutputStream(java.io.ByteArrayOutputStream) LoggerFactory(org.slf4j.LoggerFactory) PublicEvolving(org.apache.flink.annotation.PublicEvolving) PipelineExecutorServiceLoader(org.apache.flink.core.execution.PipelineExecutorServiceLoader) ExceptionUtils(org.apache.flink.util.ExceptionUtils) CompletableFuture(java.util.concurrent.CompletableFuture) ShutdownHookUtil(org.apache.flink.util.ShutdownHookUtil) ArrayList(java.util.ArrayList) JobListener(org.apache.flink.core.execution.JobListener) StreamGraph(org.apache.flink.streaming.api.graph.StreamGraph) DeploymentOptions(org.apache.flink.configuration.DeploymentOptions) ObjectOutputStream(java.io.ObjectOutputStream) Preconditions.checkNotNull(org.apache.flink.util.Preconditions.checkNotNull) DetachedJobExecutionResult(org.apache.flink.core.execution.DetachedJobExecutionResult) Logger(org.slf4j.Logger) FlinkRuntimeException(org.apache.flink.util.FlinkRuntimeException) StreamExecutionEnvironmentFactory(org.apache.flink.streaming.api.environment.StreamExecutionEnvironmentFactory) Collection(java.util.Collection) Configuration(org.apache.flink.configuration.Configuration) IOException(java.io.IOException) JobClient(org.apache.flink.core.execution.JobClient) Serializable(java.io.Serializable) JobExecutionResult(org.apache.flink.api.common.JobExecutionResult) TimeUnit(java.util.concurrent.TimeUnit) MapDifference(org.apache.flink.shaded.guava30.com.google.common.collect.MapDifference) Maps(org.apache.flink.shaded.guava30.com.google.common.collect.Maps) List(java.util.List) ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) ConfigurationNotAllowedMessage(org.apache.flink.runtime.dispatcher.ConfigurationNotAllowedMessage) Internal(org.apache.flink.annotation.Internal) Collections(java.util.Collections) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) DetachedJobExecutionResult(org.apache.flink.core.execution.DetachedJobExecutionResult) JobExecutionResult(org.apache.flink.api.common.JobExecutionResult) JobListener(org.apache.flink.core.execution.JobListener) JobClient(org.apache.flink.core.execution.JobClient)

Example 2 with JobListener

use of org.apache.flink.core.execution.JobListener in project flink by apache.

the class ContextEnvironment method execute.

@Override
public JobExecutionResult execute(String jobName) throws Exception {
    final JobClient jobClient = executeAsync(jobName);
    final List<JobListener> jobListeners = getJobListeners();
    try {
        final JobExecutionResult jobExecutionResult = getJobExecutionResult(jobClient);
        jobListeners.forEach(jobListener -> jobListener.onJobExecuted(jobExecutionResult, null));
        return jobExecutionResult;
    } catch (Throwable t) {
        jobListeners.forEach(jobListener -> jobListener.onJobExecuted(null, ExceptionUtils.stripExecutionException(t)));
        ExceptionUtils.rethrowException(t);
        // never reached, only make javac happy
        return null;
    }
}
Also used : DetachedJobExecutionResult(org.apache.flink.core.execution.DetachedJobExecutionResult) ExecutionEnvironmentFactory(org.apache.flink.api.java.ExecutionEnvironmentFactory) Logger(org.slf4j.Logger) FlinkRuntimeException(org.apache.flink.util.FlinkRuntimeException) Configuration(org.apache.flink.configuration.Configuration) LoggerFactory(org.slf4j.LoggerFactory) PipelineExecutorServiceLoader(org.apache.flink.core.execution.PipelineExecutorServiceLoader) ExceptionUtils(org.apache.flink.util.ExceptionUtils) CompletableFuture(java.util.concurrent.CompletableFuture) ShutdownHookUtil(org.apache.flink.util.ShutdownHookUtil) JobClient(org.apache.flink.core.execution.JobClient) JobExecutionResult(org.apache.flink.api.common.JobExecutionResult) TimeUnit(java.util.concurrent.TimeUnit) List(java.util.List) ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) JobListener(org.apache.flink.core.execution.JobListener) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) DeploymentOptions(org.apache.flink.configuration.DeploymentOptions) Preconditions.checkNotNull(org.apache.flink.util.Preconditions.checkNotNull) DetachedJobExecutionResult(org.apache.flink.core.execution.DetachedJobExecutionResult) JobExecutionResult(org.apache.flink.api.common.JobExecutionResult) JobListener(org.apache.flink.core.execution.JobListener) JobClient(org.apache.flink.core.execution.JobClient)

Example 3 with JobListener

use of org.apache.flink.core.execution.JobListener in project flink by apache.

the class ExecutionEnvironment method executeAsync.

/**
 * Triggers the program execution asynchronously. The environment will execute all parts of the
 * program that have resulted in a "sink" operation. Sink operations are for example printing
 * results ({@link DataSet#print()}, writing results (e.g. {@link DataSet#writeAsText(String)},
 * {@link DataSet#write(org.apache.flink.api.common.io.FileOutputFormat, String)}, or other
 * generic data sinks created with {@link
 * DataSet#output(org.apache.flink.api.common.io.OutputFormat)}.
 *
 * <p>The program execution will be logged and displayed with the given job name.
 *
 * @return A {@link JobClient} that can be used to communicate with the submitted job, completed
 *     on submission succeeded.
 * @throws Exception Thrown, if the program submission fails.
 */
@PublicEvolving
public JobClient executeAsync(String jobName) throws Exception {
    checkNotNull(configuration.get(DeploymentOptions.TARGET), "No execution.target specified in your configuration file.");
    final Plan plan = createProgramPlan(jobName);
    final PipelineExecutorFactory executorFactory = executorServiceLoader.getExecutorFactory(configuration);
    checkNotNull(executorFactory, "Cannot find compatible factory for specified execution.target (=%s)", configuration.get(DeploymentOptions.TARGET));
    CompletableFuture<JobClient> jobClientFuture = executorFactory.getExecutor(configuration).execute(plan, configuration, userClassloader);
    try {
        JobClient jobClient = jobClientFuture.get();
        jobListeners.forEach(jobListener -> jobListener.onJobSubmitted(jobClient, null));
        return jobClient;
    } catch (Throwable t) {
        jobListeners.forEach(jobListener -> jobListener.onJobSubmitted(null, t));
        ExceptionUtils.rethrow(t);
        // make javac happy, this code path will not be reached
        return null;
    }
}
Also used : CollectionInputFormat(org.apache.flink.api.java.io.CollectionInputFormat) PojoTypeInfo(org.apache.flink.api.java.typeutils.PojoTypeInfo) Arrays(java.util.Arrays) Tuple2(org.apache.flink.api.java.tuple.Tuple2) PipelineOptions(org.apache.flink.configuration.PipelineOptions) LoggerFactory(org.slf4j.LoggerFactory) ExceptionUtils(org.apache.flink.util.ExceptionUtils) FileInputFormat(org.apache.flink.api.common.io.FileInputFormat) DataSink(org.apache.flink.api.java.operators.DataSink) BasicTypeInfo(org.apache.flink.api.common.typeinfo.BasicTypeInfo) InstantiationUtil(org.apache.flink.util.InstantiationUtil) WrappingRuntimeException(org.apache.flink.util.WrappingRuntimeException) SplittableIterator(org.apache.flink.util.SplittableIterator) InvalidProgramException(org.apache.flink.api.common.InvalidProgramException) Path(org.apache.flink.core.fs.Path) Plan(org.apache.flink.api.common.Plan) PrimitiveInputFormat(org.apache.flink.api.java.io.PrimitiveInputFormat) InputFormat(org.apache.flink.api.common.io.InputFormat) Public(org.apache.flink.annotation.Public) Preconditions.checkNotNull(org.apache.flink.util.Preconditions.checkNotNull) TypeInformation(org.apache.flink.api.common.typeinfo.TypeInformation) DefaultExecutorServiceLoader(org.apache.flink.core.execution.DefaultExecutorServiceLoader) ValueTypeInfo(org.apache.flink.api.java.typeutils.ValueTypeInfo) PipelineExecutorFactory(org.apache.flink.core.execution.PipelineExecutorFactory) Collection(java.util.Collection) DistributedCacheEntry(org.apache.flink.api.common.cache.DistributedCache.DistributedCacheEntry) DistributedCache(org.apache.flink.api.common.cache.DistributedCache) Preconditions(org.apache.flink.util.Preconditions) Serializable(java.io.Serializable) JobExecutionResult(org.apache.flink.api.common.JobExecutionResult) List(java.util.List) TextValueInputFormat(org.apache.flink.api.java.io.TextValueInputFormat) TypeExtractor(org.apache.flink.api.java.typeutils.TypeExtractor) ResultTypeQueryable(org.apache.flink.api.java.typeutils.ResultTypeQueryable) DataSource(org.apache.flink.api.java.operators.DataSource) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) IteratorInputFormat(org.apache.flink.api.java.io.IteratorInputFormat) FlinkException(org.apache.flink.util.FlinkException) CsvReader(org.apache.flink.api.java.io.CsvReader) PublicEvolving(org.apache.flink.annotation.PublicEvolving) PipelineExecutorServiceLoader(org.apache.flink.core.execution.PipelineExecutorServiceLoader) CompletableFuture(java.util.concurrent.CompletableFuture) RestartStrategies(org.apache.flink.api.common.restartstrategy.RestartStrategies) PipelineExecutor(org.apache.flink.core.execution.PipelineExecutor) ArrayList(java.util.ArrayList) Calendar(java.util.Calendar) ReadableConfig(org.apache.flink.configuration.ReadableConfig) JobListener(org.apache.flink.core.execution.JobListener) DeploymentOptions(org.apache.flink.configuration.DeploymentOptions) RestOptions(org.apache.flink.configuration.RestOptions) DetachedJobExecutionResult(org.apache.flink.core.execution.DetachedJobExecutionResult) Serializer(com.esotericsoftware.kryo.Serializer) Logger(org.slf4j.Logger) Iterator(java.util.Iterator) ParallelIteratorInputFormat(org.apache.flink.api.java.io.ParallelIteratorInputFormat) Configuration(org.apache.flink.configuration.Configuration) JobClient(org.apache.flink.core.execution.JobClient) Operator(org.apache.flink.api.java.operators.Operator) TextInputFormat(org.apache.flink.api.java.io.TextInputFormat) StringValue(org.apache.flink.types.StringValue) NumberSequenceIterator(org.apache.flink.util.NumberSequenceIterator) Internal(org.apache.flink.annotation.Internal) PlanGenerator(org.apache.flink.api.java.utils.PlanGenerator) PipelineExecutorFactory(org.apache.flink.core.execution.PipelineExecutorFactory) Plan(org.apache.flink.api.common.Plan) JobClient(org.apache.flink.core.execution.JobClient) PublicEvolving(org.apache.flink.annotation.PublicEvolving)

Example 4 with JobListener

use of org.apache.flink.core.execution.JobListener in project flink by apache.

the class JobListenerITCase method testExecuteCallsJobListenerOnMainThreadOnBatchEnvironment.

@Test
public void testExecuteCallsJobListenerOnMainThreadOnBatchEnvironment() throws Exception {
    AtomicReference<Thread> threadReference = new AtomicReference<>();
    ExecutionEnvironment env = new ExecutionEnvironment(getClientConfiguration());
    env.registerJobListener(new JobListener() {

        @Override
        public void onJobSubmitted(JobClient jobClient, Throwable t) {
            threadReference.set(Thread.currentThread());
        }

        @Override
        public void onJobExecuted(JobExecutionResult jobExecutionResult, Throwable throwable) {
        }
    });
    env.fromElements(1, 2, 3, 4, 5).output(new DiscardingOutputFormat<>());
    env.execute();
    assertThat(Thread.currentThread(), is(threadReference.get()));
}
Also used : JobExecutionResult(org.apache.flink.api.common.JobExecutionResult) ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) AtomicReference(java.util.concurrent.atomic.AtomicReference) JobListener(org.apache.flink.core.execution.JobListener) JobClient(org.apache.flink.core.execution.JobClient) Test(org.junit.Test)

Example 5 with JobListener

use of org.apache.flink.core.execution.JobListener in project flink by apache.

the class JobListenerITCase method testExecuteAsyncCallsJobListenerOnStreamingEnvironment.

@Test
public void testExecuteAsyncCallsJobListenerOnStreamingEnvironment() throws Exception {
    AtomicReference<JobID> jobIdReference = new AtomicReference<>();
    OneShotLatch submissionLatch = new OneShotLatch();
    StreamExecutionEnvironment env = new StreamExecutionEnvironment(getClientConfiguration());
    env.registerJobListener(new JobListener() {

        @Override
        public void onJobSubmitted(JobClient jobClient, Throwable t) {
            jobIdReference.set(jobClient.getJobID());
            submissionLatch.trigger();
        }

        @Override
        public void onJobExecuted(JobExecutionResult jobExecutionResult, Throwable throwable) {
        }
    });
    env.fromElements(1, 2, 3, 4, 5).addSink(new DiscardingSink<>());
    JobClient jobClient = env.executeAsync();
    submissionLatch.await(2000L, TimeUnit.MILLISECONDS);
    // when executing asynchronously we don't get an "executed" callback
    assertThat(jobClient.getJobID(), is(jobIdReference.get()));
}
Also used : JobExecutionResult(org.apache.flink.api.common.JobExecutionResult) OneShotLatch(org.apache.flink.core.testutils.OneShotLatch) AtomicReference(java.util.concurrent.atomic.AtomicReference) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) JobListener(org.apache.flink.core.execution.JobListener) JobClient(org.apache.flink.core.execution.JobClient) JobID(org.apache.flink.api.common.JobID) Test(org.junit.Test)

Aggregations

JobListener (org.apache.flink.core.execution.JobListener)16 JobExecutionResult (org.apache.flink.api.common.JobExecutionResult)14 JobClient (org.apache.flink.core.execution.JobClient)14 StreamExecutionEnvironment (org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)9 AtomicReference (java.util.concurrent.atomic.AtomicReference)8 Test (org.junit.Test)8 ExecutionEnvironment (org.apache.flink.api.java.ExecutionEnvironment)6 List (java.util.List)5 CompletableFuture (java.util.concurrent.CompletableFuture)5 Configuration (org.apache.flink.configuration.Configuration)5 DeploymentOptions (org.apache.flink.configuration.DeploymentOptions)5 DetachedJobExecutionResult (org.apache.flink.core.execution.DetachedJobExecutionResult)5 PipelineExecutorServiceLoader (org.apache.flink.core.execution.PipelineExecutorServiceLoader)5 ExceptionUtils (org.apache.flink.util.ExceptionUtils)5 FlinkException (org.apache.flink.util.FlinkException)5 Preconditions.checkNotNull (org.apache.flink.util.Preconditions.checkNotNull)5 WrappingRuntimeException (org.apache.flink.util.WrappingRuntimeException)5 Serializable (java.io.Serializable)4 ArrayList (java.util.ArrayList)4 Arrays (java.util.Arrays)4