Search in sources :

Example 1 with PipelineExecutorFactory

use of org.apache.flink.core.execution.PipelineExecutorFactory in project flink by apache.

the class ExecutionEnvironment method executeAsync.

/**
 * Triggers the program execution asynchronously. The environment will execute all parts of the
 * program that have resulted in a "sink" operation. Sink operations are for example printing
 * results ({@link DataSet#print()}, writing results (e.g. {@link DataSet#writeAsText(String)},
 * {@link DataSet#write(org.apache.flink.api.common.io.FileOutputFormat, String)}, or other
 * generic data sinks created with {@link
 * DataSet#output(org.apache.flink.api.common.io.OutputFormat)}.
 *
 * <p>The program execution will be logged and displayed with the given job name.
 *
 * @return A {@link JobClient} that can be used to communicate with the submitted job, completed
 *     on submission succeeded.
 * @throws Exception Thrown, if the program submission fails.
 */
@PublicEvolving
public JobClient executeAsync(String jobName) throws Exception {
    checkNotNull(configuration.get(DeploymentOptions.TARGET), "No execution.target specified in your configuration file.");
    final Plan plan = createProgramPlan(jobName);
    final PipelineExecutorFactory executorFactory = executorServiceLoader.getExecutorFactory(configuration);
    checkNotNull(executorFactory, "Cannot find compatible factory for specified execution.target (=%s)", configuration.get(DeploymentOptions.TARGET));
    CompletableFuture<JobClient> jobClientFuture = executorFactory.getExecutor(configuration).execute(plan, configuration, userClassloader);
    try {
        JobClient jobClient = jobClientFuture.get();
        jobListeners.forEach(jobListener -> jobListener.onJobSubmitted(jobClient, null));
        return jobClient;
    } catch (Throwable t) {
        jobListeners.forEach(jobListener -> jobListener.onJobSubmitted(null, t));
        ExceptionUtils.rethrow(t);
        // make javac happy, this code path will not be reached
        return null;
    }
}
Also used : CollectionInputFormat(org.apache.flink.api.java.io.CollectionInputFormat) PojoTypeInfo(org.apache.flink.api.java.typeutils.PojoTypeInfo) Arrays(java.util.Arrays) Tuple2(org.apache.flink.api.java.tuple.Tuple2) PipelineOptions(org.apache.flink.configuration.PipelineOptions) LoggerFactory(org.slf4j.LoggerFactory) ExceptionUtils(org.apache.flink.util.ExceptionUtils) FileInputFormat(org.apache.flink.api.common.io.FileInputFormat) DataSink(org.apache.flink.api.java.operators.DataSink) BasicTypeInfo(org.apache.flink.api.common.typeinfo.BasicTypeInfo) InstantiationUtil(org.apache.flink.util.InstantiationUtil) WrappingRuntimeException(org.apache.flink.util.WrappingRuntimeException) SplittableIterator(org.apache.flink.util.SplittableIterator) InvalidProgramException(org.apache.flink.api.common.InvalidProgramException) Path(org.apache.flink.core.fs.Path) Plan(org.apache.flink.api.common.Plan) PrimitiveInputFormat(org.apache.flink.api.java.io.PrimitiveInputFormat) InputFormat(org.apache.flink.api.common.io.InputFormat) Public(org.apache.flink.annotation.Public) Preconditions.checkNotNull(org.apache.flink.util.Preconditions.checkNotNull) TypeInformation(org.apache.flink.api.common.typeinfo.TypeInformation) DefaultExecutorServiceLoader(org.apache.flink.core.execution.DefaultExecutorServiceLoader) ValueTypeInfo(org.apache.flink.api.java.typeutils.ValueTypeInfo) PipelineExecutorFactory(org.apache.flink.core.execution.PipelineExecutorFactory) Collection(java.util.Collection) DistributedCacheEntry(org.apache.flink.api.common.cache.DistributedCache.DistributedCacheEntry) DistributedCache(org.apache.flink.api.common.cache.DistributedCache) Preconditions(org.apache.flink.util.Preconditions) Serializable(java.io.Serializable) JobExecutionResult(org.apache.flink.api.common.JobExecutionResult) List(java.util.List) TextValueInputFormat(org.apache.flink.api.java.io.TextValueInputFormat) TypeExtractor(org.apache.flink.api.java.typeutils.TypeExtractor) ResultTypeQueryable(org.apache.flink.api.java.typeutils.ResultTypeQueryable) DataSource(org.apache.flink.api.java.operators.DataSource) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) IteratorInputFormat(org.apache.flink.api.java.io.IteratorInputFormat) FlinkException(org.apache.flink.util.FlinkException) CsvReader(org.apache.flink.api.java.io.CsvReader) PublicEvolving(org.apache.flink.annotation.PublicEvolving) PipelineExecutorServiceLoader(org.apache.flink.core.execution.PipelineExecutorServiceLoader) CompletableFuture(java.util.concurrent.CompletableFuture) RestartStrategies(org.apache.flink.api.common.restartstrategy.RestartStrategies) PipelineExecutor(org.apache.flink.core.execution.PipelineExecutor) ArrayList(java.util.ArrayList) Calendar(java.util.Calendar) ReadableConfig(org.apache.flink.configuration.ReadableConfig) JobListener(org.apache.flink.core.execution.JobListener) DeploymentOptions(org.apache.flink.configuration.DeploymentOptions) RestOptions(org.apache.flink.configuration.RestOptions) DetachedJobExecutionResult(org.apache.flink.core.execution.DetachedJobExecutionResult) Serializer(com.esotericsoftware.kryo.Serializer) Logger(org.slf4j.Logger) Iterator(java.util.Iterator) ParallelIteratorInputFormat(org.apache.flink.api.java.io.ParallelIteratorInputFormat) Configuration(org.apache.flink.configuration.Configuration) JobClient(org.apache.flink.core.execution.JobClient) Operator(org.apache.flink.api.java.operators.Operator) TextInputFormat(org.apache.flink.api.java.io.TextInputFormat) StringValue(org.apache.flink.types.StringValue) NumberSequenceIterator(org.apache.flink.util.NumberSequenceIterator) Internal(org.apache.flink.annotation.Internal) PlanGenerator(org.apache.flink.api.java.utils.PlanGenerator) PipelineExecutorFactory(org.apache.flink.core.execution.PipelineExecutorFactory) Plan(org.apache.flink.api.common.Plan) JobClient(org.apache.flink.core.execution.JobClient) PublicEvolving(org.apache.flink.annotation.PublicEvolving)

Example 2 with PipelineExecutorFactory

use of org.apache.flink.core.execution.PipelineExecutorFactory in project flink by apache.

the class StreamExecutionEnvironment method executeAsync.

/**
 * Triggers the program execution asynchronously. The environment will execute all parts of the
 * program that have resulted in a "sink" operation. Sink operations are for example printing
 * results or forwarding them to a message queue.
 *
 * @param streamGraph the stream graph representing the transformations
 * @return A {@link JobClient} that can be used to communicate with the submitted job, completed
 *     on submission succeeded.
 * @throws Exception which occurs during job execution.
 */
@Internal
public JobClient executeAsync(StreamGraph streamGraph) throws Exception {
    checkNotNull(streamGraph, "StreamGraph cannot be null.");
    checkNotNull(configuration.get(DeploymentOptions.TARGET), "No execution.target specified in your configuration file.");
    final PipelineExecutorFactory executorFactory = executorServiceLoader.getExecutorFactory(configuration);
    checkNotNull(executorFactory, "Cannot find compatible factory for specified execution.target (=%s)", configuration.get(DeploymentOptions.TARGET));
    CompletableFuture<JobClient> jobClientFuture = executorFactory.getExecutor(configuration).execute(streamGraph, configuration, userClassloader);
    try {
        JobClient jobClient = jobClientFuture.get();
        jobListeners.forEach(jobListener -> jobListener.onJobSubmitted(jobClient, null));
        return jobClient;
    } catch (ExecutionException executionException) {
        final Throwable strippedException = ExceptionUtils.stripExecutionException(executionException);
        jobListeners.forEach(jobListener -> jobListener.onJobSubmitted(null, strippedException));
        throw new FlinkException(String.format("Failed to execute job '%s'.", streamGraph.getJobName()), strippedException);
    }
}
Also used : Arrays(java.util.Arrays) SocketTextStreamFunction(org.apache.flink.streaming.api.functions.source.SocketTextStreamFunction) Tuple3(org.apache.flink.api.java.tuple.Tuple3) Tuple2(org.apache.flink.api.java.tuple.Tuple2) FromIteratorFunction(org.apache.flink.streaming.api.functions.source.FromIteratorFunction) StreamGraphGenerator(org.apache.flink.streaming.api.graph.StreamGraphGenerator) MemorySize(org.apache.flink.configuration.MemorySize) StateBackend(org.apache.flink.runtime.state.StateBackend) WrappingRuntimeException(org.apache.flink.util.WrappingRuntimeException) SplittableIterator(org.apache.flink.util.SplittableIterator) InvalidProgramException(org.apache.flink.api.common.InvalidProgramException) StateBackendLoader(org.apache.flink.runtime.state.StateBackendLoader) Path(org.apache.flink.core.fs.Path) ResourceSpec(org.apache.flink.api.common.operators.ResourceSpec) Map(java.util.Map) StatefulSequenceSource(org.apache.flink.streaming.api.functions.source.StatefulSequenceSource) StreamGraph(org.apache.flink.streaming.api.graph.StreamGraph) FileMonitoringFunction(org.apache.flink.streaming.api.functions.source.FileMonitoringFunction) DefaultExecutorServiceLoader(org.apache.flink.core.execution.DefaultExecutorServiceLoader) ContinuousFileMonitoringFunction(org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction) PipelineExecutorFactory(org.apache.flink.core.execution.PipelineExecutorFactory) FromElementsFunction(org.apache.flink.streaming.api.functions.source.FromElementsFunction) TernaryBoolean(org.apache.flink.util.TernaryBoolean) Serializable(java.io.Serializable) TypeExtractor(org.apache.flink.api.java.typeutils.TypeExtractor) ResultTypeQueryable(org.apache.flink.api.java.typeutils.ResultTypeQueryable) Source(org.apache.flink.api.connector.source.Source) Time(org.apache.flink.api.common.time.Time) Boundedness(org.apache.flink.api.connector.source.Boundedness) FlinkException(org.apache.flink.util.FlinkException) NumberSequenceSource(org.apache.flink.api.connector.source.lib.NumberSequenceSource) PipelineExecutorServiceLoader(org.apache.flink.core.execution.PipelineExecutorServiceLoader) SlotSharingGroupUtils(org.apache.flink.api.common.operators.util.SlotSharingGroupUtils) UnmodifiableConfiguration(org.apache.flink.configuration.UnmodifiableConfiguration) ArrayList(java.util.ArrayList) SourceFunction(org.apache.flink.streaming.api.functions.source.SourceFunction) ReadableConfig(org.apache.flink.configuration.ReadableConfig) JobListener(org.apache.flink.core.execution.JobListener) InvalidTypesException(org.apache.flink.api.common.functions.InvalidTypesException) RestOptions(org.apache.flink.configuration.RestOptions) ConfigOption(org.apache.flink.configuration.ConfigOption) Nullable(javax.annotation.Nullable) DetachedJobExecutionResult(org.apache.flink.core.execution.DetachedJobExecutionResult) TimeCharacteristic(org.apache.flink.streaming.api.TimeCharacteristic) Serializer(com.esotericsoftware.kryo.Serializer) SingleOutputStreamOperator(org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator) MissingTypeInfo(org.apache.flink.api.java.typeutils.MissingTypeInfo) IOException(java.io.IOException) JobClient(org.apache.flink.core.execution.JobClient) DynamicCodeLoadingException(org.apache.flink.util.DynamicCodeLoadingException) ExecutionException(java.util.concurrent.ExecutionException) TextInputFormat(org.apache.flink.api.java.io.TextInputFormat) CoreOptions(org.apache.flink.configuration.CoreOptions) Transformation(org.apache.flink.api.dag.Transformation) PojoTypeInfo(org.apache.flink.api.java.typeutils.PojoTypeInfo) FileProcessingMode(org.apache.flink.streaming.api.functions.source.FileProcessingMode) PipelineOptions(org.apache.flink.configuration.PipelineOptions) CheckpointingMode(org.apache.flink.streaming.api.CheckpointingMode) ExceptionUtils(org.apache.flink.util.ExceptionUtils) SlotSharingGroup(org.apache.flink.api.common.operators.SlotSharingGroup) FileInputFormat(org.apache.flink.api.common.io.FileInputFormat) BasicTypeInfo(org.apache.flink.api.common.typeinfo.BasicTypeInfo) InstantiationUtil(org.apache.flink.util.InstantiationUtil) InputFormat(org.apache.flink.api.common.io.InputFormat) Public(org.apache.flink.annotation.Public) URI(java.net.URI) Preconditions.checkNotNull(org.apache.flink.util.Preconditions.checkNotNull) TypeInformation(org.apache.flink.api.common.typeinfo.TypeInformation) ExecutionOptions(org.apache.flink.configuration.ExecutionOptions) TimestampedFileInputSplit(org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit) Collection(java.util.Collection) WatermarkStrategy(org.apache.flink.api.common.eventtime.WatermarkStrategy) DistributedCache(org.apache.flink.api.common.cache.DistributedCache) Preconditions(org.apache.flink.util.Preconditions) StringUtils(org.apache.flink.util.StringUtils) ResourceProfile(org.apache.flink.runtime.clusterframework.types.ResourceProfile) JobExecutionResult(org.apache.flink.api.common.JobExecutionResult) List(java.util.List) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) Experimental(org.apache.flink.annotation.Experimental) Optional(java.util.Optional) ParallelSourceFunction(org.apache.flink.streaming.api.functions.source.ParallelSourceFunction) Utils(org.apache.flink.api.java.Utils) FileReadFunction(org.apache.flink.streaming.api.functions.source.FileReadFunction) KeyGroupRangeAssignment(org.apache.flink.runtime.state.KeyGroupRangeAssignment) FilePathFilter(org.apache.flink.api.common.io.FilePathFilter) PublicEvolving(org.apache.flink.annotation.PublicEvolving) HashMap(java.util.HashMap) CompletableFuture(java.util.concurrent.CompletableFuture) RestartStrategies(org.apache.flink.api.common.restartstrategy.RestartStrategies) DataStreamSource(org.apache.flink.streaming.api.datastream.DataStreamSource) PipelineExecutor(org.apache.flink.core.execution.PipelineExecutor) DeploymentOptions(org.apache.flink.configuration.DeploymentOptions) ContinuousFileReaderOperatorFactory(org.apache.flink.streaming.api.functions.source.ContinuousFileReaderOperatorFactory) Iterator(java.util.Iterator) StateChangelogOptions(org.apache.flink.configuration.StateChangelogOptions) Configuration(org.apache.flink.configuration.Configuration) InputFormatSourceFunction(org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction) DataStream(org.apache.flink.streaming.api.datastream.DataStream) Internal(org.apache.flink.annotation.Internal) FromSplittableIteratorFunction(org.apache.flink.streaming.api.functions.source.FromSplittableIteratorFunction) ClosureCleaner(org.apache.flink.api.java.ClosureCleaner) RuntimeExecutionMode(org.apache.flink.api.common.RuntimeExecutionMode) StreamSource(org.apache.flink.streaming.api.operators.StreamSource) PipelineExecutorFactory(org.apache.flink.core.execution.PipelineExecutorFactory) ExecutionException(java.util.concurrent.ExecutionException) JobClient(org.apache.flink.core.execution.JobClient) FlinkException(org.apache.flink.util.FlinkException) Internal(org.apache.flink.annotation.Internal)

Aggregations

Serializer (com.esotericsoftware.kryo.Serializer)2 Serializable (java.io.Serializable)2 ArrayList (java.util.ArrayList)2 Arrays (java.util.Arrays)2 Collection (java.util.Collection)2 Iterator (java.util.Iterator)2 List (java.util.List)2 CompletableFuture (java.util.concurrent.CompletableFuture)2 Internal (org.apache.flink.annotation.Internal)2 Public (org.apache.flink.annotation.Public)2 PublicEvolving (org.apache.flink.annotation.PublicEvolving)2 ExecutionConfig (org.apache.flink.api.common.ExecutionConfig)2 InvalidProgramException (org.apache.flink.api.common.InvalidProgramException)2 JobExecutionResult (org.apache.flink.api.common.JobExecutionResult)2 DistributedCache (org.apache.flink.api.common.cache.DistributedCache)2 FileInputFormat (org.apache.flink.api.common.io.FileInputFormat)2 InputFormat (org.apache.flink.api.common.io.InputFormat)2 RestartStrategies (org.apache.flink.api.common.restartstrategy.RestartStrategies)2 BasicTypeInfo (org.apache.flink.api.common.typeinfo.BasicTypeInfo)2 TypeInformation (org.apache.flink.api.common.typeinfo.TypeInformation)2