Search in sources :

Example 1 with ClusterClientProvider

use of org.apache.flink.client.program.ClusterClientProvider in project flink by apache.

the class AbstractSessionClusterExecutor method execute.

@Override
public CompletableFuture<JobClient> execute(@Nonnull final Pipeline pipeline, @Nonnull final Configuration configuration, @Nonnull final ClassLoader userCodeClassloader) throws Exception {
    final JobGraph jobGraph = PipelineExecutorUtils.getJobGraph(pipeline, configuration);
    try (final ClusterDescriptor<ClusterID> clusterDescriptor = clusterClientFactory.createClusterDescriptor(configuration)) {
        final ClusterID clusterID = clusterClientFactory.getClusterId(configuration);
        checkState(clusterID != null);
        final ClusterClientProvider<ClusterID> clusterClientProvider = clusterDescriptor.retrieve(clusterID);
        ClusterClient<ClusterID> clusterClient = clusterClientProvider.getClusterClient();
        return clusterClient.submitJob(jobGraph).thenApplyAsync(FunctionUtils.uncheckedFunction(jobId -> {
            ClientUtils.waitUntilJobInitializationFinished(() -> clusterClient.getJobStatus(jobId).get(), () -> clusterClient.requestJobResult(jobId).get(), userCodeClassloader);
            return jobId;
        })).thenApplyAsync(jobID -> (JobClient) new ClusterClientJobClientAdapter<>(clusterClientProvider, jobID, userCodeClassloader)).whenCompleteAsync((ignored1, ignored2) -> clusterClient.close());
    }
}
Also used : Preconditions.checkState(org.apache.flink.util.Preconditions.checkState) ClientUtils(org.apache.flink.client.ClientUtils) ClusterClientFactory(org.apache.flink.client.deployment.ClusterClientFactory) Pipeline(org.apache.flink.api.dag.Pipeline) Configuration(org.apache.flink.configuration.Configuration) JobGraph(org.apache.flink.runtime.jobgraph.JobGraph) CompletableFuture(java.util.concurrent.CompletableFuture) PipelineExecutor(org.apache.flink.core.execution.PipelineExecutor) JobClient(org.apache.flink.core.execution.JobClient) ClusterClientJobClientAdapter(org.apache.flink.client.deployment.ClusterClientJobClientAdapter) ClusterDescriptor(org.apache.flink.client.deployment.ClusterDescriptor) ClusterClient(org.apache.flink.client.program.ClusterClient) FunctionUtils(org.apache.flink.util.function.FunctionUtils) Internal(org.apache.flink.annotation.Internal) ClusterClientProvider(org.apache.flink.client.program.ClusterClientProvider) Preconditions.checkNotNull(org.apache.flink.util.Preconditions.checkNotNull) Nonnull(javax.annotation.Nonnull) JobGraph(org.apache.flink.runtime.jobgraph.JobGraph) JobClient(org.apache.flink.core.execution.JobClient)

Example 2 with ClusterClientProvider

use of org.apache.flink.client.program.ClusterClientProvider in project flink by apache.

the class KubernetesClusterDescriptor method deployClusterInternal.

private ClusterClientProvider<String> deployClusterInternal(String entryPoint, ClusterSpecification clusterSpecification, boolean detached) throws ClusterDeploymentException {
    final ClusterEntrypoint.ExecutionMode executionMode = detached ? ClusterEntrypoint.ExecutionMode.DETACHED : ClusterEntrypoint.ExecutionMode.NORMAL;
    flinkConfig.setString(ClusterEntrypoint.INTERNAL_CLUSTER_EXECUTION_MODE, executionMode.toString());
    flinkConfig.setString(KubernetesConfigOptionsInternal.ENTRY_POINT_CLASS, entryPoint);
    // Rpc, blob, rest, taskManagerRpc ports need to be exposed, so update them to fixed values.
    KubernetesUtils.checkAndUpdatePortConfigOption(flinkConfig, BlobServerOptions.PORT, Constants.BLOB_SERVER_PORT);
    KubernetesUtils.checkAndUpdatePortConfigOption(flinkConfig, TaskManagerOptions.RPC_PORT, Constants.TASK_MANAGER_RPC_PORT);
    KubernetesUtils.checkAndUpdatePortConfigOption(flinkConfig, RestOptions.BIND_PORT, Constants.REST_PORT);
    if (HighAvailabilityMode.isHighAvailabilityModeActivated(flinkConfig)) {
        flinkConfig.setString(HighAvailabilityOptions.HA_CLUSTER_ID, clusterId);
        KubernetesUtils.checkAndUpdatePortConfigOption(flinkConfig, HighAvailabilityOptions.HA_JOB_MANAGER_PORT_RANGE, flinkConfig.get(JobManagerOptions.PORT));
    }
    try {
        final KubernetesJobManagerParameters kubernetesJobManagerParameters = new KubernetesJobManagerParameters(flinkConfig, clusterSpecification);
        final FlinkPod podTemplate = kubernetesJobManagerParameters.getPodTemplateFilePath().map(file -> KubernetesUtils.loadPodFromTemplateFile(client, file, Constants.MAIN_CONTAINER_NAME)).orElse(new FlinkPod.Builder().build());
        final KubernetesJobManagerSpecification kubernetesJobManagerSpec = KubernetesJobManagerFactory.buildKubernetesJobManagerSpecification(podTemplate, kubernetesJobManagerParameters);
        client.createJobManagerComponent(kubernetesJobManagerSpec);
        return createClusterClientProvider(clusterId);
    } catch (Exception e) {
        try {
            LOG.warn("Failed to create the Kubernetes cluster \"{}\", try to clean up the residual resources.", clusterId);
            client.stopAndCleanupCluster(clusterId);
        } catch (Exception e1) {
            LOG.info("Failed to stop and clean up the Kubernetes cluster \"{}\".", clusterId, e1);
        }
        throw new ClusterDeploymentException("Could not create Kubernetes cluster \"" + clusterId + "\".", e);
    }
}
Also used : FlinkException(org.apache.flink.util.FlinkException) ClusterSpecification(org.apache.flink.client.deployment.ClusterSpecification) JobGraph(org.apache.flink.runtime.jobgraph.JobGraph) LoggerFactory(org.slf4j.LoggerFactory) KubernetesConfigOptionsInternal(org.apache.flink.kubernetes.configuration.KubernetesConfigOptionsInternal) KubernetesService(org.apache.flink.kubernetes.kubeclient.resources.KubernetesService) RestClusterClient(org.apache.flink.client.program.rest.RestClusterClient) ApplicationConfiguration(org.apache.flink.client.deployment.application.ApplicationConfiguration) TaskManagerOptions(org.apache.flink.configuration.TaskManagerOptions) RestOptions(org.apache.flink.configuration.RestOptions) Endpoint(org.apache.flink.kubernetes.kubeclient.Endpoint) StandaloneClientHAServices(org.apache.flink.runtime.highavailability.nonha.standalone.StandaloneClientHAServices) Preconditions.checkNotNull(org.apache.flink.util.Preconditions.checkNotNull) ClusterRetrieveException(org.apache.flink.client.deployment.ClusterRetrieveException) KubernetesJobManagerParameters(org.apache.flink.kubernetes.kubeclient.parameters.KubernetesJobManagerParameters) HighAvailabilityMode(org.apache.flink.runtime.jobmanager.HighAvailabilityMode) KubernetesDeploymentTarget(org.apache.flink.kubernetes.configuration.KubernetesDeploymentTarget) PackagedProgramUtils(org.apache.flink.client.program.PackagedProgramUtils) BlobServerOptions(org.apache.flink.configuration.BlobServerOptions) KubernetesUtils(org.apache.flink.kubernetes.utils.KubernetesUtils) KubernetesConfigOptions(org.apache.flink.kubernetes.configuration.KubernetesConfigOptions) Logger(org.slf4j.Logger) Configuration(org.apache.flink.configuration.Configuration) KubernetesJobManagerSpecification(org.apache.flink.kubernetes.kubeclient.KubernetesJobManagerSpecification) KubernetesApplicationClusterEntrypoint(org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint) HighAvailabilityServicesUtils(org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils) JobManagerOptions(org.apache.flink.configuration.JobManagerOptions) FlinkPod(org.apache.flink.kubernetes.kubeclient.FlinkPod) Preconditions(org.apache.flink.util.Preconditions) KubernetesSessionClusterEntrypoint(org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint) File(java.io.File) ClusterDeploymentException(org.apache.flink.client.deployment.ClusterDeploymentException) List(java.util.List) ClusterDescriptor(org.apache.flink.client.deployment.ClusterDescriptor) ClusterClient(org.apache.flink.client.program.ClusterClient) KubernetesJobManagerFactory(org.apache.flink.kubernetes.kubeclient.factory.KubernetesJobManagerFactory) Optional(java.util.Optional) AddressResolution(org.apache.flink.runtime.rpc.AddressResolution) ClusterClientProvider(org.apache.flink.client.program.ClusterClientProvider) ClusterEntrypoint(org.apache.flink.runtime.entrypoint.ClusterEntrypoint) Constants(org.apache.flink.kubernetes.utils.Constants) HighAvailabilityOptions(org.apache.flink.configuration.HighAvailabilityOptions) FlinkKubeClient(org.apache.flink.kubernetes.kubeclient.FlinkKubeClient) ClusterDeploymentException(org.apache.flink.client.deployment.ClusterDeploymentException) FlinkPod(org.apache.flink.kubernetes.kubeclient.FlinkPod) KubernetesApplicationClusterEntrypoint(org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint) KubernetesSessionClusterEntrypoint(org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint) ClusterEntrypoint(org.apache.flink.runtime.entrypoint.ClusterEntrypoint) KubernetesJobManagerSpecification(org.apache.flink.kubernetes.kubeclient.KubernetesJobManagerSpecification) FlinkException(org.apache.flink.util.FlinkException) ClusterRetrieveException(org.apache.flink.client.deployment.ClusterRetrieveException) ClusterDeploymentException(org.apache.flink.client.deployment.ClusterDeploymentException) KubernetesJobManagerParameters(org.apache.flink.kubernetes.kubeclient.parameters.KubernetesJobManagerParameters)

Aggregations

ClusterDescriptor (org.apache.flink.client.deployment.ClusterDescriptor)2 ClusterClient (org.apache.flink.client.program.ClusterClient)2 ClusterClientProvider (org.apache.flink.client.program.ClusterClientProvider)2 Configuration (org.apache.flink.configuration.Configuration)2 JobGraph (org.apache.flink.runtime.jobgraph.JobGraph)2 File (java.io.File)1 List (java.util.List)1 Optional (java.util.Optional)1 CompletableFuture (java.util.concurrent.CompletableFuture)1 Nonnull (javax.annotation.Nonnull)1 Internal (org.apache.flink.annotation.Internal)1 Pipeline (org.apache.flink.api.dag.Pipeline)1 ClientUtils (org.apache.flink.client.ClientUtils)1 ClusterClientFactory (org.apache.flink.client.deployment.ClusterClientFactory)1 ClusterClientJobClientAdapter (org.apache.flink.client.deployment.ClusterClientJobClientAdapter)1 ClusterDeploymentException (org.apache.flink.client.deployment.ClusterDeploymentException)1 ClusterRetrieveException (org.apache.flink.client.deployment.ClusterRetrieveException)1 ClusterSpecification (org.apache.flink.client.deployment.ClusterSpecification)1 ApplicationConfiguration (org.apache.flink.client.deployment.application.ApplicationConfiguration)1 PackagedProgramUtils (org.apache.flink.client.program.PackagedProgramUtils)1