Search in sources :

Example 46 with JobStatus

use of org.apache.flink.api.common.JobStatus in project flink by apache.

the class MiniDispatcher method jobReachedTerminalState.

@Override
protected CleanupJobState jobReachedTerminalState(ExecutionGraphInfo executionGraphInfo) {
    final ArchivedExecutionGraph archivedExecutionGraph = executionGraphInfo.getArchivedExecutionGraph();
    final CleanupJobState cleanupHAState = super.jobReachedTerminalState(executionGraphInfo);
    JobStatus jobStatus = Objects.requireNonNull(archivedExecutionGraph.getState(), "JobStatus should not be null here.");
    if (jobStatus.isGloballyTerminalState() && (jobCancelled || executionMode == ClusterEntrypoint.ExecutionMode.DETACHED)) {
        // shut down if job is cancelled or we don't have to wait for the execution result
        // retrieval
        log.info("Shutting down cluster with state {}, jobCancelled: {}, executionMode: {}", jobStatus, jobCancelled, executionMode);
        shutDownFuture.complete(ApplicationStatus.fromJobStatus(jobStatus));
    }
    return cleanupHAState;
}
Also used : JobStatus(org.apache.flink.api.common.JobStatus) ArchivedExecutionGraph(org.apache.flink.runtime.executiongraph.ArchivedExecutionGraph)

Example 47 with JobStatus

use of org.apache.flink.api.common.JobStatus in project flink by apache.

the class JobDetails method createDetailsForJob.

public static JobDetails createDetailsForJob(AccessExecutionGraph job) {
    JobStatus status = job.getState();
    long started = job.getStatusTimestamp(JobStatus.INITIALIZING);
    long finished = status.isGloballyTerminalState() ? job.getStatusTimestamp(status) : -1L;
    long duration = (finished >= 0L ? finished : System.currentTimeMillis()) - started;
    int[] countsPerStatus = new int[ExecutionState.values().length];
    long lastChanged = 0;
    int numTotalTasks = 0;
    for (AccessExecutionJobVertex ejv : job.getVerticesTopologically()) {
        AccessExecutionVertex[] taskVertices = ejv.getTaskVertices();
        numTotalTasks += taskVertices.length;
        for (AccessExecutionVertex taskVertex : taskVertices) {
            ExecutionState state = taskVertex.getExecutionState();
            countsPerStatus[state.ordinal()]++;
            lastChanged = Math.max(lastChanged, taskVertex.getStateTimestamp(state));
        }
    }
    lastChanged = Math.max(lastChanged, finished);
    return new JobDetails(job.getJobID(), job.getJobName(), started, finished, duration, status, lastChanged, countsPerStatus, numTotalTasks);
}
Also used : JobStatus(org.apache.flink.api.common.JobStatus) AccessExecutionJobVertex(org.apache.flink.runtime.executiongraph.AccessExecutionJobVertex) ExecutionState(org.apache.flink.runtime.execution.ExecutionState) AccessExecutionVertex(org.apache.flink.runtime.executiongraph.AccessExecutionVertex)

Example 48 with JobStatus

use of org.apache.flink.api.common.JobStatus in project flink by apache.

the class JobExecutionResultHandler method handleRequest.

@Override
protected CompletableFuture<JobExecutionResultResponseBody> handleRequest(@Nonnull final HandlerRequest<EmptyRequestBody> request, @Nonnull final RestfulGateway gateway) throws RestHandlerException {
    final JobID jobId = request.getPathParameter(JobIDPathParameter.class);
    final CompletableFuture<JobStatus> jobStatusFuture = gateway.requestJobStatus(jobId, timeout);
    return jobStatusFuture.thenCompose(jobStatus -> {
        if (jobStatus.isGloballyTerminalState()) {
            return gateway.requestJobResult(jobId, timeout).thenApply(JobExecutionResultResponseBody::created);
        } else {
            return CompletableFuture.completedFuture(JobExecutionResultResponseBody.inProgress());
        }
    }).exceptionally(throwable -> {
        throw propagateException(throwable);
    });
}
Also used : JobStatus(org.apache.flink.api.common.JobStatus) JobExecutionResultHeaders(org.apache.flink.runtime.rest.messages.job.JobExecutionResultHeaders) GatewayRetriever(org.apache.flink.runtime.webmonitor.retriever.GatewayRetriever) RestfulGateway(org.apache.flink.runtime.webmonitor.RestfulGateway) ExceptionUtils(org.apache.flink.util.ExceptionUtils) CompletableFuture(java.util.concurrent.CompletableFuture) CompletionException(java.util.concurrent.CompletionException) JobStatus(org.apache.flink.api.common.JobStatus) HttpResponseStatus(org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpResponseStatus) RestHandlerException(org.apache.flink.runtime.rest.handler.RestHandlerException) EmptyRequestBody(org.apache.flink.runtime.rest.messages.EmptyRequestBody) JobID(org.apache.flink.api.common.JobID) JobMessageParameters(org.apache.flink.runtime.rest.messages.JobMessageParameters) FlinkJobNotFoundException(org.apache.flink.runtime.messages.FlinkJobNotFoundException) Map(java.util.Map) HandlerRequest(org.apache.flink.runtime.rest.handler.HandlerRequest) Nonnull(javax.annotation.Nonnull) Time(org.apache.flink.api.common.time.Time) AbstractRestHandler(org.apache.flink.runtime.rest.handler.AbstractRestHandler) JobIDPathParameter(org.apache.flink.runtime.rest.messages.JobIDPathParameter) JobExecutionResultResponseBody(org.apache.flink.runtime.rest.messages.job.JobExecutionResultResponseBody) JobID(org.apache.flink.api.common.JobID)

Example 49 with JobStatus

use of org.apache.flink.api.common.JobStatus in project flink by apache.

the class ExecutionGraphInfoStoreTestUtils method generateTerminalExecutionGraphInfos.

/**
 * Generate a specified of ExecutionGraphInfo.
 *
 * @param number the given number
 * @return the result ExecutionGraphInfo collection
 */
static Collection<ExecutionGraphInfo> generateTerminalExecutionGraphInfos(int number) {
    final Collection<ExecutionGraphInfo> executionGraphInfos = new ArrayList<>(number);
    for (int i = 0; i < number; i++) {
        final JobStatus state = GLOBALLY_TERMINAL_JOB_STATUS.get(ThreadLocalRandom.current().nextInt(GLOBALLY_TERMINAL_JOB_STATUS.size()));
        executionGraphInfos.add(new ExecutionGraphInfo(new ArchivedExecutionGraphBuilder().setState(state).build()));
    }
    return executionGraphInfos;
}
Also used : JobStatus(org.apache.flink.api.common.JobStatus) ExecutionGraphInfo(org.apache.flink.runtime.scheduler.ExecutionGraphInfo) ArrayList(java.util.ArrayList) ArchivedExecutionGraphBuilder(org.apache.flink.runtime.rest.handler.legacy.utils.ArchivedExecutionGraphBuilder)

Example 50 with JobStatus

use of org.apache.flink.api.common.JobStatus in project flink by apache.

the class MemoryExecutionGraphInfoStoreTest method testStoredJobsOverview.

/**
 * Tests that we obtain the correct jobs overview.
 */
@Test
public void testStoredJobsOverview() throws IOException {
    final int numberExecutionGraphs = 10;
    final Collection<ExecutionGraphInfo> executionGraphInfos = generateTerminalExecutionGraphInfos(numberExecutionGraphs);
    final List<JobStatus> jobStatuses = executionGraphInfos.stream().map(ExecutionGraphInfo::getArchivedExecutionGraph).map(ArchivedExecutionGraph::getState).collect(Collectors.toList());
    final JobsOverview expectedJobsOverview = JobsOverview.create(jobStatuses);
    try (final MemoryExecutionGraphInfoStore executionGraphInfoStore = createMemoryExecutionGraphInfoStore()) {
        for (ExecutionGraphInfo executionGraphInfo : executionGraphInfos) {
            executionGraphInfoStore.put(executionGraphInfo);
        }
        assertThat(executionGraphInfoStore.getStoredJobsOverview(), Matchers.equalTo(expectedJobsOverview));
    }
}
Also used : JobStatus(org.apache.flink.api.common.JobStatus) ExecutionGraphInfo(org.apache.flink.runtime.scheduler.ExecutionGraphInfo) JobsOverview(org.apache.flink.runtime.messages.webmonitor.JobsOverview) Test(org.junit.Test)

Aggregations

JobStatus (org.apache.flink.api.common.JobStatus)62 Test (org.junit.Test)28 JobID (org.apache.flink.api.common.JobID)19 CompletableFuture (java.util.concurrent.CompletableFuture)15 JobGraph (org.apache.flink.runtime.jobgraph.JobGraph)14 FlinkException (org.apache.flink.util.FlinkException)8 ExecutionException (java.util.concurrent.ExecutionException)7 IOException (java.io.IOException)6 ArrayList (java.util.ArrayList)6 Time (org.apache.flink.api.common.time.Time)6 ExecutionGraphInfo (org.apache.flink.runtime.scheduler.ExecutionGraphInfo)6 TaskExecutionState (org.apache.flink.runtime.taskmanager.TaskExecutionState)6 Collections (java.util.Collections)5 HashMap (java.util.HashMap)5 ExecutionState (org.apache.flink.runtime.execution.ExecutionState)5 FutureUtils (org.apache.flink.util.concurrent.FutureUtils)5 TimeUnit (java.util.concurrent.TimeUnit)4 Configuration (org.apache.flink.configuration.Configuration)4 ExecutionAttemptID (org.apache.flink.runtime.executiongraph.ExecutionAttemptID)4 Acknowledge (org.apache.flink.runtime.messages.Acknowledge)4