Examples with Job com.google.api.services.dataflow.model.Job used on opensource projects

Example 61 with Job

use of com.google.api.services.dataflow.model.Job in project beam by apache.

the class DataflowPipelineTranslator method translate.

/**
 * Translates a {@link Pipeline} into a {@code JobSpecification}.
 */
public JobSpecification translate(Pipeline pipeline, RunnerApi.Pipeline pipelineProto, SdkComponents sdkComponents, DataflowRunner runner, List<DataflowPackage> packages) {
    Translator translator = new Translator(pipeline, runner, sdkComponents);
    Job result = translator.translate(packages);
    return new JobSpecification(result, pipelineProto, Collections.unmodifiableMap(translator.stepNames));
}

Example 62 with Job

use of com.google.api.services.dataflow.model.Job in project beam by apache.

the class DataflowPipelineJob method cancel.

@Override
public State cancel() throws IOException {
    // Enforce that a cancel() call on the job is done at most once - as
    // a workaround for Dataflow service's current bugs with multiple
    // cancellation, where it may sometimes return an error when cancelling
    // a job that was already cancelled, but still report the job state as
    // RUNNING.
    // To partially work around these issues, we absorb duplicate cancel()
    // calls. This, of course, doesn't address the case when the job terminates
    // externally almost concurrently to calling cancel(), but at least it
    // makes it possible to safely call cancel() multiple times and from
    // multiple threads in one program.
    FutureTask<State> tentativeCancelTask = new FutureTask<>(() -> {
        Job content = new Job();
        content.setProjectId(getProjectId());
        String currentJobId = getJobId();
        content.setId(currentJobId);
        content.setRequestedState("JOB_STATE_CANCELLED");
        try {
            Job job = dataflowClient.updateJob(currentJobId, content);
            return MonitoringUtil.toState(job.getCurrentState());
        } catch (IOException e) {
            State state = getState();
            if (state.isTerminal()) {
                LOG.warn("Cancel failed because job is already terminated. State is {}", state);
                return state;
            } else if (e.getMessage().contains("has terminated")) {
                // This handles the case where the getState() call above returns RUNNING but the
                // cancel was rejected because the job is in fact done. Hopefully, someday we can
                // delete this code if there is better consistency between the State and whether
                // Cancel succeeds.
                // 
                // Example message:
                // Workflow modification failed. Causes: (7603adc9e9bff51e): Cannot perform
                // operation 'cancel' on Job: 2017-04-01_22_50_59-9269855660514862348. Job has
                // terminated in state SUCCESS: Workflow job:
                // 2017-04-01_22_50_59-9269855660514862348 succeeded.
                LOG.warn("Cancel failed because job is already terminated.", e);
                return state;
            } else {
                String errorMsg = String.format("Failed to cancel job in state %s, " + "please go to the Developers Console to cancel it manually: %s", state, MonitoringUtil.getJobMonitoringPageURL(getProjectId(), getRegion(), getJobId()));
                LOG.warn(errorMsg);
                throw new IOException(errorMsg, e);
            }
        }
    });
    if (cancelState.compareAndSet(null, tentativeCancelTask)) {
        // This thread should perform cancellation, while others will
        // only wait for the result.
        cancelState.get().run();
    }
    try {
        return cancelState.get().get();
    } catch (InterruptedException | ExecutionException e) {
        throw new IOException(e);
    }
}

Example 63 with Job

use of com.google.api.services.dataflow.model.Job in project beam by apache.

the class DataflowPipelineJob method getStateWithRetries.

State getStateWithRetries(BackOff attempts, Sleeper sleeper) throws IOException {
    if (terminalState != null) {
        return terminalState;
    }
    Job job = getJobWithRetries(attempts, sleeper);
    latestStateString = job.getCurrentState();
    return MonitoringUtil.toState(latestStateString);
}