Examples with OutputCommitter org.apache.hadoop.mapred.OutputCommitter used on opensource projects

Example 1 with OutputCommitter

use of org.apache.hadoop.mapred.OutputCommitter in project ignite by apache.

the class HadoopV1OutputCollector method commit.

/**
 * Commit task.
 *
 * @throws IOException In failed.
 */
public void commit() throws IOException {
    if (writer != null) {
        OutputCommitter outputCommitter = jobConf.getOutputCommitter();
        TaskAttemptContext taskCtx = new TaskAttemptContextImpl(jobConf, attempt);
        if (outputCommitter.needsTaskCommit(taskCtx))
            outputCommitter.commitTask(taskCtx);
    }
}

Example 2 with OutputCommitter

use of org.apache.hadoop.mapred.OutputCommitter in project ignite by apache.

the class HadoopV1SetupTask method run.

/**
 * {@inheritDoc}
 */
@Override
public void run(HadoopTaskContext taskCtx) throws IgniteCheckedException {
    HadoopV2TaskContext ctx = (HadoopV2TaskContext) taskCtx;
    try {
        ctx.jobConf().getOutputFormat().checkOutputSpecs(null, ctx.jobConf());
        OutputCommitter committer = ctx.jobConf().getOutputCommitter();
        if (committer != null)
            committer.setupJob(ctx.jobContext());
    } catch (IOException e) {
        throw new IgniteCheckedException(e);
    }
}

Example 3 with OutputCommitter

use of org.apache.hadoop.mapred.OutputCommitter in project ignite by apache.

the class HadoopV1CleanupTask method run.

/**
 * {@inheritDoc}
 */
@Override
public void run(HadoopTaskContext taskCtx) throws IgniteCheckedException {
    HadoopV2TaskContext ctx = (HadoopV2TaskContext) taskCtx;
    JobContext jobCtx = ctx.jobContext();
    try {
        OutputCommitter committer = jobCtx.getJobConf().getOutputCommitter();
        if (abort)
            committer.abortJob(jobCtx, JobStatus.State.FAILED);
        else
            committer.commitJob(jobCtx);
    } catch (IOException e) {
        throw new IgniteCheckedException(e);
    }
}

Example 4 with OutputCommitter

use of org.apache.hadoop.mapred.OutputCommitter in project flink by apache.

the class HadoopOutputFormatBase method finalizeGlobal.

@Override
public void finalizeGlobal(int parallelism) throws IOException {
    try {
        JobContext jobContext = new JobContextImpl(this.jobConf, new JobID());
        OutputCommitter outputCommitter = this.jobConf.getOutputCommitter();
        // finalize HDFS output format
        outputCommitter.commitJob(jobContext);
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}