Examples with OutputCommitter org.apache.hadoop.mapreduce.OutputCommitter used on opensource projects

Example 46 with OutputCommitter

use of org.apache.hadoop.mapreduce.OutputCommitter in project cdap by caskdata.

the class MultipleOutputsCommitter method setupJob.

@Override
public void setupJob(JobContext jobContext) throws IOException {
    for (Map.Entry<String, OutputCommitter> committer : committers.entrySet()) {
        JobContext namedJobContext = MultipleOutputs.getNamedJobContext(jobContext, committer.getKey());
        committer.getValue().setupJob(namedJobContext);
    }
}

Example 47 with OutputCommitter

use of org.apache.hadoop.mapreduce.OutputCommitter in project cdap by caskdata.

the class MultipleOutputsMainOutputWrapper method getOutputCommitter.

@Override
public synchronized OutputCommitter getOutputCommitter(TaskAttemptContext context) throws IOException, InterruptedException {
    // return a MultipleOutputsCommitter that commits for the root output format as well as all delegate outputformats
    if (committer == null) {
        // use a linked hash map: it preserves the order of insertion, so the output committers are called in the
        // same order as outputs were added. This makes multi-output a little more predictable (and testable).
        Map<String, OutputCommitter> committers = new LinkedHashMap<>();
        for (String name : MultipleOutputs.getNamedOutputsList(context)) {
            Class<? extends OutputFormat> namedOutputFormatClass = MultipleOutputs.getNamedOutputFormatClass(context, name);
            TaskAttemptContext namedContext = MultipleOutputs.getNamedTaskContext(context, name);
            OutputFormat<K, V> outputFormat = ReflectionUtils.newInstance(namedOutputFormatClass, namedContext.getConfiguration());
            committers.put(name, outputFormat.getOutputCommitter(namedContext));
        }
        committer = new MultipleOutputsCommitter(committers);
    }
    return committer;
}