Search in sources :

Example 1 with TaskContext

use of org.apache.hadoop.mapred.nativetask.TaskContext in project hadoop by apache.

the class NativeCollectorOnlyHandler method create.

public static <K, V> NativeCollectorOnlyHandler<K, V> create(TaskContext context) throws IOException {
    ICombineHandler combinerHandler = null;
    try {
        final TaskContext combineContext = context.copyOf();
        combineContext.setInputKeyClass(context.getOutputKeyClass());
        combineContext.setInputValueClass(context.getOutputValueClass());
        combinerHandler = CombinerHandler.create(combineContext);
    } catch (final ClassNotFoundException e) {
        throw new IOException(e);
    }
    if (null != combinerHandler) {
        LOG.info("[NativeCollectorOnlyHandler] combiner is not null");
    }
    final INativeHandler nativeHandler = NativeBatchProcessor.create(NAME, context.getConf(), DataChannel.OUT);
    final BufferPusher<K, V> kvPusher = new BufferPusher<K, V>((Class<K>) context.getOutputKeyClass(), (Class<V>) context.getOutputValueClass(), nativeHandler);
    return new NativeCollectorOnlyHandler<K, V>(context, nativeHandler, kvPusher, combinerHandler);
}
Also used : TaskContext(org.apache.hadoop.mapred.nativetask.TaskContext) ICombineHandler(org.apache.hadoop.mapred.nativetask.ICombineHandler) INativeHandler(org.apache.hadoop.mapred.nativetask.INativeHandler) IOException(java.io.IOException)

Example 2 with TaskContext

use of org.apache.hadoop.mapred.nativetask.TaskContext in project hadoop by apache.

the class TestNativeCollectorOnlyHandler method setUp.

@Before
public void setUp() throws IOException {
    this.nativeHandler = Mockito.mock(INativeHandler.class);
    this.pusher = Mockito.mock(BufferPusher.class);
    this.combiner = Mockito.mock(ICombineHandler.class);
    JobConf jobConf = new JobConf();
    jobConf.set(OutputUtil.NATIVE_TASK_OUTPUT_MANAGER, "org.apache.hadoop.mapred.nativetask.util.LocalJobOutputFiles");
    jobConf.set("mapred.local.dir", LOCAL_DIR);
    this.taskContext = new TaskContext(jobConf, BytesWritable.class, BytesWritable.class, BytesWritable.class, BytesWritable.class, null, null);
    Mockito.when(nativeHandler.getInputBuffer()).thenReturn(new InputBuffer(BufferType.HEAP_BUFFER, 100));
}
Also used : TaskContext(org.apache.hadoop.mapred.nativetask.TaskContext) InputBuffer(org.apache.hadoop.mapred.nativetask.buffer.InputBuffer) INativeHandler(org.apache.hadoop.mapred.nativetask.INativeHandler) ICombineHandler(org.apache.hadoop.mapred.nativetask.ICombineHandler) BytesWritable(org.apache.hadoop.io.BytesWritable) JobConf(org.apache.hadoop.mapred.JobConf) Before(org.junit.Before)

Aggregations

ICombineHandler (org.apache.hadoop.mapred.nativetask.ICombineHandler)2 INativeHandler (org.apache.hadoop.mapred.nativetask.INativeHandler)2 TaskContext (org.apache.hadoop.mapred.nativetask.TaskContext)2 IOException (java.io.IOException)1 BytesWritable (org.apache.hadoop.io.BytesWritable)1 JobConf (org.apache.hadoop.mapred.JobConf)1 InputBuffer (org.apache.hadoop.mapred.nativetask.buffer.InputBuffer)1 Before (org.junit.Before)1