Search in sources :

Example 1 with BufferPusher

use of org.apache.hadoop.mapred.nativetask.handlers.BufferPusher in project hadoop by apache.

the class TestBufferPushPull method testPush.

@Test
public void testPush() throws Exception {
    final byte[] buff = new byte[BUFFER_LENGTH];
    final InputBuffer input = new InputBuffer(buff);
    final OutputBuffer out = new OutputBuffer(buff);
    final Class<BytesWritable> iKClass = BytesWritable.class;
    final Class<BytesWritable> iVClass = BytesWritable.class;
    final RecordWriterForPush writer = new RecordWriterForPush() {

        @Override
        public void write(BytesWritable key, BytesWritable value) throws IOException {
            final KV expect = dataInput[count++];
            Assert.assertEquals(expect.key.toString(), key.toString());
            Assert.assertEquals(expect.value.toString(), value.toString());
        }
    };
    final BufferPushee pushee = new BufferPushee(iKClass, iVClass, writer);
    final PushTarget handler = new PushTarget(out) {

        @Override
        public void sendData() throws IOException {
            final int outputLength = out.length();
            input.rewind(0, outputLength);
            out.rewind();
            pushee.collect(input);
        }
    };
    final BufferPusher pusher = new BufferPusher(iKClass, iVClass, handler);
    writer.reset();
    for (int i = 0; i < INPUT_KV_COUNT; i++) {
        pusher.collect(dataInput[i].key, dataInput[i].value);
    }
    pusher.close();
    pushee.close();
}
Also used : BufferPusher(org.apache.hadoop.mapred.nativetask.handlers.BufferPusher) DataInputBuffer(org.apache.hadoop.io.DataInputBuffer) BytesWritable(org.apache.hadoop.io.BytesWritable) KV(org.apache.hadoop.mapred.nativetask.testutil.TestInput.KV) BufferPushee(org.apache.hadoop.mapred.nativetask.handlers.BufferPushee) Test(org.junit.Test)

Aggregations

BytesWritable (org.apache.hadoop.io.BytesWritable)1 DataInputBuffer (org.apache.hadoop.io.DataInputBuffer)1 BufferPushee (org.apache.hadoop.mapred.nativetask.handlers.BufferPushee)1 BufferPusher (org.apache.hadoop.mapred.nativetask.handlers.BufferPusher)1 KV (org.apache.hadoop.mapred.nativetask.testutil.TestInput.KV)1 Test (org.junit.Test)1