Search in sources :

Example 66 with BytesWritable

use of org.apache.hadoop.io.BytesWritable in project hadoop by apache.

the class TestTupleWritable method makeRandomWritables.

private Writable[] makeRandomWritables() {
    Random r = new Random();
    Writable[] writs = { new BooleanWritable(r.nextBoolean()), new FloatWritable(r.nextFloat()), new FloatWritable(r.nextFloat()), new IntWritable(r.nextInt()), new LongWritable(r.nextLong()), new BytesWritable("dingo".getBytes()), new LongWritable(r.nextLong()), new IntWritable(r.nextInt()), new BytesWritable("yak".getBytes()), new IntWritable(r.nextInt()) };
    return writs;
}
Also used : FloatWritable(org.apache.hadoop.io.FloatWritable) Random(java.util.Random) BooleanWritable(org.apache.hadoop.io.BooleanWritable) Writable(org.apache.hadoop.io.Writable) LongWritable(org.apache.hadoop.io.LongWritable) BooleanWritable(org.apache.hadoop.io.BooleanWritable) BytesWritable(org.apache.hadoop.io.BytesWritable) FloatWritable(org.apache.hadoop.io.FloatWritable) IntWritable(org.apache.hadoop.io.IntWritable) BytesWritable(org.apache.hadoop.io.BytesWritable) LongWritable(org.apache.hadoop.io.LongWritable) IntWritable(org.apache.hadoop.io.IntWritable)

Example 67 with BytesWritable

use of org.apache.hadoop.io.BytesWritable in project hadoop by apache.

the class TestTupleWritable method testNestedIterable.

@Test
public void testNestedIterable() throws Exception {
    Random r = new Random();
    Writable[] writs = { new BooleanWritable(r.nextBoolean()), new FloatWritable(r.nextFloat()), new FloatWritable(r.nextFloat()), new IntWritable(r.nextInt()), new LongWritable(r.nextLong()), new BytesWritable("dingo".getBytes()), new LongWritable(r.nextLong()), new IntWritable(r.nextInt()), new BytesWritable("yak".getBytes()), new IntWritable(r.nextInt()) };
    TupleWritable sTuple = makeTuple(writs);
    assertTrue("Bad count", writs.length == verifIter(writs, sTuple, 0));
}
Also used : FloatWritable(org.apache.hadoop.io.FloatWritable) Random(java.util.Random) BooleanWritable(org.apache.hadoop.io.BooleanWritable) Writable(org.apache.hadoop.io.Writable) LongWritable(org.apache.hadoop.io.LongWritable) BooleanWritable(org.apache.hadoop.io.BooleanWritable) BytesWritable(org.apache.hadoop.io.BytesWritable) FloatWritable(org.apache.hadoop.io.FloatWritable) IntWritable(org.apache.hadoop.io.IntWritable) BytesWritable(org.apache.hadoop.io.BytesWritable) LongWritable(org.apache.hadoop.io.LongWritable) IntWritable(org.apache.hadoop.io.IntWritable) Test(org.junit.Test)

Example 68 with BytesWritable

use of org.apache.hadoop.io.BytesWritable in project hadoop by apache.

the class RawBytesInputWriter method writeRawBytes.

private void writeRawBytes(Writable writable) throws IOException {
    if (writable instanceof BytesWritable) {
        BytesWritable bw = (BytesWritable) writable;
        byte[] bytes = bw.getBytes();
        int length = bw.getLength();
        clientOut.writeInt(length);
        clientOut.write(bytes, 0, length);
    } else {
        bufferOut.reset();
        writable.write(bufferDataOut);
        byte[] bytes = bufferOut.toByteArray();
        clientOut.writeInt(bytes.length);
        clientOut.write(bytes);
    }
}
Also used : BytesWritable(org.apache.hadoop.io.BytesWritable)

Example 69 with BytesWritable

use of org.apache.hadoop.io.BytesWritable in project hadoop by apache.

the class RawBytesOutputReader method initialize.

@Override
public void initialize(PipeMapRed pipeMapRed) throws IOException {
    super.initialize(pipeMapRed);
    clientIn = pipeMapRed.getClientInput();
    key = new BytesWritable();
    value = new BytesWritable();
}
Also used : BytesWritable(org.apache.hadoop.io.BytesWritable)

Example 70 with BytesWritable

use of org.apache.hadoop.io.BytesWritable in project hadoop by apache.

the class TextInputWriter method writeUTF8.

// Write an object to the output stream using UTF-8 encoding
protected void writeUTF8(Object object) throws IOException {
    byte[] bval;
    int valSize;
    if (object instanceof BytesWritable) {
        BytesWritable val = (BytesWritable) object;
        bval = val.getBytes();
        valSize = val.getLength();
    } else if (object instanceof Text) {
        Text val = (Text) object;
        bval = val.getBytes();
        valSize = val.getLength();
    } else {
        String sval = object.toString();
        bval = sval.getBytes("UTF-8");
        valSize = bval.length;
    }
    clientOut.write(bval, 0, valSize);
}
Also used : BytesWritable(org.apache.hadoop.io.BytesWritable) Text(org.apache.hadoop.io.Text)

Aggregations

BytesWritable (org.apache.hadoop.io.BytesWritable)275 Text (org.apache.hadoop.io.Text)73 LongWritable (org.apache.hadoop.io.LongWritable)59 Test (org.junit.Test)53 ObjectInspector (org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector)46 IntWritable (org.apache.hadoop.io.IntWritable)44 ArrayList (java.util.ArrayList)39 Path (org.apache.hadoop.fs.Path)38 IOException (java.io.IOException)36 Configuration (org.apache.hadoop.conf.Configuration)33 FloatWritable (org.apache.hadoop.io.FloatWritable)33 Writable (org.apache.hadoop.io.Writable)32 BooleanWritable (org.apache.hadoop.io.BooleanWritable)31 List (java.util.List)30 SequenceFile (org.apache.hadoop.io.SequenceFile)27 Random (java.util.Random)24 DoubleWritable (org.apache.hadoop.hive.serde2.io.DoubleWritable)24 ShortWritable (org.apache.hadoop.hive.serde2.io.ShortWritable)23 ByteWritable (org.apache.hadoop.hive.serde2.io.ByteWritable)22 FileSystem (org.apache.hadoop.fs.FileSystem)21