Search in sources :

Example 1 with ArrayWritable

use of org.apache.hadoop.io.ArrayWritable in project hadoop by apache.

the class TypedBytesWritableInput method readArray.

public ArrayWritable readArray(ArrayWritable aw) throws IOException {
    if (aw == null) {
        aw = new ArrayWritable(TypedBytesWritable.class);
    } else if (!aw.getValueClass().equals(TypedBytesWritable.class)) {
        throw new RuntimeException("value class has to be TypedBytesWritable");
    }
    int length = in.readVectorHeader();
    Writable[] writables = new Writable[length];
    for (int i = 0; i < length; i++) {
        writables[i] = new TypedBytesWritable(in.readRaw());
    }
    aw.set(writables);
    return aw;
}
Also used : ArrayWritable(org.apache.hadoop.io.ArrayWritable) SortedMapWritable(org.apache.hadoop.io.SortedMapWritable) VLongWritable(org.apache.hadoop.io.VLongWritable) Writable(org.apache.hadoop.io.Writable) MapWritable(org.apache.hadoop.io.MapWritable) DoubleWritable(org.apache.hadoop.io.DoubleWritable) LongWritable(org.apache.hadoop.io.LongWritable) BooleanWritable(org.apache.hadoop.io.BooleanWritable) ByteWritable(org.apache.hadoop.io.ByteWritable) BytesWritable(org.apache.hadoop.io.BytesWritable) VIntWritable(org.apache.hadoop.io.VIntWritable) ArrayWritable(org.apache.hadoop.io.ArrayWritable) FloatWritable(org.apache.hadoop.io.FloatWritable) IntWritable(org.apache.hadoop.io.IntWritable)

Example 2 with ArrayWritable

use of org.apache.hadoop.io.ArrayWritable in project hive by apache.

the class TestAbstractParquetMapInspector method testNullContainer.

@Test
public void testNullContainer() {
    final ArrayWritable map = new ArrayWritable(ArrayWritable.class, null);
    assertEquals("Wrong size", -1, inspector.getMapSize(map));
    assertNull("Should be null", inspector.getMap(map));
}
Also used : ArrayWritable(org.apache.hadoop.io.ArrayWritable) Test(org.junit.Test)

Example 3 with ArrayWritable

use of org.apache.hadoop.io.ArrayWritable in project hive by apache.

the class TestAbstractParquetMapInspector method testEmptyContainer.

@Test
public void testEmptyContainer() {
    final ArrayWritable map = new ArrayWritable(ArrayWritable.class, new ArrayWritable[0]);
    assertEquals("Wrong size", 0, inspector.getMapSize(map));
    assertNotNull("Should not be null", inspector.getMap(map));
}
Also used : ArrayWritable(org.apache.hadoop.io.ArrayWritable) Test(org.junit.Test)

Example 4 with ArrayWritable

use of org.apache.hadoop.io.ArrayWritable in project hive by apache.

the class TestDeepParquetHiveMapInspector method testRegularMap.

@Test
public void testRegularMap() {
    final Writable[] entry1 = new Writable[] { new IntWritable(0), new IntWritable(1) };
    final Writable[] entry2 = new Writable[] { new IntWritable(2), new IntWritable(3) };
    final ArrayWritable map = new ArrayWritable(ArrayWritable.class, new Writable[] { new ArrayWritable(Writable.class, entry1), new ArrayWritable(Writable.class, entry2) });
    assertEquals("Wrong result of inspection", new IntWritable(1), inspector.getMapValueElement(map, new IntWritable(0)));
    assertEquals("Wrong result of inspection", new IntWritable(3), inspector.getMapValueElement(map, new IntWritable(2)));
    assertEquals("Wrong result of inspection", new IntWritable(1), inspector.getMapValueElement(map, new ShortWritable((short) 0)));
    assertEquals("Wrong result of inspection", new IntWritable(3), inspector.getMapValueElement(map, new ShortWritable((short) 2)));
}
Also used : ArrayWritable(org.apache.hadoop.io.ArrayWritable) Writable(org.apache.hadoop.io.Writable) ShortWritable(org.apache.hadoop.hive.serde2.io.ShortWritable) ArrayWritable(org.apache.hadoop.io.ArrayWritable) IntWritable(org.apache.hadoop.io.IntWritable) ShortWritable(org.apache.hadoop.hive.serde2.io.ShortWritable) IntWritable(org.apache.hadoop.io.IntWritable) Test(org.junit.Test)

Example 5 with ArrayWritable

use of org.apache.hadoop.io.ArrayWritable in project hive by apache.

the class TestParquetHiveArrayInspector method testNullContainer.

@Test
public void testNullContainer() {
    final ArrayWritable list = new ArrayWritable(ArrayWritable.class, null);
    assertEquals("Wrong size", -1, inspector.getListLength(list));
    assertNull("Should be null", inspector.getList(list));
    assertNull("Should be null", inspector.getListElement(list, 0));
}
Also used : ArrayWritable(org.apache.hadoop.io.ArrayWritable) Test(org.junit.Test)

Aggregations

ArrayWritable (org.apache.hadoop.io.ArrayWritable)72 Test (org.junit.Test)41 IntWritable (org.apache.hadoop.io.IntWritable)31 Writable (org.apache.hadoop.io.Writable)29 Path (org.apache.hadoop.fs.Path)18 DoubleWritable (org.apache.hadoop.hive.serde2.io.DoubleWritable)18 LongWritable (org.apache.hadoop.io.LongWritable)18 RecordConsumer (org.apache.parquet.io.api.RecordConsumer)18 ShortWritable (org.apache.hadoop.hive.serde2.io.ShortWritable)15 ArrayList (java.util.ArrayList)13 BytesWritable (org.apache.hadoop.io.BytesWritable)10 List (java.util.List)9 BooleanWritable (org.apache.hadoop.io.BooleanWritable)8 FloatWritable (org.apache.hadoop.io.FloatWritable)8 StructObjectInspector (org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector)6 NullWritable (org.apache.hadoop.io.NullWritable)6 Text (org.apache.hadoop.io.Text)6 ByteWritable (org.apache.hadoop.hive.serde2.io.ByteWritable)5 TimestampWritable (org.apache.hadoop.hive.serde2.io.TimestampWritable)5 PrimitiveObjectInspector (org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector)5