Search in sources :

Example 56 with BasicBSONObject

use of org.bson.BasicBSONObject in project mongo-hadoop by mongodb.

the class BSONSerDeTest method testDouble.

@Test
public void testDouble() throws SerDeException {
    String columnNames = "doub";
    String columnTypes = "double";
    Double value = 1.1D;
    BSONSerDe serde = new BSONSerDe();
    Object result = helpDeserialize(serde, columnNames, columnTypes, value);
    assertThat(value, equalTo(result));
    ObjectInspector innerInspector = PrimitiveObjectInspectorFactory.getPrimitiveObjectInspectorFromClass(Double.class);
    BasicBSONObject bObject = new BasicBSONObject();
    Object serialized = helpSerialize(columnNames, innerInspector, bObject, value, serde);
    assertThat(new BSONWritable(bObject), equalTo(serialized));
}
Also used : BSONWritable(com.mongodb.hadoop.io.BSONWritable) BasicBSONObject(org.bson.BasicBSONObject) ListObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector) MapObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.MapObjectInspector) StructObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector) ObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector) BasicBSONObject(org.bson.BasicBSONObject) Test(org.junit.Test)

Example 57 with BasicBSONObject

use of org.bson.BasicBSONObject in project mongo-hadoop by mongodb.

the class BSONSerDeTest method testNumericCasts.

@Test
public void testNumericCasts() throws SerDeException {
    BSONSerDe serde = new BSONSerDe();
    String colName = "cast";
    Number[] nums = { 42.0D, 42, (short) 42, 42.0f, 42L };
    Class[] numericClasses = { Double.class, Integer.class, Short.class, Float.class, Long.class };
    for (Number num : nums) {
        // Double
        Object result = helpDeserialize(serde, colName, "double", num);
        assertThat(num.doubleValue(), equalTo(result));
        // Int
        result = helpDeserialize(serde, colName, "int", num);
        assertThat(num.intValue(), equalTo(result));
        // Short
        result = helpDeserialize(serde, colName, "smallint", num);
        assertThat(num.shortValue(), equalTo(result));
        // Float
        result = helpDeserialize(serde, colName, "float", num);
        assertThat(num.floatValue(), equalTo(result));
        // Long
        result = helpDeserialize(serde, colName, "bigint", num);
        assertThat(num.longValue(), equalTo(result));
        for (Class klass : numericClasses) {
            ObjectInspector oi = PrimitiveObjectInspectorFactory.getPrimitiveObjectInspectorFromClass(klass);
            BasicBSONObject obj = new BasicBSONObject();
            Object serialized = helpSerialize(colName, oi, obj, num, serde);
            assertThat(new BSONWritable(obj), equalTo(serialized));
        }
    }
}
Also used : BSONWritable(com.mongodb.hadoop.io.BSONWritable) BasicBSONObject(org.bson.BasicBSONObject) ListObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector) MapObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.MapObjectInspector) StructObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector) ObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector) BasicBSONObject(org.bson.BasicBSONObject) Test(org.junit.Test)

Example 58 with BasicBSONObject

use of org.bson.BasicBSONObject in project mongo-hadoop by mongodb.

the class MongoUpdateOutputReaderTest method testUpdate.

@Test
public void testUpdate() throws IOException {
    BasicBSONObject query = new BasicDBObject("i", 42);
    BasicBSONObject modifiers = new BasicDBObject("$set", new BasicDBObject("a", "b"));
    DBObject update = new BasicDBObjectBuilder().add("_id", query).add("modifiers", modifiers).push("options").add("multi", true).add("upsert", false).pop().get();
    MongoUpdateWritable muw = new MongoUpdateWritable(query, modifiers, false, true, false);
    PipeMapRed pipeMapRed = mock(PipeMapRed.class);
    when(pipeMapRed.getClientInput()).thenReturn(inputFromBSONObject(update));
    MongoUpdateOutputReader reader = new MongoUpdateOutputReader();
    reader.initialize(pipeMapRed);
    assertTrue(reader.readKeyValue());
    assertEquals(muw, reader.getCurrentValue());
}
Also used : BasicBSONObject(org.bson.BasicBSONObject) BasicDBObject(com.mongodb.BasicDBObject) BasicDBObjectBuilder(com.mongodb.BasicDBObjectBuilder) MongoUpdateWritable(com.mongodb.hadoop.io.MongoUpdateWritable) BasicDBObject(com.mongodb.BasicDBObject) DBObject(com.mongodb.DBObject) PipeMapRed(org.apache.hadoop.streaming.PipeMapRed) Test(org.junit.Test)

Example 59 with BasicBSONObject

use of org.bson.BasicBSONObject in project mongo-hadoop by mongodb.

the class MongoRecordReader method nextKeyValue.

public boolean nextKeyValue() throws IOException {
    try {
        if (!cursor.hasNext()) {
            LOG.info("Read " + seen + " documents from:");
            LOG.info(split.toString());
            return false;
        }
        DBObject next = cursor.next();
        this.currentVal.setDoc(next);
        this.currentKey.setDoc(new BasicBSONObject("_id", next.get("_id")));
        seen++;
        return true;
    } catch (MongoException e) {
        throw new IOException("Couldn't get next key/value from mongodb: ", e);
    }
}
Also used : BasicBSONObject(org.bson.BasicBSONObject) MongoException(com.mongodb.MongoException) IOException(java.io.IOException) DBObject(com.mongodb.DBObject)

Example 60 with BasicBSONObject

use of org.bson.BasicBSONObject in project mongo-hadoop by mongodb.

the class BSONWritable method toBSON.

/**
     * Unwrap a (usually Writable) Object, getting back a value suitable for
     * putting into a BSONObject. If the given object is not Writable, then
     * simply return the Object back.
     *
     * @param x the Object to turn into BSON.
     * @return the BSON representation of the Object.
     */
@SuppressWarnings("unchecked")
public static Object toBSON(final Object x) {
    if (x == null) {
        return null;
    }
    if (x instanceof Text) {
        return x.toString();
    }
    if (x instanceof BSONWritable) {
        return ((BSONWritable) x).getDoc();
    }
    if (x instanceof Writable) {
        if (x instanceof AbstractMapWritable) {
            if (!(x instanceof Map)) {
                throw new IllegalArgumentException(String.format("Cannot turn %s into BSON, since it does " + "not implement java.util.Map.", x.getClass().getName()));
            }
            Map<Writable, Writable> map = (Map<Writable, Writable>) x;
            BasicBSONObject bson = new BasicBSONObject();
            for (Map.Entry<Writable, Writable> entry : map.entrySet()) {
                bson.put(entry.getKey().toString(), toBSON(entry.getValue()));
            }
            return bson;
        }
        if (x instanceof ArrayWritable) {
            Writable[] o = ((ArrayWritable) x).get();
            Object[] a = new Object[o.length];
            for (int i = 0; i < o.length; i++) {
                a[i] = toBSON(o[i]);
            }
            return a;
        }
        if (x instanceof NullWritable) {
            return null;
        }
        if (x instanceof BooleanWritable) {
            return ((BooleanWritable) x).get();
        }
        if (x instanceof BytesWritable) {
            return ((BytesWritable) x).getBytes();
        }
        if (x instanceof ByteWritable) {
            return ((ByteWritable) x).get();
        }
        if (x instanceof DoubleWritable) {
            return ((DoubleWritable) x).get();
        }
        if (x instanceof FloatWritable) {
            return ((FloatWritable) x).get();
        }
        if (x instanceof LongWritable) {
            return ((LongWritable) x).get();
        }
        if (x instanceof IntWritable) {
            return ((IntWritable) x).get();
        }
    // TODO - Support counters
    }
    return x;
}
Also used : NullWritable(org.apache.hadoop.io.NullWritable) Writable(org.apache.hadoop.io.Writable) DoubleWritable(org.apache.hadoop.io.DoubleWritable) LongWritable(org.apache.hadoop.io.LongWritable) ByteWritable(org.apache.hadoop.io.ByteWritable) BytesWritable(org.apache.hadoop.io.BytesWritable) ArrayWritable(org.apache.hadoop.io.ArrayWritable) IntWritable(org.apache.hadoop.io.IntWritable) AbstractMapWritable(org.apache.hadoop.io.AbstractMapWritable) BooleanWritable(org.apache.hadoop.io.BooleanWritable) FloatWritable(org.apache.hadoop.io.FloatWritable) Text(org.apache.hadoop.io.Text) BytesWritable(org.apache.hadoop.io.BytesWritable) DoubleWritable(org.apache.hadoop.io.DoubleWritable) NullWritable(org.apache.hadoop.io.NullWritable) BasicBSONObject(org.bson.BasicBSONObject) FloatWritable(org.apache.hadoop.io.FloatWritable) ArrayWritable(org.apache.hadoop.io.ArrayWritable) BooleanWritable(org.apache.hadoop.io.BooleanWritable) AbstractMapWritable(org.apache.hadoop.io.AbstractMapWritable) BasicBSONObject(org.bson.BasicBSONObject) BasicDBObject(com.mongodb.BasicDBObject) BSONObject(org.bson.BSONObject) LongWritable(org.apache.hadoop.io.LongWritable) Map(java.util.Map) ByteWritable(org.apache.hadoop.io.ByteWritable) IntWritable(org.apache.hadoop.io.IntWritable)

Aggregations

BasicBSONObject (org.bson.BasicBSONObject)71 Test (org.junit.Test)23 BSONObject (org.bson.BSONObject)20 StructObjectInspector (org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector)14 BSONWritable (com.mongodb.hadoop.io.BSONWritable)13 ArrayList (java.util.ArrayList)13 ListObjectInspector (org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector)13 MapObjectInspector (org.apache.hadoop.hive.serde2.objectinspector.MapObjectInspector)13 ObjectInspector (org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector)13 BasicDBObject (com.mongodb.BasicDBObject)11 ObjectId (org.bson.types.ObjectId)11 IOException (java.io.IOException)7 DataBag (org.apache.pig.data.DataBag)6 Tuple (org.apache.pig.data.Tuple)5 Mongo (com.mongodb.Mongo)4 Date (java.util.Date)4 Map (java.util.Map)4 DoubleWritable (org.apache.hadoop.io.DoubleWritable)4 Text (org.apache.hadoop.io.Text)4 SQLExpr (com.alibaba.druid.sql.ast.SQLExpr)3