Search in sources :

Example 1 with CopyableGenericRecord

use of org.apache.gobblin.fork.CopyableGenericRecord in project incubator-gobblin by apache.

the class TestConverter2 method convertRecord.

@Override
public Iterable<CopyableGenericRecord> convertRecord(CopyableSchema schema, String inputRecord, WorkUnitState workUnit) throws DataConversionException {
    JsonElement element = GSON.fromJson(inputRecord, JsonElement.class);
    Map<String, Object> fields = GSON.fromJson(element, FIELD_ENTRY_TYPE);
    try {
        Schema avroSchema = schema.copy();
        GenericRecord record = new GenericData.Record(avroSchema);
        for (Map.Entry<String, Object> entry : fields.entrySet()) {
            if (entry.getValue() instanceof Double) {
                // Gson reads the integers in the input Json documents as doubles, so we have
                // to convert doubles to integers here as the Avro schema specifies integers.
                record.put(entry.getKey(), ((Double) entry.getValue()).intValue());
            } else {
                record.put(entry.getKey(), entry.getValue());
            }
        }
        return new SingleRecordIterable<CopyableGenericRecord>(new CopyableGenericRecord(record));
    } catch (CopyNotSupportedException cnse) {
        throw new DataConversionException(cnse);
    }
}
Also used : CopyableSchema(org.apache.gobblin.fork.CopyableSchema) Schema(org.apache.avro.Schema) SingleRecordIterable(org.apache.gobblin.converter.SingleRecordIterable) CopyableGenericRecord(org.apache.gobblin.fork.CopyableGenericRecord) JsonElement(com.google.gson.JsonElement) GenericRecord(org.apache.avro.generic.GenericRecord) CopyableGenericRecord(org.apache.gobblin.fork.CopyableGenericRecord) GenericRecord(org.apache.avro.generic.GenericRecord) CopyableGenericRecord(org.apache.gobblin.fork.CopyableGenericRecord) DataConversionException(org.apache.gobblin.converter.DataConversionException) Map(java.util.Map) CopyNotSupportedException(org.apache.gobblin.fork.CopyNotSupportedException)

Aggregations

JsonElement (com.google.gson.JsonElement)1 Map (java.util.Map)1 Schema (org.apache.avro.Schema)1 GenericRecord (org.apache.avro.generic.GenericRecord)1 DataConversionException (org.apache.gobblin.converter.DataConversionException)1 SingleRecordIterable (org.apache.gobblin.converter.SingleRecordIterable)1 CopyNotSupportedException (org.apache.gobblin.fork.CopyNotSupportedException)1 CopyableGenericRecord (org.apache.gobblin.fork.CopyableGenericRecord)1 CopyableSchema (org.apache.gobblin.fork.CopyableSchema)1