Examples with SingleRecordIterable org.apache.gobblin.converter.SingleRecordIterable used on opensource projects

Example 6 with SingleRecordIterable

use of org.apache.gobblin.converter.SingleRecordIterable in project incubator-gobblin by apache.

the class AvroToRestJsonEntryConverter method convertRecord.

/**
 * Use resource key(Optional) and rest json entry as a template and fill in template using Avro as a reference.
 * e.g:
 *  Rest JSON entry HOCON template:
 *    AccountId=${sf_account_id},Member_Id__c=${member_id}
 *  Avro:
 *    {"sf_account_id":{"string":"0016000000UiCYHAA3"},"member_id":{"long":296458833}}
 *
 *  Converted Json:
 *    {"AccountId":"0016000000UiCYHAA3","Member_Id__c":296458833}
 *
 *  As it's template based approach, it can produce nested JSON structure even Avro is flat (or vice versa).
 *
 * e.g:
 *  Rest resource template:
 *    /sobject/account/memberId/${member_id}
 *  Avro:
 *    {"sf_account_id":{"string":"0016000000UiCYHAA3"},"member_id":{"long":296458833}}
 *  Converted resource:
 *    /sobject/account/memberId/296458833
 *
 *  Converted resource will be used to form end point.
 *    http://www.server.com:9090/sobject/account/memberId/296458833
 *
 * {@inheritDoc}
 * @see org.apache.gobblin.converter.Converter#convertRecord(java.lang.Object, java.lang.Object, org.apache.gobblin.configuration.WorkUnitState)
 */
@Override
public Iterable<RestEntry<JsonObject>> convertRecord(Void outputSchema, GenericRecord inputRecord, WorkUnitState workUnit) throws DataConversionException {
    Config srcConfig = ConfigFactory.parseString(inputRecord.toString(), ConfigParseOptions.defaults().setSyntax(ConfigSyntax.JSON));
    String resourceKey = workUnit.getProp(CONVERTER_AVRO_REST_ENTRY_RESOURCE_KEY, "");
    if (!StringUtils.isEmpty(resourceKey)) {
        final String dummyKey = "DUMMY";
        Config tmpConfig = ConfigFactory.parseString(dummyKey + "=" + resourceKey).resolveWith(srcConfig);
        resourceKey = tmpConfig.getString(dummyKey);
    }
    String hoconInput = workUnit.getProp(CONVERTER_AVRO_REST_JSON_ENTRY_TEMPLATE);
    if (StringUtils.isEmpty(hoconInput)) {
        return new SingleRecordIterable<>(new RestEntry<>(resourceKey, parser.parse(inputRecord.toString()).getAsJsonObject()));
    }
    Config destConfig = ConfigFactory.parseString(hoconInput).resolveWith(srcConfig);
    JsonObject json = parser.parse(destConfig.root().render(ConfigRenderOptions.concise())).getAsJsonObject();
    return new SingleRecordIterable<>(new RestEntry<>(resourceKey, json));
}

Example 7 with SingleRecordIterable

use of org.apache.gobblin.converter.SingleRecordIterable in project incubator-gobblin by apache.

the class BytesToJsonConverter method convertRecord.

@Override
public Iterable<JsonObject> convertRecord(String outputSchema, byte[] inputRecord, WorkUnitState workUnit) throws DataConversionException {
    if (inputRecord == null) {
        throw new DataConversionException("Input record is null");
    }
    String jsonString = new String(inputRecord, Charsets.UTF_8);
    JsonParser parser = new JsonParser();
    JsonObject outputRecord = parser.parse(jsonString).getAsJsonObject();
    return new SingleRecordIterable<>(outputRecord);
}

Example 8 with SingleRecordIterable

use of org.apache.gobblin.converter.SingleRecordIterable in project incubator-gobblin by apache.

the class JsonStringToJsonIntermediateConverter method convertRecord.

/**
 * Takes in a record with format String and Uses the inputSchema to convert the record to a JsonObject
 * @return a JsonObject representing the record
 * @throws IOException
 */
@Override
public Iterable<JsonObject> convertRecord(JsonArray outputSchema, String strInputRecord, WorkUnitState workUnit) throws DataConversionException {
    JsonParser jsonParser = new JsonParser();
    JsonObject inputRecord = (JsonObject) jsonParser.parse(strInputRecord);
    if (!this.unpackComplexSchemas) {
        return new SingleRecordIterable<>(inputRecord);
    }
    JsonSchema schema = new JsonSchema(outputSchema);
    JsonObject rec = parse(inputRecord, schema);
    return new SingleRecordIterable(rec);
}

Example 9 with SingleRecordIterable

use of org.apache.gobblin.converter.SingleRecordIterable in project incubator-gobblin by apache.

the class GrokToJsonConverter method convertRecord.

/**
 * Converts Text (String) to JSON based on a Grok regexp expression.
 * By default, fields between Text and JSON are mapped by Grok SEMANTIC which is the identifier you give to the piece of text being matched in your Grok expression.
 *
 * e.g:
 * {@inheritDoc}
 * @see Converter#convertRecord(Object, Object, WorkUnitState)
 */
@Override
public Iterable<JsonObject> convertRecord(JsonArray outputSchema, String inputRecord, WorkUnitState workUnit) throws DataConversionException {
    JsonObject outputRecord = createOutput(outputSchema, inputRecord);
    LOG.debug("Converted into " + outputRecord);
    return new SingleRecordIterable<JsonObject>(outputRecord);
}

Example 10 with SingleRecordIterable

use of org.apache.gobblin.converter.SingleRecordIterable in project incubator-gobblin by apache.

the class AnyToCouchbaseJsonConverter method convertRecord.

@Override
public Iterable<RawJsonDocument> convertRecord(String outputSchema, Object inputRecord, WorkUnitState workUnit) throws DataConversionException {
    JsonElement jsonElement = GSON.toJsonTree(inputRecord);
    if (!jsonElement.isJsonObject()) {
        throw new DataConversionException("Expecting json element " + jsonElement.toString() + " to be of type JsonObject.");
    }
    JsonObject jsonObject = jsonElement.getAsJsonObject();
    if (!jsonObject.has(keyField)) {
        throw new DataConversionException("Could not find key field " + keyField + " in json object " + jsonObject.toString());
    }
    JsonElement keyValueElement = jsonObject.get(keyField);
    String keyString;
    try {
        keyString = keyValueElement.getAsString();
    } catch (Exception e) {
        throw new DataConversionException("Could not get the key " + keyValueElement.toString() + " as a string", e);
    }
    String valueString = GSON.toJson(jsonElement);
    RawJsonDocument jsonDocument = RawJsonDocument.create(keyString, valueString);
    return new SingleRecordIterable<>(jsonDocument);
}