Search in sources :

Example 1 with DatasetWriter

use of org.kitesdk.data.DatasetWriter in project nifi by apache.

the class StoreInKiteDataset method onTrigger.

@Override
public void onTrigger(ProcessContext context, final ProcessSession session) throws ProcessException {
    FlowFile flowFile = session.get();
    if (flowFile == null) {
        return;
    }
    final View<Record> target = load(context, flowFile);
    final Schema schema = target.getDataset().getDescriptor().getSchema();
    try {
        StopWatch timer = new StopWatch(true);
        session.read(flowFile, new InputStreamCallback() {

            @Override
            public void process(InputStream in) throws IOException {
                try (DataFileStream<Record> stream = new DataFileStream<>(in, AvroUtil.newDatumReader(schema, Record.class))) {
                    IncompatibleSchemaException.check(SchemaValidationUtil.canRead(stream.getSchema(), schema), "Incompatible file schema %s, expected %s", stream.getSchema(), schema);
                    long written = 0L;
                    try (DatasetWriter<Record> writer = target.newWriter()) {
                        for (Record record : stream) {
                            writer.write(record);
                            written += 1;
                        }
                    } finally {
                        session.adjustCounter("Stored records", written, true);
                    }
                }
            }
        });
        timer.stop();
        session.getProvenanceReporter().send(flowFile, target.getUri().toString(), timer.getDuration(TimeUnit.MILLISECONDS), true);
        session.transfer(flowFile, SUCCESS);
    } catch (ProcessException | DatasetIOException e) {
        getLogger().error("Failed to read FlowFile", e);
        session.transfer(flowFile, FAILURE);
    } catch (ValidationException e) {
        getLogger().error(e.getMessage());
        getLogger().debug("Incompatible schema error", e);
        session.transfer(flowFile, INCOMPATIBLE);
    }
}
Also used : FlowFile(org.apache.nifi.flowfile.FlowFile) ValidationException(org.kitesdk.data.ValidationException) InputStream(java.io.InputStream) Schema(org.apache.avro.Schema) DatasetIOException(org.kitesdk.data.DatasetIOException) IOException(java.io.IOException) DataFileStream(org.apache.avro.file.DataFileStream) DatasetWriter(org.kitesdk.data.DatasetWriter) StopWatch(org.apache.nifi.util.StopWatch) ProcessException(org.apache.nifi.processor.exception.ProcessException) InputStreamCallback(org.apache.nifi.processor.io.InputStreamCallback) Record(org.apache.avro.generic.GenericData.Record) DatasetIOException(org.kitesdk.data.DatasetIOException)

Aggregations

IOException (java.io.IOException)1 InputStream (java.io.InputStream)1 Schema (org.apache.avro.Schema)1 DataFileStream (org.apache.avro.file.DataFileStream)1 Record (org.apache.avro.generic.GenericData.Record)1 FlowFile (org.apache.nifi.flowfile.FlowFile)1 ProcessException (org.apache.nifi.processor.exception.ProcessException)1 InputStreamCallback (org.apache.nifi.processor.io.InputStreamCallback)1 StopWatch (org.apache.nifi.util.StopWatch)1 DatasetIOException (org.kitesdk.data.DatasetIOException)1 DatasetWriter (org.kitesdk.data.DatasetWriter)1 ValidationException (org.kitesdk.data.ValidationException)1