Examples with SpecificDatumWriter org.apache.avro.specific.SpecificDatumWriter used on opensource projects

Example 46 with SpecificDatumWriter

use of org.apache.avro.specific.SpecificDatumWriter in project gobblin by apache.

the class FsSpecProducer method writeAvroJobSpec.

private void writeAvroJobSpec(AvroJobSpec jobSpec) throws IOException {
    DatumWriter<AvroJobSpec> datumWriter = new SpecificDatumWriter<>(AvroJobSpec.SCHEMA$);
    DataFileWriter<AvroJobSpec> dataFileWriter = new DataFileWriter<>(datumWriter);
    Path jobSpecPath = new Path(this.specConsumerPath, annotateSpecFileName(jobSpec.getUri()));
    // Write the new JobSpec to a temporary path first.
    Path tmpDir = new Path(this.specConsumerPath, UUID.randomUUID().toString());
    if (!fs.exists(tmpDir)) {
        fs.mkdirs(tmpDir);
    }
    Path tmpJobSpecPath = new Path(tmpDir, jobSpec.getUri());
    OutputStream out = fs.create(tmpJobSpecPath);
    dataFileWriter.create(AvroJobSpec.SCHEMA$, out);
    dataFileWriter.append(jobSpec);
    dataFileWriter.close();
    // Rename the JobSpec from temporary to final location.
    HadoopUtils.renamePath(fs, tmpJobSpecPath, jobSpecPath, true);
    // Delete the temporary path once the jobspec has been moved to its final publish location.
    log.info("Deleting {}", tmpJobSpecPath.getParent().toString());
    fs.delete(tmpJobSpecPath.getParent(), true);
}

Example 47 with SpecificDatumWriter

use of org.apache.avro.specific.SpecificDatumWriter in project spring-batch by spring-projects.

the class AvroTestUtils method createTestDataWithNoEmbeddedSchema.

static void createTestDataWithNoEmbeddedSchema() throws Exception {
    DatumWriter<User> userDatumWriter = new SpecificDatumWriter<>(User.class);
    FileOutputStream fileOutputStream = new FileOutputStream("user-data-no-schema.avro");
    Encoder encoder = EncoderFactory.get().binaryEncoder(fileOutputStream, null);
    userDatumWriter.write(new User("David", 20, "blue"), encoder);
    userDatumWriter.write(new User("Sue", 4, "red"), encoder);
    userDatumWriter.write(new User("Alana", 13, "yellow"), encoder);
    userDatumWriter.write(new User("Joe", 1, "pink"), encoder);
    encoder.flush();
    fileOutputStream.flush();
    fileOutputStream.close();
}

Example 48 with SpecificDatumWriter

use of org.apache.avro.specific.SpecificDatumWriter in project kloadgen by corunet.

the class GenericAvroRecordBinarySerializer method serialize.

@Override
public byte[] serialize(String s, T data) {
    DatumWriter<T> writer = new SpecificDatumWriter<>(data.getSchema());
    try (final ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
        Encoder encoder = EncoderFactory.get().binaryEncoder(baos, null);
        writer.write(data, encoder);
        encoder.flush();
        return baos.toByteArray();
    } catch (IOException e) {
        log.error("Serialization error for date: {}", data, e);
        return new byte[] {};
    }
}

Example 49 with SpecificDatumWriter

use of org.apache.avro.specific.SpecificDatumWriter in project kloadgen by corunet.

the class GenericAvroRecordSerializer method serialize.

@Override
public byte[] serialize(String topic, T record) {
    DatumWriter<T> writer = new SpecificDatumWriter<>(record.getSchema());
    byte[] data = new byte[0];
    ByteArrayOutputStream stream = new ByteArrayOutputStream();
    Encoder jsonEncoder;
    try {
        jsonEncoder = EncoderFactory.get().jsonEncoder(record.getSchema(), stream);
        writer.write(record, jsonEncoder);
        jsonEncoder.flush();
        data = stream.toByteArray();
    } catch (IOException e) {
        log.error("Serialization error:" + e.getMessage());
    }
    return data;
}

Example 50 with SpecificDatumWriter

use of org.apache.avro.specific.SpecificDatumWriter in project java-common-libs by opencb.

the class AvroDataWriter method open.

@Override
public boolean open() {
    /**
     * Open output stream *
     */
    try {
        DatumWriter<T> datumWriter = new SpecificDatumWriter<>();
        avroWriter = new DataFileWriter<>(datumWriter);
        avroWriter.setCodec(gzip ? CodecFactory.deflateCodec(CodecFactory.DEFAULT_DEFLATE_LEVEL) : CodecFactory.nullCodec());
        avroWriter.create(schema, outputPath.toFile());
    } catch (IOException e) {
        throw new UncheckedIOException(e);
    }
    return true;
}