Search in sources :

Example 6 with AvroGeneratedUser

use of org.apache.beam.sdk.io.AvroGeneratedUser in project beam by apache.

the class AvroUtilsTest method testAvroSchemaCoders.

@Test
public void testAvroSchemaCoders() {
    Pipeline pipeline = Pipeline.create();
    org.apache.avro.Schema schema = org.apache.avro.Schema.createRecord("TestSubRecord", "TestSubRecord doc", "org.apache.beam.sdk.schemas.utils", false, getAvroSubSchemaFields());
    GenericRecord record = new GenericRecordBuilder(getAvroSubSchema("simple")).set("bool", true).set("int", 42).build();
    PCollection<GenericRecord> records = pipeline.apply(Create.of(record).withCoder(AvroCoder.of(schema)));
    assertFalse(records.hasSchema());
    records.setCoder(AvroUtils.schemaCoder(schema));
    assertTrue(records.hasSchema());
    CoderProperties.coderSerializable(records.getCoder());
    AvroGeneratedUser user = new AvroGeneratedUser("foo", 42, "green");
    PCollection<AvroGeneratedUser> users = pipeline.apply(Create.of(user).withCoder(AvroCoder.of(AvroGeneratedUser.class)));
    assertFalse(users.hasSchema());
    users.setCoder(AvroUtils.schemaCoder((AvroCoder<AvroGeneratedUser>) users.getCoder()));
    assertTrue(users.hasSchema());
    CoderProperties.coderSerializable(users.getCoder());
}
Also used : AvroGeneratedUser(org.apache.beam.sdk.io.AvroGeneratedUser) GenericRecordBuilder(org.apache.avro.generic.GenericRecordBuilder) GenericRecord(org.apache.avro.generic.GenericRecord) AvroCoder(org.apache.beam.sdk.coders.AvroCoder) Pipeline(org.apache.beam.sdk.Pipeline) Test(org.junit.Test)

Aggregations

AvroGeneratedUser (org.apache.beam.sdk.io.AvroGeneratedUser)6 Test (org.junit.Test)5 GenericRecord (org.apache.avro.generic.GenericRecord)3 ArrayList (java.util.ArrayList)2 KV (org.apache.beam.sdk.values.KV)2 Matchers.containsString (org.hamcrest.Matchers.containsString)2 GenericRecordBuilder (org.apache.avro.generic.GenericRecordBuilder)1 Pipeline (org.apache.beam.sdk.Pipeline)1 AvroCoder (org.apache.beam.sdk.coders.AvroCoder)1 FakeSnowflakeBasicDataSource (org.apache.beam.sdk.io.snowflake.test.FakeSnowflakeBasicDataSource)1 FakeSnowflakeBatchServiceImpl (org.apache.beam.sdk.io.snowflake.test.FakeSnowflakeBatchServiceImpl)1 IntegerDeserializer (org.apache.kafka.common.serialization.IntegerDeserializer)1 BeforeClass (org.junit.BeforeClass)1