Search in sources :

Example 1 with Address

use of org.apache.flink.formats.avro.generated.Address in project flink by apache.

the class AvroTypesITCase method testAvroObjectAccess.

@Test
public void testAvroObjectAccess() {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
    DataStream<User> ds = testData(env);
    Table t = tEnv.fromDataStream(ds, selectFields(ds));
    Table result = t.filter($("type_nested").isNotNull()).select($("type_nested").flatten()).as("city", "num", "state", "street", "zip");
    List<Address> results = CollectionUtil.iteratorToList(DataStreamUtils.collect(tEnv.toAppendStream(result, Address.class)));
    String expected = USER_1.getTypeNested().toString();
    TestBaseUtils.compareResultAsText(results, expected);
}
Also used : User(org.apache.flink.formats.avro.generated.User) Table(org.apache.flink.table.api.Table) Address(org.apache.flink.formats.avro.generated.Address) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) StreamTableEnvironment(org.apache.flink.table.api.bridge.java.StreamTableEnvironment) Test(org.junit.Test)

Example 2 with Address

use of org.apache.flink.formats.avro.generated.Address in project flink by apache.

the class EncoderDecoderTest method testGeneratedObjectWithNullableFields.

@Test
public void testGeneratedObjectWithNullableFields() {
    List<CharSequence> strings = Arrays.asList(new CharSequence[] { "These", "strings", "should", "be", "recognizable", "as", "a", "meaningful", "sequence" });
    List<Boolean> bools = Arrays.asList(true, true, false, false, true, false, true, true);
    Map<CharSequence, Long> map = new HashMap<>();
    map.put("1", 1L);
    map.put("2", 2L);
    map.put("3", 3L);
    byte[] b = new byte[16];
    new Random().nextBytes(b);
    Fixed16 f = new Fixed16(b);
    Address addr = new Address(239, "6th Main", "Bangalore", "Karnataka", "560075");
    User user = new User("Freudenreich", 1337, "macintosh gray", 1234567890L, 3.1415926, null, true, strings, bools, null, Colors.GREEN, map, f, Boolean.TRUE, addr, ByteBuffer.wrap(b), LocalDate.parse("2014-03-01"), LocalTime.parse("12:12:12"), LocalTime.ofSecondOfDay(0).plus(123456L, ChronoUnit.MICROS), Instant.parse("2014-03-01T12:12:12.321Z"), Instant.ofEpochSecond(0).plus(123456L, ChronoUnit.MICROS), ByteBuffer.wrap(// 20.00
    BigDecimal.valueOf(2000, 2).unscaledValue().toByteArray()), new Fixed2(BigDecimal.valueOf(2000, 2).unscaledValue().toByteArray()));
    testObjectSerialization(user);
}
Also used : User(org.apache.flink.formats.avro.generated.User) Address(org.apache.flink.formats.avro.generated.Address) HashMap(java.util.HashMap) Fixed2(org.apache.flink.formats.avro.generated.Fixed2) Fixed16(org.apache.flink.formats.avro.generated.Fixed16) Random(java.util.Random) Test(org.junit.Test)

Example 3 with Address

use of org.apache.flink.formats.avro.generated.Address in project flink by apache.

the class AvroStreamingFileSinkITCase method testWriteAvroSpecific.

@Test
public void testWriteAvroSpecific() throws Exception {
    File folder = TEMPORARY_FOLDER.newFolder();
    List<Address> data = Arrays.asList(new Address(1, "a", "b", "c", "12345"), new Address(2, "p", "q", "r", "12345"), new Address(3, "x", "y", "z", "12345"));
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    env.setParallelism(1);
    env.enableCheckpointing(100);
    AvroWriterFactory<Address> avroWriterFactory = AvroWriters.forSpecificRecord(Address.class);
    DataStream<Address> stream = env.addSource(new FiniteTestSource<>(data), TypeInformation.of(Address.class));
    stream.addSink(StreamingFileSink.forBulkFormat(Path.fromLocalFile(folder), avroWriterFactory).withBucketAssigner(new UniqueBucketAssigner<>("test")).build());
    env.execute();
    validateResults(folder, new SpecificDatumReader<>(Address.class), data);
}
Also used : Address(org.apache.flink.formats.avro.generated.Address) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) File(java.io.File) Test(org.junit.Test)

Example 4 with Address

use of org.apache.flink.formats.avro.generated.Address in project flink by apache.

the class AvroDeserializationSchemaTest method testNullRecord.

@Test
public void testNullRecord() throws Exception {
    DeserializationSchema<Address> deserializer = AvroDeserializationSchema.forSpecific(Address.class);
    Address deserializedAddress = deserializer.deserialize(null);
    assertNull(deserializedAddress);
}
Also used : Address(org.apache.flink.formats.avro.generated.Address) Test(org.junit.Test)

Example 5 with Address

use of org.apache.flink.formats.avro.generated.Address in project flink by apache.

the class AvroDeserializationSchemaTest method testSpecificRecord.

@Test
public void testSpecificRecord() throws Exception {
    DeserializationSchema<Address> deserializer = AvroDeserializationSchema.forSpecific(Address.class);
    byte[] encodedAddress = writeRecord(address);
    Address deserializedAddress = deserializer.deserialize(encodedAddress);
    assertEquals(address, deserializedAddress);
}
Also used : Address(org.apache.flink.formats.avro.generated.Address) Test(org.junit.Test)

Aggregations

Address (org.apache.flink.formats.avro.generated.Address)8 User (org.apache.flink.formats.avro.generated.User)5 Test (org.junit.Test)5 Fixed2 (org.apache.flink.formats.avro.generated.Fixed2)4 HashMap (java.util.HashMap)3 Fixed16 (org.apache.flink.formats.avro.generated.Fixed16)3 ArrayList (java.util.ArrayList)2 Random (java.util.Random)2 DataFileWriter (org.apache.avro.file.DataFileWriter)2 SpecificDatumWriter (org.apache.avro.specific.SpecificDatumWriter)2 StreamExecutionEnvironment (org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)2 File (java.io.File)1 SpecificRecord (org.apache.avro.specific.SpecificRecord)1 Tuple3 (org.apache.flink.api.java.tuple.Tuple3)1 Table (org.apache.flink.table.api.Table)1 StreamTableEnvironment (org.apache.flink.table.api.bridge.java.StreamTableEnvironment)1 Row (org.apache.flink.types.Row)1 Before (org.junit.Before)1