Examples with FlinkSchema org.apache.flink.streaming.connectors.pulsar.serialization.FlinkSchema used on opensource projects

Example 1 with FlinkSchema

use of org.apache.flink.streaming.connectors.pulsar.serialization.FlinkSchema in project pulsar-flink by streamnative.

the class RowDataDerSerializationSchemaTest method testProtoBufSerializeDeserialize.

@Test
public void testProtoBufSerializeDeserialize() throws Exception {
    String topicName = newTopic();
    RowType rowType = PbRowTypeInformation.generateRowType(SimpleTest.getDescriptor());
    PbRowDataSerializationSchema serializationSchema = new PbRowDataSerializationSchema(rowType, SimpleTest.class.getName());
    PbRowDataDeserializationSchema deserializationSchema = new PbRowDataDeserializationSchema(rowType, InternalTypeInfo.of(rowType), SimpleTest.class.getName(), false, false);
    SimpleTest simple = SimpleTest.newBuilder().setA(1).setB(2L).setC(false).setD(0.1f).setE(0.01).setF("haha").setG(ByteString.copyFrom(new byte[] { 1 })).setH(SimpleTest.Corpus.IMAGES).setFAbc7D(// test fieldNameToJsonName
    1).build();
    RowData rowData = deserializationSchema.deserialize(simple.toByteArray());
    org.apache.pulsar.client.api.Schema<SimpleTest> schema = org.apache.pulsar.client.api.Schema.PROTOBUF_NATIVE(SimpleTest.class);
    final FlinkSchema<RowData> rowDataFlinkSchema = new FlinkSchema<>(schema.getSchemaInfo(), serializationSchema, deserializationSchema);
    sendMessage(topicName, rowDataFlinkSchema, rowData);
    final CompletableFuture<byte[]> consumer = autoConsumer(topicName);
    RowData newRowData = deserializationSchema.deserialize(consumer.get(10000, TimeUnit.MILLISECONDS));
    newRowData = validatePbRow(newRowData, PbRowTypeInformation.generateRowType(SimpleTest.getDescriptor()));
    assertEquals(9, newRowData.getArity());
    assertEquals(1, newRowData.getInt(0));
    assertEquals(2L, newRowData.getLong(1));
    assertFalse((boolean) newRowData.getBoolean(2));
    assertEquals(Float.valueOf(0.1f), Float.valueOf(newRowData.getFloat(3)));
    assertEquals(Double.valueOf(0.01d), Double.valueOf(newRowData.getDouble(4)));
    assertEquals("haha", newRowData.getString(5).toString());
    assertEquals(1, (newRowData.getBinary(6))[0]);
    assertEquals("IMAGES", newRowData.getString(7).toString());
    assertEquals(1, newRowData.getInt(8));
}

Example 2 with FlinkSchema

use of org.apache.flink.streaming.connectors.pulsar.serialization.FlinkSchema in project pulsar-flink by streamnative.

the class DynamicPulsarSerializationSchema method buildSchema.

private FlinkSchema<RowData> buildSchema() {
    if (StringUtils.isBlank(valueFormatType)) {
        return new FlinkSchema<>(Schema.BYTES.getSchemaInfo(), valueSerialization, null);
    }
    Configuration configuration = new Configuration();
    hackPbSerializationSchema(configuration);
    SchemaInfo schemaInfo = SchemaUtils.tableSchemaToSchemaInfo(valueFormatType, valueDataType, configuration);
    return new FlinkSchema<>(schemaInfo, valueSerialization, null);
}

Example 3 with FlinkSchema

use of org.apache.flink.streaming.connectors.pulsar.serialization.FlinkSchema in project pulsar-flink by streamnative.

the class RowDataDerSerializationSchemaTest method toPulsarSchema.

public <T> org.apache.pulsar.client.api.Schema<T> toPulsarSchema(SchemaType schemaType, Schema avroSchema, SerializationSchema<T> serializationSchema, DeserializationSchema<T> deserializationSchema) {
    byte[] schemaBytes = avroSchema.toString().getBytes(StandardCharsets.UTF_8);
    SchemaInfoImpl si = new SchemaInfoImpl();
    si.setName("Record");
    si.setSchema(schemaBytes);
    si.setType(schemaType);
    return new FlinkSchema<>(si, serializationSchema, deserializationSchema);
}