Examples with RowFormatInfo org.apache.inlong.sort.formats.common.RowFormatInfo used on opensource projects

Example 31 with RowFormatInfo

use of org.apache.inlong.sort.formats.common.RowFormatInfo in project incubator-inlong by apache.

the class KvFormatFactory method createProjectedDeserializationSchema.

@Override
public DeserializationSchema<Row> createProjectedDeserializationSchema(Map<String, String> properties, int[] fields) {
    final DescriptorProperties descriptorProperties = getValidatedProperties(properties);
    final RowFormatInfo rowFormatInfo = TableFormatUtils.getRowFormatInfo(descriptorProperties);
    final RowFormatInfo projectedRowFormatInfo = TableFormatUtils.projectRowFormatInfo(rowFormatInfo, fields);
    return buildDeserializationSchema(descriptorProperties, projectedRowFormatInfo);
}

Example 32 with RowFormatInfo

use of org.apache.inlong.sort.formats.common.RowFormatInfo in project incubator-inlong by apache.

the class KvFormatFactory method buildDeserializationSchema.

private static KvDeserializationSchema buildDeserializationSchema(DescriptorProperties descriptorProperties, RowFormatInfo rowFormatInfo) {
    for (FormatInfo formatInfo : rowFormatInfo.getFieldFormatInfos()) {
        if (!(formatInfo instanceof BasicFormatInfo)) {
            throw new ValidationException("Currently only basic formats " + "are supported in kv formats.");
        }
    }
    KvDeserializationSchema.Builder builder = new KvDeserializationSchema.Builder(rowFormatInfo);
    descriptorProperties.getOptionalString(TableFormatConstants.FORMAT_CHARSET).ifPresent(builder::setCharset);
    descriptorProperties.getOptionalCharacter(TableFormatConstants.FORMAT_ENTRY_DELIMITER).ifPresent(builder::setEntryDelimiter);
    descriptorProperties.getOptionalCharacter(TableFormatConstants.FORMAT_KV_DELIMITER).ifPresent(builder::setKvDelimiter);
    descriptorProperties.getOptionalCharacter(TableFormatConstants.FORMAT_ESCAPE_CHARACTER).ifPresent(builder::setEscapeCharacter);
    descriptorProperties.getOptionalCharacter(TableFormatConstants.FORMAT_QUOTE_CHARACTER).ifPresent(builder::setQuoteCharacter);
    descriptorProperties.getOptionalString(TableFormatConstants.FORMAT_NULL_LITERAL).ifPresent(builder::setNullLiteral);
    return builder.build();
}

Example 33 with RowFormatInfo

use of org.apache.inlong.sort.formats.common.RowFormatInfo in project incubator-inlong by apache.

the class CsvSerializationSchemaTest method testBasicSerialization.

private static <T> void testBasicSerialization(Consumer<CsvSerializationSchema.Builder> config, BasicFormatInfo<T> basicFormatInfo, T record, String expectedText) {
    RowFormatInfo rowFormatInfo = new RowFormatInfo(new String[] { "f" }, new FormatInfo[] { basicFormatInfo });
    CsvSerializationSchema.Builder builder = new CsvSerializationSchema.Builder(rowFormatInfo);
    config.accept(builder);
    CsvSerializationSchema serializer = builder.build();
    String text = new String(serializer.serialize(Row.of(record)));
    assertEquals(expectedText, text);
}

Example 34 with RowFormatInfo

use of org.apache.inlong.sort.formats.common.RowFormatInfo in project incubator-inlong by apache.

the class MultiTopicPulsarSourceFunctionTest method producerMessageToPulsar.

protected void producerMessageToPulsar(String pulsarBrokerUrl) throws Exception {
    RowSerializer rowSerializer = CommonUtils.generateRowSerializer(new RowFormatInfo(new String[] { "f1", "f2" }, new FormatInfo[] { StringFormatInfo.INSTANCE, StringFormatInfo.INSTANCE }));
    DataOutputSerializer dataOutputSerializer = new DataOutputSerializer(1024);
    try (PulsarClient client = PulsarClient.builder().serviceUrl(pulsarBrokerUrl).build();
        Producer<byte[]> producer = client.newProducer().topic(TEST_TOPIC).create()) {
        for (int cnt = 0; cnt < TOTAL_COUNT; cnt++) {
            Row row = Row.of(String.valueOf(cnt), String.valueOf(cnt));
            rowSerializer.serialize(row, dataOutputSerializer);
            ByteArrayOutputStream byt = new ByteArrayOutputStream();
            ObjectOutputStream objectOutputStream = new ObjectOutputStream(byt);
            objectOutputStream.writeObject(new SerializedRecord(1L, 0, dataOutputSerializer.getCopyOfBuffer()));
            producer.send(byt.toByteArray());
        }
    }
}