Search in sources :

Example 1 with TimestampFormat

use of org.apache.flink.formats.common.TimestampFormat in project flink by apache.

the class DebeziumJsonDecodingFormat method createRuntimeDecoder.

@Override
public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context, DataType physicalDataType, int[][] projections) {
    physicalDataType = Projection.of(projections).project(physicalDataType);
    final List<ReadableMetadata> readableMetadata = metadataKeys.stream().map(k -> Stream.of(ReadableMetadata.values()).filter(rm -> rm.key.equals(k)).findFirst().orElseThrow(IllegalStateException::new)).collect(Collectors.toList());
    final List<DataTypes.Field> metadataFields = readableMetadata.stream().map(m -> DataTypes.FIELD(m.key, m.dataType)).collect(Collectors.toList());
    final DataType producedDataType = DataTypeUtils.appendRowFields(physicalDataType, metadataFields);
    final TypeInformation<RowData> producedTypeInfo = context.createTypeInformation(producedDataType);
    return new DebeziumJsonDeserializationSchema(physicalDataType, readableMetadata, producedTypeInfo, schemaInclude, ignoreParseErrors, timestampFormat);
}
Also used : DataType(org.apache.flink.table.types.DataType) DynamicTableSource(org.apache.flink.table.connector.source.DynamicTableSource) RowData(org.apache.flink.table.data.RowData) TimestampData(org.apache.flink.table.data.TimestampData) ChangelogMode(org.apache.flink.table.connector.ChangelogMode) DataTypes(org.apache.flink.table.api.DataTypes) TimestampFormat(org.apache.flink.formats.common.TimestampFormat) ProjectableDecodingFormat(org.apache.flink.table.connector.format.ProjectableDecodingFormat) Collectors(java.util.stream.Collectors) DeserializationSchema(org.apache.flink.api.common.serialization.DeserializationSchema) LinkedHashMap(java.util.LinkedHashMap) DecodingFormat(org.apache.flink.table.connector.format.DecodingFormat) StringData(org.apache.flink.table.data.StringData) List(java.util.List) GenericRowData(org.apache.flink.table.data.GenericRowData) Stream(java.util.stream.Stream) RowKind(org.apache.flink.types.RowKind) GenericMapData(org.apache.flink.table.data.GenericMapData) Map(java.util.Map) MetadataConverter(org.apache.flink.formats.json.debezium.DebeziumJsonDeserializationSchema.MetadataConverter) TypeInformation(org.apache.flink.api.common.typeinfo.TypeInformation) Collections(java.util.Collections) Projection(org.apache.flink.table.connector.Projection) DataTypeUtils(org.apache.flink.table.types.utils.DataTypeUtils) RowData(org.apache.flink.table.data.RowData) GenericRowData(org.apache.flink.table.data.GenericRowData) DataType(org.apache.flink.table.types.DataType)

Example 2 with TimestampFormat

use of org.apache.flink.formats.common.TimestampFormat in project flink by apache.

the class DebeziumJsonFormatFactory method createEncodingFormat.

@Override
public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context, ReadableConfig formatOptions) {
    FactoryUtil.validateFactoryOptions(this, formatOptions);
    validateEncodingFormatOptions(formatOptions);
    TimestampFormat timestampFormat = JsonFormatOptionsUtil.getTimestampFormat(formatOptions);
    JsonFormatOptions.MapNullKeyMode mapNullKeyMode = JsonFormatOptionsUtil.getMapNullKeyMode(formatOptions);
    String mapNullKeyLiteral = formatOptions.get(JSON_MAP_NULL_KEY_LITERAL);
    final boolean encodeDecimalAsPlainNumber = formatOptions.get(ENCODE_DECIMAL_AS_PLAIN_NUMBER);
    return new EncodingFormat<SerializationSchema<RowData>>() {

        @Override
        public ChangelogMode getChangelogMode() {
            return ChangelogMode.newBuilder().addContainedKind(RowKind.INSERT).addContainedKind(RowKind.UPDATE_BEFORE).addContainedKind(RowKind.UPDATE_AFTER).addContainedKind(RowKind.DELETE).build();
        }

        @Override
        public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context, DataType consumedDataType) {
            final RowType rowType = (RowType) consumedDataType.getLogicalType();
            return new DebeziumJsonSerializationSchema(rowType, timestampFormat, mapNullKeyMode, mapNullKeyLiteral, encodeDecimalAsPlainNumber);
        }
    };
}
Also used : EncodingFormat(org.apache.flink.table.connector.format.EncodingFormat) JsonFormatOptions(org.apache.flink.formats.json.JsonFormatOptions) RowData(org.apache.flink.table.data.RowData) DataType(org.apache.flink.table.types.DataType) RowType(org.apache.flink.table.types.logical.RowType) TimestampFormat(org.apache.flink.formats.common.TimestampFormat)

Example 3 with TimestampFormat

use of org.apache.flink.formats.common.TimestampFormat in project flink by apache.

the class DebeziumJsonFormatFactory method createDecodingFormat.

@Override
public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context, ReadableConfig formatOptions) {
    FactoryUtil.validateFactoryOptions(this, formatOptions);
    validateDecodingFormatOptions(formatOptions);
    final boolean schemaInclude = formatOptions.get(SCHEMA_INCLUDE);
    final boolean ignoreParseErrors = formatOptions.get(IGNORE_PARSE_ERRORS);
    final TimestampFormat timestampFormat = JsonFormatOptionsUtil.getTimestampFormat(formatOptions);
    return new DebeziumJsonDecodingFormat(schemaInclude, ignoreParseErrors, timestampFormat);
}
Also used : TimestampFormat(org.apache.flink.formats.common.TimestampFormat)

Example 4 with TimestampFormat

use of org.apache.flink.formats.common.TimestampFormat in project flink by apache.

the class OggJsonFormatFactory method createDecodingFormat.

@Override
public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context, ReadableConfig formatOptions) {
    FactoryUtil.validateFactoryOptions(this, formatOptions);
    validateDecodingFormatOptions(formatOptions);
    final boolean ignoreParseErrors = formatOptions.get(IGNORE_PARSE_ERRORS);
    final TimestampFormat timestampFormat = JsonFormatOptionsUtil.getTimestampFormat(formatOptions);
    return new OggJsonDecodingFormat(ignoreParseErrors, timestampFormat);
}
Also used : TimestampFormat(org.apache.flink.formats.common.TimestampFormat)

Example 5 with TimestampFormat

use of org.apache.flink.formats.common.TimestampFormat in project flink by apache.

the class MaxwellJsonDecodingFormat method createRuntimeDecoder.

@Override
public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context, DataType physicalDataType, int[][] projections) {
    physicalDataType = Projection.of(projections).project(physicalDataType);
    final List<ReadableMetadata> readableMetadata = metadataKeys.stream().map(k -> Stream.of(ReadableMetadata.values()).filter(rm -> rm.key.equals(k)).findFirst().orElseThrow(() -> new IllegalStateException(String.format("Could not find the requested metadata key: %s", k)))).collect(Collectors.toList());
    final List<DataTypes.Field> metadataFields = readableMetadata.stream().map(m -> DataTypes.FIELD(m.key, m.dataType)).collect(Collectors.toList());
    final DataType producedDataType = DataTypeUtils.appendRowFields(physicalDataType, metadataFields);
    final TypeInformation<RowData> producedTypeInfo = context.createTypeInformation(producedDataType);
    return new MaxwellJsonDeserializationSchema(physicalDataType, readableMetadata, producedTypeInfo, ignoreParseErrors, timestampFormat);
}
Also used : DataType(org.apache.flink.table.types.DataType) DynamicTableSource(org.apache.flink.table.connector.source.DynamicTableSource) RowData(org.apache.flink.table.data.RowData) TimestampData(org.apache.flink.table.data.TimestampData) ChangelogMode(org.apache.flink.table.connector.ChangelogMode) DataTypes(org.apache.flink.table.api.DataTypes) TimestampFormat(org.apache.flink.formats.common.TimestampFormat) ProjectableDecodingFormat(org.apache.flink.table.connector.format.ProjectableDecodingFormat) Collectors(java.util.stream.Collectors) DeserializationSchema(org.apache.flink.api.common.serialization.DeserializationSchema) LinkedHashMap(java.util.LinkedHashMap) DecodingFormat(org.apache.flink.table.connector.format.DecodingFormat) List(java.util.List) GenericRowData(org.apache.flink.table.data.GenericRowData) Stream(java.util.stream.Stream) RowKind(org.apache.flink.types.RowKind) Map(java.util.Map) MetadataConverter(org.apache.flink.formats.json.maxwell.MaxwellJsonDeserializationSchema.MetadataConverter) TypeInformation(org.apache.flink.api.common.typeinfo.TypeInformation) Collections(java.util.Collections) Projection(org.apache.flink.table.connector.Projection) DataTypeUtils(org.apache.flink.table.types.utils.DataTypeUtils) RowData(org.apache.flink.table.data.RowData) GenericRowData(org.apache.flink.table.data.GenericRowData) DataType(org.apache.flink.table.types.DataType)

Aggregations

TimestampFormat (org.apache.flink.formats.common.TimestampFormat)14 RowData (org.apache.flink.table.data.RowData)10 DataType (org.apache.flink.table.types.DataType)10 RowType (org.apache.flink.table.types.logical.RowType)6 EncodingFormat (org.apache.flink.table.connector.format.EncodingFormat)5 Collections (java.util.Collections)4 LinkedHashMap (java.util.LinkedHashMap)4 List (java.util.List)4 Map (java.util.Map)4 Collectors (java.util.stream.Collectors)4 Stream (java.util.stream.Stream)4 DeserializationSchema (org.apache.flink.api.common.serialization.DeserializationSchema)4 TypeInformation (org.apache.flink.api.common.typeinfo.TypeInformation)4 JsonFormatOptions (org.apache.flink.formats.json.JsonFormatOptions)4 DataTypes (org.apache.flink.table.api.DataTypes)4 ChangelogMode (org.apache.flink.table.connector.ChangelogMode)4 DecodingFormat (org.apache.flink.table.connector.format.DecodingFormat)4 ProjectableDecodingFormat (org.apache.flink.table.connector.format.ProjectableDecodingFormat)4 DynamicTableSource (org.apache.flink.table.connector.source.DynamicTableSource)4 GenericRowData (org.apache.flink.table.data.GenericRowData)4