Search in sources :

Example 1 with TableFactoryHelper

use of org.apache.flink.table.factories.FactoryUtil.TableFactoryHelper in project flink by apache.

the class HBase2DynamicTableFactory method createDynamicTableSource.

@Override
public DynamicTableSource createDynamicTableSource(Context context) {
    TableFactoryHelper helper = createTableFactoryHelper(this, context);
    helper.validateExcept(PROPERTIES_PREFIX);
    final ReadableConfig tableOptions = helper.getOptions();
    validatePrimaryKey(context.getPhysicalRowDataType(), context.getPrimaryKeyIndexes());
    String tableName = tableOptions.get(TABLE_NAME);
    Configuration hbaseConf = getHBaseConfiguration(tableOptions);
    HBaseLookupOptions lookupOptions = getHBaseLookupOptions(tableOptions);
    String nullStringLiteral = tableOptions.get(NULL_STRING_LITERAL);
    HBaseTableSchema hbaseSchema = HBaseTableSchema.fromDataType(context.getPhysicalRowDataType());
    return new HBaseDynamicTableSource(hbaseConf, tableName, hbaseSchema, nullStringLiteral, lookupOptions);
}
Also used : HBaseLookupOptions(org.apache.flink.connector.hbase.options.HBaseLookupOptions) HBaseConnectorOptionsUtil.getHBaseLookupOptions(org.apache.flink.connector.hbase.table.HBaseConnectorOptionsUtil.getHBaseLookupOptions) HBaseDynamicTableSource(org.apache.flink.connector.hbase2.source.HBaseDynamicTableSource) ReadableConfig(org.apache.flink.configuration.ReadableConfig) Configuration(org.apache.hadoop.conf.Configuration) HBaseConnectorOptionsUtil.getHBaseConfiguration(org.apache.flink.connector.hbase.table.HBaseConnectorOptionsUtil.getHBaseConfiguration) TableFactoryHelper(org.apache.flink.table.factories.FactoryUtil.TableFactoryHelper) FactoryUtil.createTableFactoryHelper(org.apache.flink.table.factories.FactoryUtil.createTableFactoryHelper) HBaseTableSchema(org.apache.flink.connector.hbase.util.HBaseTableSchema)

Example 2 with TableFactoryHelper

use of org.apache.flink.table.factories.FactoryUtil.TableFactoryHelper in project flink by apache.

the class HBase1DynamicTableFactory method createDynamicTableSink.

@Override
public DynamicTableSink createDynamicTableSink(Context context) {
    TableFactoryHelper helper = createTableFactoryHelper(this, context);
    helper.validateExcept(PROPERTIES_PREFIX);
    final ReadableConfig tableOptions = helper.getOptions();
    validatePrimaryKey(context.getPhysicalRowDataType(), context.getPrimaryKeyIndexes());
    String tableName = tableOptions.get(TABLE_NAME);
    Configuration hbaseConf = getHBaseConfiguration(tableOptions);
    HBaseWriteOptions hBaseWriteOptions = getHBaseWriteOptions(tableOptions);
    String nullStringLiteral = tableOptions.get(NULL_STRING_LITERAL);
    HBaseTableSchema hbaseSchema = HBaseTableSchema.fromDataType(context.getPhysicalRowDataType());
    return new HBaseDynamicTableSink(tableName, hbaseSchema, hbaseConf, hBaseWriteOptions, nullStringLiteral);
}
Also used : ReadableConfig(org.apache.flink.configuration.ReadableConfig) Configuration(org.apache.hadoop.conf.Configuration) HBaseConnectorOptionsUtil.getHBaseConfiguration(org.apache.flink.connector.hbase.table.HBaseConnectorOptionsUtil.getHBaseConfiguration) TableFactoryHelper(org.apache.flink.table.factories.FactoryUtil.TableFactoryHelper) FactoryUtil.createTableFactoryHelper(org.apache.flink.table.factories.FactoryUtil.createTableFactoryHelper) HBaseTableSchema(org.apache.flink.connector.hbase.util.HBaseTableSchema) HBaseDynamicTableSink(org.apache.flink.connector.hbase1.sink.HBaseDynamicTableSink) HBaseWriteOptions(org.apache.flink.connector.hbase.options.HBaseWriteOptions) HBaseConnectorOptionsUtil.getHBaseWriteOptions(org.apache.flink.connector.hbase.table.HBaseConnectorOptionsUtil.getHBaseWriteOptions)

Example 3 with TableFactoryHelper

use of org.apache.flink.table.factories.FactoryUtil.TableFactoryHelper in project flink by apache.

the class HBase1DynamicTableFactory method createDynamicTableSource.

@Override
public DynamicTableSource createDynamicTableSource(Context context) {
    TableFactoryHelper helper = createTableFactoryHelper(this, context);
    helper.validateExcept(PROPERTIES_PREFIX);
    final ReadableConfig tableOptions = helper.getOptions();
    validatePrimaryKey(context.getPhysicalRowDataType(), context.getPrimaryKeyIndexes());
    String tableName = tableOptions.get(TABLE_NAME);
    Configuration hbaseClientConf = getHBaseConfiguration(tableOptions);
    String nullStringLiteral = tableOptions.get(NULL_STRING_LITERAL);
    HBaseTableSchema hbaseSchema = HBaseTableSchema.fromDataType(context.getPhysicalRowDataType());
    return new HBaseDynamicTableSource(hbaseClientConf, tableName, hbaseSchema, nullStringLiteral, getHBaseLookupOptions(tableOptions));
}
Also used : HBaseDynamicTableSource(org.apache.flink.connector.hbase1.source.HBaseDynamicTableSource) ReadableConfig(org.apache.flink.configuration.ReadableConfig) Configuration(org.apache.hadoop.conf.Configuration) HBaseConnectorOptionsUtil.getHBaseConfiguration(org.apache.flink.connector.hbase.table.HBaseConnectorOptionsUtil.getHBaseConfiguration) TableFactoryHelper(org.apache.flink.table.factories.FactoryUtil.TableFactoryHelper) FactoryUtil.createTableFactoryHelper(org.apache.flink.table.factories.FactoryUtil.createTableFactoryHelper) HBaseTableSchema(org.apache.flink.connector.hbase.util.HBaseTableSchema)

Example 4 with TableFactoryHelper

use of org.apache.flink.table.factories.FactoryUtil.TableFactoryHelper in project flink by apache.

the class KafkaDynamicTableFactory method createDynamicTableSink.

@Override
public DynamicTableSink createDynamicTableSink(Context context) {
    final TableFactoryHelper helper = FactoryUtil.createTableFactoryHelper(this, autoCompleteSchemaRegistrySubject(context));
    final Optional<EncodingFormat<SerializationSchema<RowData>>> keyEncodingFormat = getKeyEncodingFormat(helper);
    final EncodingFormat<SerializationSchema<RowData>> valueEncodingFormat = getValueEncodingFormat(helper);
    helper.validateExcept(PROPERTIES_PREFIX);
    final ReadableConfig tableOptions = helper.getOptions();
    final DeliveryGuarantee deliveryGuarantee = validateDeprecatedSemantic(tableOptions);
    validateTableSinkOptions(tableOptions);
    KafkaConnectorOptionsUtil.validateDeliveryGuarantee(tableOptions);
    validatePKConstraints(context.getObjectIdentifier(), context.getPrimaryKeyIndexes(), context.getCatalogTable().getOptions(), valueEncodingFormat);
    final DataType physicalDataType = context.getPhysicalRowDataType();
    final int[] keyProjection = createKeyFormatProjection(tableOptions, physicalDataType);
    final int[] valueProjection = createValueFormatProjection(tableOptions, physicalDataType);
    final String keyPrefix = tableOptions.getOptional(KEY_FIELDS_PREFIX).orElse(null);
    final Integer parallelism = tableOptions.getOptional(SINK_PARALLELISM).orElse(null);
    return createKafkaTableSink(physicalDataType, keyEncodingFormat.orElse(null), valueEncodingFormat, keyProjection, valueProjection, keyPrefix, tableOptions.get(TOPIC).get(0), getKafkaProperties(context.getCatalogTable().getOptions()), getFlinkKafkaPartitioner(tableOptions, context.getClassLoader()).orElse(null), deliveryGuarantee, parallelism, tableOptions.get(TRANSACTIONAL_ID_PREFIX));
}
Also used : EncodingFormat(org.apache.flink.table.connector.format.EncodingFormat) RowData(org.apache.flink.table.data.RowData) ReadableConfig(org.apache.flink.configuration.ReadableConfig) DeliveryGuarantee(org.apache.flink.connector.base.DeliveryGuarantee) SerializationSchema(org.apache.flink.api.common.serialization.SerializationSchema) TableFactoryHelper(org.apache.flink.table.factories.FactoryUtil.TableFactoryHelper) DataType(org.apache.flink.table.types.DataType)

Example 5 with TableFactoryHelper

use of org.apache.flink.table.factories.FactoryUtil.TableFactoryHelper in project flink by apache.

the class KafkaDynamicTableFactory method createDynamicTableSource.

@Override
public DynamicTableSource createDynamicTableSource(Context context) {
    final TableFactoryHelper helper = FactoryUtil.createTableFactoryHelper(this, context);
    final Optional<DecodingFormat<DeserializationSchema<RowData>>> keyDecodingFormat = getKeyDecodingFormat(helper);
    final DecodingFormat<DeserializationSchema<RowData>> valueDecodingFormat = getValueDecodingFormat(helper);
    helper.validateExcept(PROPERTIES_PREFIX);
    final ReadableConfig tableOptions = helper.getOptions();
    validateTableSourceOptions(tableOptions);
    validatePKConstraints(context.getObjectIdentifier(), context.getPrimaryKeyIndexes(), context.getCatalogTable().getOptions(), valueDecodingFormat);
    final StartupOptions startupOptions = getStartupOptions(tableOptions);
    final Properties properties = getKafkaProperties(context.getCatalogTable().getOptions());
    // add topic-partition discovery
    final Optional<Long> partitionDiscoveryInterval = tableOptions.getOptional(SCAN_TOPIC_PARTITION_DISCOVERY).map(Duration::toMillis);
    properties.setProperty(KafkaSourceOptions.PARTITION_DISCOVERY_INTERVAL_MS.key(), partitionDiscoveryInterval.orElse(-1L).toString());
    final DataType physicalDataType = context.getPhysicalRowDataType();
    final int[] keyProjection = createKeyFormatProjection(tableOptions, physicalDataType);
    final int[] valueProjection = createValueFormatProjection(tableOptions, physicalDataType);
    final String keyPrefix = tableOptions.getOptional(KEY_FIELDS_PREFIX).orElse(null);
    return createKafkaTableSource(physicalDataType, keyDecodingFormat.orElse(null), valueDecodingFormat, keyProjection, valueProjection, keyPrefix, getSourceTopics(tableOptions), getSourceTopicPattern(tableOptions), properties, startupOptions.startupMode, startupOptions.specificOffsets, startupOptions.startupTimestampMillis, context.getObjectIdentifier().asSummaryString());
}
Also used : DecodingFormat(org.apache.flink.table.connector.format.DecodingFormat) TableFactoryHelper(org.apache.flink.table.factories.FactoryUtil.TableFactoryHelper) Duration(java.time.Duration) KafkaConnectorOptionsUtil.getKafkaProperties(org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptionsUtil.getKafkaProperties) Properties(java.util.Properties) DeserializationSchema(org.apache.flink.api.common.serialization.DeserializationSchema) RowData(org.apache.flink.table.data.RowData) ReadableConfig(org.apache.flink.configuration.ReadableConfig) DataType(org.apache.flink.table.types.DataType) KafkaConnectorOptionsUtil.getStartupOptions(org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptionsUtil.getStartupOptions) StartupOptions(org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptionsUtil.StartupOptions)

Aggregations

TableFactoryHelper (org.apache.flink.table.factories.FactoryUtil.TableFactoryHelper)8 ReadableConfig (org.apache.flink.configuration.ReadableConfig)6 HBaseConnectorOptionsUtil.getHBaseConfiguration (org.apache.flink.connector.hbase.table.HBaseConnectorOptionsUtil.getHBaseConfiguration)4 HBaseTableSchema (org.apache.flink.connector.hbase.util.HBaseTableSchema)4 RowData (org.apache.flink.table.data.RowData)4 FactoryUtil.createTableFactoryHelper (org.apache.flink.table.factories.FactoryUtil.createTableFactoryHelper)4 Configuration (org.apache.hadoop.conf.Configuration)4 DeserializationSchema (org.apache.flink.api.common.serialization.DeserializationSchema)2 SerializationSchema (org.apache.flink.api.common.serialization.SerializationSchema)2 HBaseWriteOptions (org.apache.flink.connector.hbase.options.HBaseWriteOptions)2 HBaseConnectorOptionsUtil.getHBaseWriteOptions (org.apache.flink.connector.hbase.table.HBaseConnectorOptionsUtil.getHBaseWriteOptions)2 DecodingFormat (org.apache.flink.table.connector.format.DecodingFormat)2 EncodingFormat (org.apache.flink.table.connector.format.EncodingFormat)2 DataType (org.apache.flink.table.types.DataType)2 Duration (java.time.Duration)1 Properties (java.util.Properties)1 DeliveryGuarantee (org.apache.flink.connector.base.DeliveryGuarantee)1 HBaseLookupOptions (org.apache.flink.connector.hbase.options.HBaseLookupOptions)1 HBaseConnectorOptionsUtil.getHBaseLookupOptions (org.apache.flink.connector.hbase.table.HBaseConnectorOptionsUtil.getHBaseLookupOptions)1 HBaseDynamicTableSink (org.apache.flink.connector.hbase1.sink.HBaseDynamicTableSink)1