Search in sources :

Example 6 with MQConsumeException

use of com.qlangtech.tis.async.message.client.consumer.MQConsumeException in project plugins by qlangtech.

the class FlinkCDCPostgreSQLSourceFunction method start.

@Override
public JobExecutionResult start(TargetResName dataxName, IDataxReader dataSource, List<ISelectedTab> tabs, IDataxProcessor dataXProcessor) throws MQConsumeException {
    try {
        BasicDataXRdbmsReader rdbmsReader = (BasicDataXRdbmsReader) dataSource;
        final BasicDataSourceFactory dsFactory = (BasicDataSourceFactory) rdbmsReader.getDataSourceFactory();
        BasicDataSourceFactory.ISchemaSupported schemaSupported = (BasicDataSourceFactory.ISchemaSupported) dsFactory;
        if (StringUtils.isEmpty(schemaSupported.getDBSchema())) {
            throw new IllegalStateException("dsFactory:" + dsFactory.dbName + " relevant dbSchema can not be null");
        }
        SourceChannel sourceChannel = new SourceChannel(SourceChannel.getSourceFunction(dsFactory, (tab) -> schemaSupported.getDBSchema() + "." + tab.getTabName(), tabs, (dbHost, dbs, tbs, debeziumProperties) -> {
            DateTimeConverter.setDatetimeConverters(PGDateTimeConverter.class.getName(), debeziumProperties);
            return dbs.stream().map((dbname) -> {
                SourceFunction<DTO> sourceFunction = PostgreSQLSource.<DTO>builder().hostname(dbHost).port(dsFactory.port).database(// monitor postgres database
                dbname).schemaList(// monitor inventory schema
                schemaSupported.getDBSchema()).tableList(// monitor products table
                tbs.toArray(new String[tbs.size()])).username(dsFactory.userName).password(dsFactory.password).debeziumProperties(debeziumProperties).deserializer(// converts SourceRecord to JSON String
                new TISDeserializationSchema()).build();
                return new ReaderSource(dbHost + ":" + dsFactory.port + "_" + dbname, sourceFunction);
            }).collect(Collectors.toList());
        }));
        for (ISelectedTab tab : tabs) {
            sourceChannel.addFocusTab(tab.getName());
        }
        return (JobExecutionResult) getConsumerHandle().consume(dataxName, sourceChannel, dataXProcessor);
    } catch (Exception e) {
        throw new MQConsumeException(e.getMessage(), e);
    }
}
Also used : MQConsumeException(com.qlangtech.tis.async.message.client.consumer.MQConsumeException) DateTimeConverter(com.qlangtech.plugins.incr.flink.cdc.valconvert.DateTimeConverter) IConsumerHandle(com.qlangtech.tis.async.message.client.consumer.IConsumerHandle) StringUtils(org.apache.commons.lang.StringUtils) TargetResName(com.qlangtech.tis.coredefine.module.action.TargetResName) BasicDataXRdbmsReader(com.qlangtech.tis.plugin.datax.common.BasicDataXRdbmsReader) ISelectedTab(com.qlangtech.tis.plugin.ds.ISelectedTab) DTO(com.qlangtech.tis.realtime.transfer.DTO) TISDeserializationSchema(com.qlangtech.plugins.incr.flink.cdc.TISDeserializationSchema) Collectors(java.util.stream.Collectors) BasicDataSourceFactory(com.qlangtech.tis.plugin.ds.BasicDataSourceFactory) SourceChannel(com.qlangtech.plugins.incr.flink.cdc.SourceChannel) ReaderSource(com.qlangtech.tis.realtime.ReaderSource) JobExecutionResult(org.apache.flink.api.common.JobExecutionResult) List(java.util.List) SourceFunction(org.apache.flink.streaming.api.functions.source.SourceFunction) IDataxReader(com.qlangtech.tis.datax.IDataxReader) IAsyncMsgDeserialize(com.qlangtech.tis.async.message.client.consumer.IAsyncMsgDeserialize) IDataxProcessor(com.qlangtech.tis.datax.IDataxProcessor) IMQListener(com.qlangtech.tis.async.message.client.consumer.IMQListener) PostgreSQLSource(com.ververica.cdc.connectors.postgres.PostgreSQLSource) BasicDataSourceFactory(com.qlangtech.tis.plugin.ds.BasicDataSourceFactory) SourceChannel(com.qlangtech.plugins.incr.flink.cdc.SourceChannel) ISelectedTab(com.qlangtech.tis.plugin.ds.ISelectedTab) MQConsumeException(com.qlangtech.tis.async.message.client.consumer.MQConsumeException) JobExecutionResult(org.apache.flink.api.common.JobExecutionResult) TISDeserializationSchema(com.qlangtech.plugins.incr.flink.cdc.TISDeserializationSchema) MQConsumeException(com.qlangtech.tis.async.message.client.consumer.MQConsumeException) BasicDataXRdbmsReader(com.qlangtech.tis.plugin.datax.common.BasicDataXRdbmsReader) ReaderSource(com.qlangtech.tis.realtime.ReaderSource) DTO(com.qlangtech.tis.realtime.transfer.DTO)

Aggregations

MQConsumeException (com.qlangtech.tis.async.message.client.consumer.MQConsumeException)6 ISelectedTab (com.qlangtech.tis.plugin.ds.ISelectedTab)5 JobExecutionResult (org.apache.flink.api.common.JobExecutionResult)5 SourceChannel (com.qlangtech.plugins.incr.flink.cdc.SourceChannel)4 TISDeserializationSchema (com.qlangtech.plugins.incr.flink.cdc.TISDeserializationSchema)4 IMQListener (com.qlangtech.tis.async.message.client.consumer.IMQListener)4 TargetResName (com.qlangtech.tis.coredefine.module.action.TargetResName)4 BasicDataXRdbmsReader (com.qlangtech.tis.plugin.datax.common.BasicDataXRdbmsReader)4 BasicDataSourceFactory (com.qlangtech.tis.plugin.ds.BasicDataSourceFactory)4 ReaderSource (com.qlangtech.tis.realtime.ReaderSource)4 DTO (com.qlangtech.tis.realtime.transfer.DTO)4 List (java.util.List)4 Collectors (java.util.stream.Collectors)4 IAsyncMsgDeserialize (com.qlangtech.tis.async.message.client.consumer.IAsyncMsgDeserialize)3 IConsumerHandle (com.qlangtech.tis.async.message.client.consumer.IConsumerHandle)3 IDataxProcessor (com.qlangtech.tis.datax.IDataxProcessor)3 IDataxReader (com.qlangtech.tis.datax.IDataxReader)3 DateTimeConverter (com.qlangtech.plugins.incr.flink.cdc.valconvert.DateTimeConverter)2 Collections (java.util.Collections)2 StringUtils (org.apache.commons.lang.StringUtils)2