Search in sources :

Example 1 with CUDCDCTestSuit

use of com.qlangtech.plugins.incr.flink.cdc.CUDCDCTestSuit in project plugins by qlangtech.

the class TestFlinkCDCMySQLSourceFactory method testBinlogConsumeWithDataStreamRegisterTable.

@Test
public void testBinlogConsumeWithDataStreamRegisterTable() throws Exception {
    FlinkCDCMySQLSourceFactory mysqlCDCFactory = new FlinkCDCMySQLSourceFactory();
    mysqlCDCFactory.startupOptions = "latest";
    final String tabName = "base";
    CUDCDCTestSuit cdcTestSuit = new CUDCDCTestSuit() {

        @Override
        protected BasicDataSourceFactory createDataSourceFactory(TargetResName dataxName) {
            return createMySqlDataSourceFactory(dataxName);
        }

        @Override
        protected String getColEscape() {
            return "`";
        }

        @Override
        protected IResultRows createConsumerHandle(String tabName) {
            return new TestTableRegisterFlinkSourceHandle(tabName, cols);
        }
    };
    cdcTestSuit.startTest(mysqlCDCFactory, tabName);
}
Also used : TargetResName(com.qlangtech.tis.coredefine.module.action.TargetResName) TestTableRegisterFlinkSourceHandle(com.qlangtech.plugins.incr.flink.cdc.source.TestTableRegisterFlinkSourceHandle) CUDCDCTestSuit(com.qlangtech.plugins.incr.flink.cdc.CUDCDCTestSuit) Test(org.junit.Test)

Example 2 with CUDCDCTestSuit

use of com.qlangtech.plugins.incr.flink.cdc.CUDCDCTestSuit in project plugins by qlangtech.

the class TestFlinkCDCMySQLSourceFactory method testBinlogConsume.

@Test
public void testBinlogConsume() throws Exception {
    FlinkCDCMySQLSourceFactory mysqlCDCFactory = new FlinkCDCMySQLSourceFactory();
    mysqlCDCFactory.startupOptions = "latest";
    final String tabName = "base";
    CUDCDCTestSuit cdcTestSuit = new CUDCDCTestSuit() {

        @Override
        protected BasicDataSourceFactory createDataSourceFactory(TargetResName dataxName) {
            return createMySqlDataSourceFactory(dataxName);
        }

        @Override
        protected String getColEscape() {
            return "`";
        }
    };
    cdcTestSuit.startTest(mysqlCDCFactory, tabName);
}
Also used : TargetResName(com.qlangtech.tis.coredefine.module.action.TargetResName) CUDCDCTestSuit(com.qlangtech.plugins.incr.flink.cdc.CUDCDCTestSuit) Test(org.junit.Test)

Example 3 with CUDCDCTestSuit

use of com.qlangtech.plugins.incr.flink.cdc.CUDCDCTestSuit in project plugins by qlangtech.

the class TestTISFlinkCDCOracleSourceFunction method testBinlogConsume.

@Test
public void testBinlogConsume() throws Exception {
    FlinkCDCOracleSourceFactory mysqlCDCFactory = new FlinkCDCOracleSourceFactory();
    mysqlCDCFactory.startupOptions = "latest";
    // debezium
    final String tabName = "DEBEZIUM.BASE";
    CUDCDCTestSuit cdcTestSuit = new CUDCDCTestSuit() {

        @Override
        protected BasicDataSourceFactory createDataSourceFactory(TargetResName dataxName) {
            return createMySqlDataSourceFactory(dataxName);
        }

        @Override
        protected void startProcessConn(Connection conn) throws SQLException {
            conn.setAutoCommit(false);
        }

        @Override
        protected int executePreparedStatement(Connection connection, PreparedStatement statement) throws SQLException {
            int count = super.executePreparedStatement(connection, statement);
            connection.commit();
            return count;
        }

        @Override
        protected int executeStatement(Connection connection, Statement statement, String sql) throws SQLException {
            int count = super.executeStatement(connection, statement, sql);
            connection.commit();
            return count;
        }
    };
    cdcTestSuit.startTest(mysqlCDCFactory, tabName);
}
Also used : TargetResName(com.qlangtech.tis.coredefine.module.action.TargetResName) PreparedStatement(java.sql.PreparedStatement) Statement(java.sql.Statement) Connection(java.sql.Connection) PreparedStatement(java.sql.PreparedStatement) CUDCDCTestSuit(com.qlangtech.plugins.incr.flink.cdc.CUDCDCTestSuit)

Example 4 with CUDCDCTestSuit

use of com.qlangtech.plugins.incr.flink.cdc.CUDCDCTestSuit in project plugins by qlangtech.

the class TestFlinkCDCPostgreSQLSourceFunction method testBinlogConsume.

@Test
public void testBinlogConsume() throws Exception {
    FlinkCDCPostreSQLSourceFactory pgCDCFactory = new FlinkCDCPostreSQLSourceFactory();
    final String tabName = "base";
    CUDCDCTestSuit cdcTestSuit = new CUDCDCTestSuit() {

        @Override
        protected BasicDataSourceFactory createDataSourceFactory(TargetResName dataxName) {
            return createPGDataSourceFactory(dataxName);
        }

        @Override
        protected int executePreparedStatement(Connection connection, PreparedStatement statement) throws SQLException {
            int updateCount = super.executePreparedStatement(connection, statement);
            connection.commit();
            return updateCount;
        }

        @Override
        protected String createTableName(String tabName) {
            return schemaName + "." + tabName;
        }

        @Override
        protected int executeStatement(Connection connection, java.sql.Statement statement, String sql) throws SQLException {
            int updateCount = super.executeStatement(connection, statement, sql);
            connection.commit();
            return updateCount;
        }

        @Override
        protected void startProcessConn(Connection conn) throws SQLException {
            super.startProcessConn(conn);
            conn.setAutoCommit(false);
        }
    };
    cdcTestSuit.startTest(pgCDCFactory, tabName);
}
Also used : TargetResName(com.qlangtech.tis.coredefine.module.action.TargetResName) PreparedStatement(java.sql.PreparedStatement) Connection(java.sql.Connection) PreparedStatement(java.sql.PreparedStatement) CUDCDCTestSuit(com.qlangtech.plugins.incr.flink.cdc.CUDCDCTestSuit) Test(org.junit.Test)

Example 5 with CUDCDCTestSuit

use of com.qlangtech.plugins.incr.flink.cdc.CUDCDCTestSuit in project plugins by qlangtech.

the class TestFlinkCDCMySQLSourceFactory method testBinlogConsumeWithDataStreamRegisterInstaneDetailTable.

/**
 * 测试 instancedetail
 *
 * @throws Exception
 */
@Test
public void testBinlogConsumeWithDataStreamRegisterInstaneDetailTable() throws Exception {
    FlinkCDCMySQLSourceFactory mysqlCDCFactory = new FlinkCDCMySQLSourceFactory();
    mysqlCDCFactory.startupOptions = "latest";
    final String tabName = "instancedetail";
    CUDCDCTestSuit cdcTestSuit = new CUDCDCTestSuit() {

        @Override
        protected BasicDataSourceFactory createDataSourceFactory(TargetResName dataxName) {
            return createMySqlDataSourceFactory(dataxName);
        }

        @Override
        protected String getColEscape() {
            return "`";
        }

        @Override
        protected IResultRows createConsumerHandle(String tabName) {
            return new TestTableRegisterFlinkSourceHandle(tabName, cols);
        }

        @Override
        protected void verfiyTableCrudProcess(String tabName, BasicDataXRdbmsReader dataxReader, ISelectedTab tab, IResultRows consumerHandle, IMQListener<JobExecutionResult> imqListener) throws MQConsumeException, InterruptedException {
            // super.verfiyTableCrudProcess(tabName, dataxReader, tab, consumerHandle, imqListener);
            List<ISelectedTab> tabs = Collections.singletonList(tab);
            List<TestRow> exampleRows = Lists.newArrayList();
            exampleRows.add(this.parseTestRow(RowKind.INSERT, TestFlinkCDCMySQLSourceFactory.class, tabName + "/insert1.txt"));
            Assert.assertEquals(1, exampleRows.size());
            imqListener.start(dataxName, dataxReader, tabs, null);
            Thread.sleep(1000);
            CloseableIterator<Row> snapshot = consumerHandle.getRowSnapshot(tabName);
            BasicDataSourceFactory dataSourceFactory = (BasicDataSourceFactory) dataxReader.getDataSourceFactory();
            Assert.assertNotNull("dataSourceFactory can not be null", dataSourceFactory);
            dataSourceFactory.visitFirstConnection((conn) -> {
                startProcessConn(conn);
                for (TestRow t : exampleRows) {
                    RowVals<Object> vals = t.vals;
                    final String insertBase = "insert into " + createTableName(tabName) + "(" + cols.stream().filter((c) -> vals.notNull(c.getName())).map((col) -> getColEscape() + col.getName() + getColEscape()).collect(Collectors.joining(" , ")) + ") " + "values(" + cols.stream().filter((c) -> vals.notNull(c.getName())).map((col) -> "?").collect(Collectors.joining(" , ")) + ")";
                    PreparedStatement statement = conn.prepareStatement(insertBase);
                    AtomicInteger ci = new AtomicInteger();
                    cols.stream().filter((c) -> vals.notNull(c.getName())).forEach((col) -> {
                        col.type.accept(new DataType.TypeVisitor<Void>() {

                            @Override
                            public Void longType(DataType type) {
                                try {
                                    statement.setLong(ci.incrementAndGet(), Long.parseLong(vals.getString(col.getName())));
                                } catch (SQLException e) {
                                    throw new RuntimeException(e);
                                }
                                return null;
                            }

                            @Override
                            public Void doubleType(DataType type) {
                                try {
                                    statement.setDouble(ci.incrementAndGet(), Double.parseDouble(vals.getString(col.getName())));
                                } catch (SQLException e) {
                                    throw new RuntimeException(e);
                                }
                                return null;
                            }

                            @Override
                            public Void dateType(DataType type) {
                                try {
                                    statement.setDate(ci.incrementAndGet(), java.sql.Date.valueOf(vals.getString(col.getName())));
                                } catch (Exception e) {
                                    throw new RuntimeException(e);
                                }
                                return null;
                            }

                            @Override
                            public Void timestampType(DataType type) {
                                try {
                                    statement.setTimestamp(ci.incrementAndGet(), java.sql.Timestamp.valueOf(vals.getString(col.getName())));
                                } catch (Exception e) {
                                    throw new RuntimeException(e);
                                }
                                return null;
                            }

                            @Override
                            public Void bitType(DataType type) {
                                try {
                                    statement.setByte(ci.incrementAndGet(), Byte.parseByte(vals.getString(col.getName())));
                                } catch (Exception e) {
                                    throw new RuntimeException(e);
                                }
                                return null;
                            }

                            @Override
                            public Void blobType(DataType type) {
                                try {
                                    try (InputStream input = new ByteArrayInputStream(vals.getString(col.getName()).getBytes(TisUTF8.get()))) {
                                        statement.setBlob(ci.incrementAndGet(), input);
                                    }
                                } catch (Exception e) {
                                    throw new RuntimeException(e);
                                }
                                return null;
                            }

                            @Override
                            public Void varcharType(DataType type) {
                                try {
                                    statement.setString(ci.incrementAndGet(), (vals.getString(col.getName())));
                                } catch (Exception e) {
                                    throw new RuntimeException(e);
                                }
                                return null;
                            }

                            @Override
                            public Void intType(DataType type) {
                                try {
                                    statement.setInt(ci.incrementAndGet(), Integer.parseInt(vals.getString(col.getName())));
                                } catch (Exception e) {
                                    throw new RuntimeException(e);
                                }
                                return null;
                            }

                            @Override
                            public Void floatType(DataType type) {
                                try {
                                    statement.setFloat(ci.incrementAndGet(), Float.parseFloat(vals.getString(col.getName())));
                                } catch (Exception e) {
                                    throw new RuntimeException(e);
                                }
                                return null;
                            }

                            @Override
                            public Void decimalType(DataType type) {
                                try {
                                    statement.setBigDecimal(ci.incrementAndGet(), BigDecimal.valueOf(Double.parseDouble(vals.getString(col.getName()))));
                                } catch (Exception e) {
                                    throw new RuntimeException(e);
                                }
                                return null;
                            }

                            @Override
                            public Void timeType(DataType type) {
                                try {
                                    statement.setTime(ci.incrementAndGet(), java.sql.Time.valueOf(vals.getString(col.getName())));
                                } catch (Exception e) {
                                    throw new RuntimeException(e);
                                }
                                return null;
                            }

                            @Override
                            public Void tinyIntType(DataType dataType) {
                                try {
                                    statement.setShort(ci.incrementAndGet(), Short.parseShort(vals.getString(col.getName())));
                                } catch (Exception e) {
                                    throw new RuntimeException(e);
                                }
                                return null;
                            }

                            @Override
                            public Void smallIntType(DataType dataType) {
                                tinyIntType(dataType);
                                return null;
                            }
                        });
                    });
                    Assert.assertEquals(1, executePreparedStatement(conn, statement));
                    statement.close();
                    sleepForAWhile();
                    System.out.println("wait to show insert rows");
                    waitForSnapshotStarted(snapshot);
                    List<TestRow> rows = fetchRows(snapshot, 1, false);
                    for (TestRow rr : rows) {
                        System.out.println("------------" + rr.get("instance_id"));
                        assertTestRow(tabName, RowKind.INSERT, consumerHandle, t, rr);
                    }
                }
            });
        }
    };
    cdcTestSuit.startTest(mysqlCDCFactory, tabName);
}
Also used : TargetResName(com.qlangtech.tis.coredefine.module.action.TargetResName) CUDCDCTestSuit(com.qlangtech.plugins.incr.flink.cdc.CUDCDCTestSuit) IResultRows(com.qlangtech.plugins.incr.flink.cdc.IResultRows) TIS(com.qlangtech.tis.TIS) RowVals(com.qlangtech.plugins.incr.flink.cdc.RowVals) ISelectedTab(com.qlangtech.tis.plugin.ds.ISelectedTab) BasicDataSourceFactory(com.qlangtech.tis.plugin.ds.BasicDataSourceFactory) BigDecimal(java.math.BigDecimal) SQLException(java.sql.SQLException) Lists(com.google.common.collect.Lists) CenterResource(com.qlangtech.tis.manage.common.CenterResource) ByteArrayInputStream(java.io.ByteArrayInputStream) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) TisUTF8(com.qlangtech.tis.manage.common.TisUTF8) TestTableRegisterFlinkSourceHandle(com.qlangtech.plugins.incr.flink.cdc.source.TestTableRegisterFlinkSourceHandle) TestRow(com.qlangtech.plugins.incr.flink.cdc.TestRow) Before(org.junit.Before) MQConsumeException(com.qlangtech.tis.async.message.client.consumer.MQConsumeException) BasicDataXRdbmsReader(com.qlangtech.tis.plugin.datax.common.BasicDataXRdbmsReader) Descriptor(com.qlangtech.tis.extension.Descriptor) Test(org.junit.Test) PreparedStatement(java.sql.PreparedStatement) Collectors(java.util.stream.Collectors) JobExecutionResult(org.apache.flink.api.common.JobExecutionResult) TISEasyMock(com.qlangtech.tis.test.TISEasyMock) CloseableIterator(org.apache.flink.util.CloseableIterator) List(java.util.List) RowKind(org.apache.flink.types.RowKind) DataType(com.qlangtech.tis.plugin.ds.DataType) Row(org.apache.flink.types.Row) Assert(org.junit.Assert) IMQListener(com.qlangtech.tis.async.message.client.consumer.IMQListener) Collections(java.util.Collections) InputStream(java.io.InputStream) BasicDataSourceFactory(com.qlangtech.tis.plugin.ds.BasicDataSourceFactory) SQLException(java.sql.SQLException) ISelectedTab(com.qlangtech.tis.plugin.ds.ISelectedTab) CUDCDCTestSuit(com.qlangtech.plugins.incr.flink.cdc.CUDCDCTestSuit) IResultRows(com.qlangtech.plugins.incr.flink.cdc.IResultRows) IMQListener(com.qlangtech.tis.async.message.client.consumer.IMQListener) TargetResName(com.qlangtech.tis.coredefine.module.action.TargetResName) DataType(com.qlangtech.tis.plugin.ds.DataType) BasicDataXRdbmsReader(com.qlangtech.tis.plugin.datax.common.BasicDataXRdbmsReader) ByteArrayInputStream(java.io.ByteArrayInputStream) InputStream(java.io.InputStream) TestRow(com.qlangtech.plugins.incr.flink.cdc.TestRow) PreparedStatement(java.sql.PreparedStatement) SQLException(java.sql.SQLException) MQConsumeException(com.qlangtech.tis.async.message.client.consumer.MQConsumeException) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) ByteArrayInputStream(java.io.ByteArrayInputStream) TestTableRegisterFlinkSourceHandle(com.qlangtech.plugins.incr.flink.cdc.source.TestTableRegisterFlinkSourceHandle) TestRow(com.qlangtech.plugins.incr.flink.cdc.TestRow) Row(org.apache.flink.types.Row) Test(org.junit.Test)

Aggregations

CUDCDCTestSuit (com.qlangtech.plugins.incr.flink.cdc.CUDCDCTestSuit)5 TargetResName (com.qlangtech.tis.coredefine.module.action.TargetResName)5 Test (org.junit.Test)4 PreparedStatement (java.sql.PreparedStatement)3 TestTableRegisterFlinkSourceHandle (com.qlangtech.plugins.incr.flink.cdc.source.TestTableRegisterFlinkSourceHandle)2 Connection (java.sql.Connection)2 Lists (com.google.common.collect.Lists)1 IResultRows (com.qlangtech.plugins.incr.flink.cdc.IResultRows)1 RowVals (com.qlangtech.plugins.incr.flink.cdc.RowVals)1 TestRow (com.qlangtech.plugins.incr.flink.cdc.TestRow)1 TIS (com.qlangtech.tis.TIS)1 IMQListener (com.qlangtech.tis.async.message.client.consumer.IMQListener)1 MQConsumeException (com.qlangtech.tis.async.message.client.consumer.MQConsumeException)1 Descriptor (com.qlangtech.tis.extension.Descriptor)1 CenterResource (com.qlangtech.tis.manage.common.CenterResource)1 TisUTF8 (com.qlangtech.tis.manage.common.TisUTF8)1 BasicDataXRdbmsReader (com.qlangtech.tis.plugin.datax.common.BasicDataXRdbmsReader)1 BasicDataSourceFactory (com.qlangtech.tis.plugin.ds.BasicDataSourceFactory)1 DataType (com.qlangtech.tis.plugin.ds.DataType)1 ISelectedTab (com.qlangtech.tis.plugin.ds.ISelectedTab)1