Search in sources :

Example 6 with Operation

use of com.hazelcast.jet.cdc.Operation in project hazelcast by hazelcast.

the class PostgresCdcListenBeforeExistsIntegrationTest method pipeline.

private Pipeline pipeline(StreamSource<ChangeRecord> source) {
    Pipeline pipeline = Pipeline.create();
    pipeline.readFrom(source).withNativeTimestamps(0).<ChangeRecord>customTransform("filter_timestamps", filterTimestampsProcessorSupplier()).setLocalParallelism(1).groupingKey(record -> (Integer) record.key().toMap().get("id")).mapStateful(LongAccumulator::new, (accumulator, rowId, record) -> {
        long count = accumulator.get();
        accumulator.add(1);
        Operation operation = record.operation();
        RecordPart value = record.value();
        TableRow row = value.toObject(TableRow.class);
        return entry(rowId + "/" + count, operation + ":" + row);
    }).setLocalParallelism(1).peek().writeTo(Sinks.map(SINK_MAP_NAME));
    return pipeline;
}
Also used : HazelcastInstance(com.hazelcast.core.HazelcastInstance) NightlyTest(com.hazelcast.test.annotation.NightlyTest) Arrays(java.util.Arrays) Connection(java.sql.Connection) Pipeline(com.hazelcast.jet.pipeline.Pipeline) Sinks(com.hazelcast.jet.pipeline.Sinks) Assert.assertTrue(org.junit.Assert.assertTrue) RecordPart(com.hazelcast.jet.cdc.RecordPart) Test(org.junit.Test) StreamSource(com.hazelcast.jet.pipeline.StreamSource) Category(org.junit.experimental.categories.Category) PreparedStatement(java.sql.PreparedStatement) SQLException(java.sql.SQLException) List(java.util.List) Operation(com.hazelcast.jet.cdc.Operation) LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) ResultSet(java.sql.ResultSet) Util.entry(com.hazelcast.jet.Util.entry) Statement(java.sql.Statement) RUNNING(com.hazelcast.jet.core.JobStatus.RUNNING) ChangeRecord(com.hazelcast.jet.cdc.ChangeRecord) JobStatus(com.hazelcast.jet.core.JobStatus) Collections(java.util.Collections) Job(com.hazelcast.jet.Job) RecordPart(com.hazelcast.jet.cdc.RecordPart) Operation(com.hazelcast.jet.cdc.Operation) ChangeRecord(com.hazelcast.jet.cdc.ChangeRecord) Pipeline(com.hazelcast.jet.pipeline.Pipeline)

Example 7 with Operation

use of com.hazelcast.jet.cdc.Operation in project hazelcast by hazelcast.

the class PostgresCdcIntegrationTest method ordersPipeline.

@Nonnull
private Pipeline ordersPipeline() {
    Pipeline pipeline = Pipeline.create();
    pipeline.readFrom(source("orders")).withoutTimestamps().groupingKey(PostgresCdcIntegrationTest::getOrderNumber).mapStateful(LongAccumulator::new, (accumulator, orderId, record) -> {
        long count = accumulator.get();
        accumulator.add(1);
        Operation operation = record.operation();
        RecordPart value = record.value();
        Order order = value.toObject(Order.class);
        return entry(orderId + "/" + count, operation + ":" + order);
    }).setLocalParallelism(1).writeTo(Sinks.map("results"));
    return pipeline;
}
Also used : LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) RecordPart(com.hazelcast.jet.cdc.RecordPart) Operation(com.hazelcast.jet.cdc.Operation) Pipeline(com.hazelcast.jet.pipeline.Pipeline) Nonnull(javax.annotation.Nonnull)

Example 8 with Operation

use of com.hazelcast.jet.cdc.Operation in project hazelcast by hazelcast.

the class MySqlCdcListenBeforeExistIntegrationTest method pipeline.

private Pipeline pipeline(StreamSource<ChangeRecord> source) {
    Pipeline pipeline = Pipeline.create();
    pipeline.readFrom(source).withNativeTimestamps(0).<ChangeRecord>customTransform("filter_timestamps", filterTimestampsProcessorSupplier()).setLocalParallelism(1).groupingKey(record -> (Integer) record.key().toMap().get("id")).mapStateful(LongAccumulator::new, (accumulator, rowId, record) -> {
        long count = accumulator.get();
        accumulator.add(1);
        Operation operation = record.operation();
        RecordPart value = record.value();
        TableRow row = value.toObject(TableRow.class);
        return entry(rowId + "/" + count, operation + ":" + row);
    }).setLocalParallelism(1).peek().writeTo(Sinks.map(SINK_MAP_NAME));
    return pipeline;
}
Also used : HazelcastInstance(com.hazelcast.core.HazelcastInstance) NightlyTest(com.hazelcast.test.annotation.NightlyTest) Arrays(java.util.Arrays) Connection(java.sql.Connection) Pipeline(com.hazelcast.jet.pipeline.Pipeline) Sinks(com.hazelcast.jet.pipeline.Sinks) RecordPart(com.hazelcast.jet.cdc.RecordPart) Test(org.junit.Test) StreamSource(com.hazelcast.jet.pipeline.StreamSource) Category(org.junit.experimental.categories.Category) SQLException(java.sql.SQLException) List(java.util.List) Operation(com.hazelcast.jet.cdc.Operation) LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) Util.entry(com.hazelcast.jet.Util.entry) Statement(java.sql.Statement) RUNNING(com.hazelcast.jet.core.JobStatus.RUNNING) ChangeRecord(com.hazelcast.jet.cdc.ChangeRecord) JobStatus(com.hazelcast.jet.core.JobStatus) Collections(java.util.Collections) Job(com.hazelcast.jet.Job) RecordPart(com.hazelcast.jet.cdc.RecordPart) Operation(com.hazelcast.jet.cdc.Operation) ChangeRecord(com.hazelcast.jet.cdc.ChangeRecord) Pipeline(com.hazelcast.jet.pipeline.Pipeline)

Example 9 with Operation

use of com.hazelcast.jet.cdc.Operation in project hazelcast by hazelcast.

the class MySqlCdcWhiteBlackListIntegrationTest method pipeline.

private Pipeline pipeline(StreamSource<ChangeRecord> source) {
    Pipeline pipeline = Pipeline.create();
    pipeline.readFrom(source).withNativeTimestamps(0).filter(t -> t.database().startsWith(DB_PREFIX)).setLocalParallelism(1).<ChangeRecord>customTransform("filter_timestamps", filterTimestampsProcessorSupplier()).setLocalParallelism(1).groupingKey(record -> (Integer) record.key().toMap().get("id")).mapStateful(LongAccumulator::new, (accumulator, rowId, record) -> {
        long count = accumulator.get();
        accumulator.add(1);
        Operation operation = record.operation();
        RecordPart value = record.value();
        TableRow row = value.toObject(TableRow.class);
        return entry(rowId + "/" + count, operation + ":" + row);
    }).setLocalParallelism(1).writeTo(Sinks.map(SINK_MAP_NAME));
    return pipeline;
}
Also used : HazelcastInstance(com.hazelcast.core.HazelcastInstance) NightlyTest(com.hazelcast.test.annotation.NightlyTest) Arrays(java.util.Arrays) Connection(java.sql.Connection) Pipeline(com.hazelcast.jet.pipeline.Pipeline) Sinks(com.hazelcast.jet.pipeline.Sinks) RecordPart(com.hazelcast.jet.cdc.RecordPart) Test(org.junit.Test) StreamSource(com.hazelcast.jet.pipeline.StreamSource) Category(org.junit.experimental.categories.Category) ArrayList(java.util.ArrayList) SQLException(java.sql.SQLException) List(java.util.List) Operation(com.hazelcast.jet.cdc.Operation) LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) Util.entry(com.hazelcast.jet.Util.entry) Statement(java.sql.Statement) ChangeRecord(com.hazelcast.jet.cdc.ChangeRecord) JobStatus(com.hazelcast.jet.core.JobStatus) Job(com.hazelcast.jet.Job) Before(org.junit.Before) LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) RecordPart(com.hazelcast.jet.cdc.RecordPart) Operation(com.hazelcast.jet.cdc.Operation) Pipeline(com.hazelcast.jet.pipeline.Pipeline)

Aggregations

LongAccumulator (com.hazelcast.jet.accumulator.LongAccumulator)9 Operation (com.hazelcast.jet.cdc.Operation)9 RecordPart (com.hazelcast.jet.cdc.RecordPart)9 Pipeline (com.hazelcast.jet.pipeline.Pipeline)9 HazelcastInstance (com.hazelcast.core.HazelcastInstance)8 Job (com.hazelcast.jet.Job)8 NightlyTest (com.hazelcast.test.annotation.NightlyTest)8 Test (org.junit.Test)8 Category (org.junit.experimental.categories.Category)8 Util.entry (com.hazelcast.jet.Util.entry)7 ChangeRecord (com.hazelcast.jet.cdc.ChangeRecord)7 JobStatus (com.hazelcast.jet.core.JobStatus)7 Sinks (com.hazelcast.jet.pipeline.Sinks)7 StreamSource (com.hazelcast.jet.pipeline.StreamSource)7 Connection (java.sql.Connection)7 Arrays (java.util.Arrays)7 List (java.util.List)7 Statement (java.sql.Statement)6 SQLException (java.sql.SQLException)5 QuickTest (com.hazelcast.test.annotation.QuickTest)4