Search in sources :

Example 21 with ProcessContinuation

use of org.apache.beam.sdk.transforms.DoFn.ProcessContinuation in project beam by apache.

the class ReadChangeStreamPartitionDoFnTest method testQueryChangeStreamMode.

@Test
public void testQueryChangeStreamMode() {
    when(queryChangeStreamAction.run(any(), any(), any(), any(), any())).thenReturn(ProcessContinuation.stop());
    final ProcessContinuation result = doFn.processElement(partition, restrictionTracker, outputReceiver, watermarkEstimator, bundleFinalizer);
    assertEquals(ProcessContinuation.stop(), result);
    verify(queryChangeStreamAction).run(partition, restrictionTracker, outputReceiver, watermarkEstimator, bundleFinalizer);
    verify(dataChangeRecordAction, never()).run(any(), any(), any(), any(), any());
    verify(heartbeatRecordAction, never()).run(any(), any(), any(), any());
    verify(childPartitionsRecordAction, never()).run(any(), any(), any(), any());
    verify(restrictionTracker, never()).tryClaim(any());
}
Also used : ProcessContinuation(org.apache.beam.sdk.transforms.DoFn.ProcessContinuation) Test(org.junit.Test)

Example 22 with ProcessContinuation

use of org.apache.beam.sdk.transforms.DoFn.ProcessContinuation in project beam by apache.

the class ReadFromKafkaDoFnTest method testProcessElement.

@Test
public void testProcessElement() throws Exception {
    MockOutputReceiver receiver = new MockOutputReceiver();
    consumer.setNumOfRecordsPerPoll(3L);
    long startOffset = 5L;
    OffsetRangeTracker tracker = new OffsetRangeTracker(new OffsetRange(startOffset, startOffset + 3));
    KafkaSourceDescriptor descriptor = KafkaSourceDescriptor.of(topicPartition, null, null, null, null, null);
    ProcessContinuation result = dofnInstance.processElement(descriptor, tracker, null, (OutputReceiver) receiver);
    assertEquals(ProcessContinuation.stop(), result);
    assertEquals(createExpectedRecords(descriptor, startOffset, 3, "key", "value"), receiver.getOutputs());
}
Also used : OffsetRange(org.apache.beam.sdk.io.range.OffsetRange) OffsetRangeTracker(org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker) ProcessContinuation(org.apache.beam.sdk.transforms.DoFn.ProcessContinuation) Test(org.junit.Test)

Aggregations

ProcessContinuation (org.apache.beam.sdk.transforms.DoFn.ProcessContinuation)22 Test (org.junit.Test)21 Timestamp (com.google.cloud.Timestamp)10 ChildPartitionsRecord (org.apache.beam.sdk.io.gcp.spanner.changestreams.model.ChildPartitionsRecord)8 Instant (org.joda.time.Instant)8 Struct (com.google.cloud.spanner.Struct)6 ChangeStreamResultSet (org.apache.beam.sdk.io.gcp.spanner.changestreams.dao.ChangeStreamResultSet)6 PartitionMetadata (org.apache.beam.sdk.io.gcp.spanner.changestreams.model.PartitionMetadata)6 ChildPartition (org.apache.beam.sdk.io.gcp.spanner.changestreams.model.ChildPartition)5 ChangeStreamResultSetMetadata (org.apache.beam.sdk.io.gcp.spanner.changestreams.dao.ChangeStreamResultSetMetadata)4 DataChangeRecord (org.apache.beam.sdk.io.gcp.spanner.changestreams.model.DataChangeRecord)4 HeartbeatRecord (org.apache.beam.sdk.io.gcp.spanner.changestreams.model.HeartbeatRecord)4 OffsetRange (org.apache.beam.sdk.io.range.OffsetRange)4 OffsetRangeTracker (org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker)4 PollFn (org.apache.beam.sdk.transforms.Watch.Growth.PollFn)2 WatchGrowthFn (org.apache.beam.sdk.transforms.Watch.WatchGrowthFn)2 SpannerException (com.google.cloud.spanner.SpannerException)1 Scope (io.opencensus.common.Scope)1 ChangeStreamRecord (org.apache.beam.sdk.io.gcp.spanner.changestreams.model.ChangeStreamRecord)1 VisibleForTesting (org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting)1