Examples with SingleShardFanOutKinesisV2 org.apache.flink.streaming.connectors.kinesis.testutils.FakeKinesisFanOutBehavioursFactory.SingleShardFanOutKinesisV2 used on opensource projects

Example 1 with SingleShardFanOutKinesisV2

use of org.apache.flink.streaming.connectors.kinesis.testutils.FakeKinesisFanOutBehavioursFactory.SingleShardFanOutKinesisV2 in project flink by apache.

the class ShardConsumerFanOutTest method testBoundedShardConsumesCorrectNumberOfMessages.

@Test
public void testBoundedShardConsumesCorrectNumberOfMessages() throws Exception {
    SingleShardFanOutKinesisV2 kinesis = FakeKinesisFanOutBehavioursFactory.boundedShard().withBatchCount(10).withRecordsPerBatch(5).build();
    // 10 batches of 5 records = 50
    assertNumberOfMessagesReceivedFromKinesis(50, kinesis, fakeSequenceNumber());
    assertEquals(1, kinesis.getNumberOfSubscribeToShardInvocations());
}

Example 2 with SingleShardFanOutKinesisV2

use of org.apache.flink.streaming.connectors.kinesis.testutils.FakeKinesisFanOutBehavioursFactory.SingleShardFanOutKinesisV2 in project flink by apache.

the class ShardConsumerFanOutTest method testBoundedShardConsumesFromTimestamp.

@Test
public void testBoundedShardConsumesFromTimestamp() throws Exception {
    String format = "yyyy-MM-dd'T'HH:mm";
    String timestamp = "2020-07-02T09:14";
    Instant expectedTimestamp = new SimpleDateFormat(format).parse(timestamp).toInstant();
    SingleShardFanOutKinesisV2 kinesis = FakeKinesisFanOutBehavioursFactory.boundedShard().build();
    Properties consumerConfig = efoProperties();
    consumerConfig.setProperty(STREAM_INITIAL_TIMESTAMP, timestamp);
    consumerConfig.setProperty(STREAM_TIMESTAMP_DATE_FORMAT, format);
    SequenceNumber sequenceNumber = SENTINEL_AT_TIMESTAMP_SEQUENCE_NUM.get();
    // Fake behaviour defaults to 10 messages
    assertNumberOfMessagesReceivedFromKinesis(10, kinesis, sequenceNumber, consumerConfig);
    StartingPosition actual = kinesis.getStartingPositionForSubscription(0);
    assertEquals(AT_TIMESTAMP, actual.type());
    assertEquals(expectedTimestamp, actual.timestamp());
}

Example 3 with SingleShardFanOutKinesisV2

use of org.apache.flink.streaming.connectors.kinesis.testutils.FakeKinesisFanOutBehavioursFactory.SingleShardFanOutKinesisV2 in project flink by apache.

the class ShardConsumerFanOutTest method testBoundedShardWithAggregatedRecords.

@Test
public void testBoundedShardWithAggregatedRecords() throws Exception {
    SingleShardFanOutKinesisV2 kinesis = FakeKinesisFanOutBehavioursFactory.boundedShard().withBatchCount(100).withRecordsPerBatch(10).withAggregationFactor(100).build();
    // 100 batches of 10 records * 100 aggregation factor = 100000
    assertNumberOfMessagesReceivedFromKinesis(100000, kinesis, fakeSequenceNumber());
}

Example 4 with SingleShardFanOutKinesisV2

use of org.apache.flink.streaming.connectors.kinesis.testutils.FakeKinesisFanOutBehavioursFactory.SingleShardFanOutKinesisV2 in project flink by apache.

the class ShardConsumerFanOutTest method testStartFromLatestIsTranslatedToTimestamp.

@Test
public void testStartFromLatestIsTranslatedToTimestamp() throws Exception {
    Instant now = Instant.now();
    SingleShardFanOutKinesisV2 kinesis = FakeKinesisFanOutBehavioursFactory.boundedShard().build();
    SequenceNumber sequenceNumber = SENTINEL_LATEST_SEQUENCE_NUM.get();
    // Fake behaviour defaults to 10 messages
    assertNumberOfMessagesReceivedFromKinesis(10, kinesis, sequenceNumber, efoProperties());
    StartingPosition actual = kinesis.getStartingPositionForSubscription(0);
    assertEquals(AT_TIMESTAMP, actual.type());
    // Considering milliseconds to make now have the same precision as actual
    assertTrue(now.toEpochMilli() <= actual.timestamp().toEpochMilli());
}

Example 5 with SingleShardFanOutKinesisV2

use of org.apache.flink.streaming.connectors.kinesis.testutils.FakeKinesisFanOutBehavioursFactory.SingleShardFanOutKinesisV2 in project flink by apache.

the class ShardConsumerFanOutTest method testEmptyShard.

@Test
public void testEmptyShard() throws Exception {
    SingleShardFanOutKinesisV2 kinesis = FakeKinesisFanOutBehavioursFactory.emptyShard();
    assertNumberOfMessagesReceivedFromKinesis(0, kinesis, fakeSequenceNumber());
    assertEquals(1, kinesis.getNumberOfSubscribeToShardInvocations());
}