Examples with SingleShardFanOutKinesisV2 org.apache.flink.streaming.connectors.kinesis.testutils.FakeKinesisFanOutBehavioursFactory.SingleShardFanOutKinesisV2 used on opensource projects

Example 11 with SingleShardFanOutKinesisV2

use of org.apache.flink.streaming.connectors.kinesis.testutils.FakeKinesisFanOutBehavioursFactory.SingleShardFanOutKinesisV2 in project flink by apache.

the class ShardConsumerFanOutTest method testBoundedShardResumingConsumptionFromAggregatedSubsequenceNumber.

@Test
public void testBoundedShardResumingConsumptionFromAggregatedSubsequenceNumber() throws Exception {
    SingleShardFanOutKinesisV2 kinesis = FakeKinesisFanOutBehavioursFactory.boundedShard().withBatchCount(10).withRecordsPerBatch(1).withAggregationFactor(10).build();
    SequenceNumber subsequenceNumber = new SequenceNumber("1", 5);
    // 10 batches of 1 record * 10 aggregation factor - 6 previously consumed subsequence
    // records (0,1,2,3,4,5) = 94
    assertNumberOfMessagesReceivedFromKinesis(94, kinesis, subsequenceNumber);
    // Starting from aggregated sequence number means we should start AT the sequence number
    assertEquals(AT_SEQUENCE_NUMBER, kinesis.getStartingPositionForSubscription(0).type());
}

Example 12 with SingleShardFanOutKinesisV2

use of org.apache.flink.streaming.connectors.kinesis.testutils.FakeKinesisFanOutBehavioursFactory.SingleShardFanOutKinesisV2 in project flink by apache.

the class FanOutRecordPublisherTest method testAggregatedRecordDurability.

@Test
public void testAggregatedRecordDurability() throws Exception {
    SingleShardFanOutKinesisV2 kinesis = FakeKinesisFanOutBehavioursFactory.boundedShard().withBatchCount(10).withAggregationFactor(5).withRecordsPerBatch(12).build();
    RecordPublisher recordPublisher = createRecordPublisher(kinesis);
    TestConsumer consumer = new TestConsumer();
    int count = 0;
    while (recordPublisher.run(consumer) == INCOMPLETE) {
        if (++count > 5) {
            break;
        }
    }
    List<UserRecord> userRecords = flattenToUserRecords(consumer.getRecordBatches());
    // Should have received 10 * 12 * 5 = 600 records
    assertEquals(600, userRecords.size());
    int sequence = 1;
    long subsequence = 0;
    for (UserRecord userRecord : userRecords) {
        assertEquals(String.valueOf(sequence), userRecord.getSequenceNumber());
        assertEquals(subsequence++, userRecord.getSubSequenceNumber());
        if (subsequence == 5) {
            sequence++;
            subsequence = 0;
        }
    }
}

Example 13 with SingleShardFanOutKinesisV2

use of org.apache.flink.streaming.connectors.kinesis.testutils.FakeKinesisFanOutBehavioursFactory.SingleShardFanOutKinesisV2 in project flink by apache.

the class FanOutRecordPublisherTest method testRecordDurability.

@Test
public void testRecordDurability() throws Exception {
    SingleShardFanOutKinesisV2 kinesis = FakeKinesisFanOutBehavioursFactory.boundedShard().withBatchCount(10).withBatchesPerSubscription(3).withRecordsPerBatch(12).build();
    RecordPublisher recordPublisher = createRecordPublisher(kinesis);
    TestConsumer consumer = new TestConsumer();
    int count = 0;
    while (recordPublisher.run(consumer) == INCOMPLETE) {
        if (++count > 4) {
            break;
        }
    }
    List<UserRecord> userRecords = flattenToUserRecords(consumer.getRecordBatches());
    // Should have received 10 * 12 = 120 records
    assertEquals(120, userRecords.size());
    int expectedSequenceNumber = 1;
    for (UserRecord record : userRecords) {
        assertEquals(String.valueOf(expectedSequenceNumber++), record.getSequenceNumber());
    }
}

Example 14 with SingleShardFanOutKinesisV2

use of org.apache.flink.streaming.connectors.kinesis.testutils.FakeKinesisFanOutBehavioursFactory.SingleShardFanOutKinesisV2 in project flink by apache.

the class FanOutRecordPublisherTest method testToSdkV2StartingPositionAtSequenceNumber.

@Test
public void testToSdkV2StartingPositionAtSequenceNumber() throws Exception {
    SingleShardFanOutKinesisV2 kinesis = emptyShard();
    RecordPublisher publisher = createRecordPublisher(kinesis, StartingPosition.restartFromSequenceNumber(AGGREGATED_SEQUENCE_NUMBER));
    publisher.run(new TestConsumer());
    assertEquals(DUMMY_SEQUENCE, kinesis.getStartingPositionForSubscription(0).sequenceNumber());
    assertEquals(AT_SEQUENCE_NUMBER, kinesis.getStartingPositionForSubscription(0).type());
}

Example 15 with SingleShardFanOutKinesisV2

use of org.apache.flink.streaming.connectors.kinesis.testutils.FakeKinesisFanOutBehavioursFactory.SingleShardFanOutKinesisV2 in project flink by apache.

the class FanOutRecordPublisherTest method testToSdkV2StartingPositionAfterSequenceNumber.

@Test
public void testToSdkV2StartingPositionAfterSequenceNumber() throws Exception {
    SingleShardFanOutKinesisV2 kinesis = emptyShard();
    RecordPublisher publisher = createRecordPublisher(kinesis, StartingPosition.continueFromSequenceNumber(SEQUENCE_NUMBER));
    publisher.run(new TestConsumer());
    assertEquals(DUMMY_SEQUENCE, kinesis.getStartingPositionForSubscription(0).sequenceNumber());
    assertEquals(AFTER_SEQUENCE_NUMBER, kinesis.getStartingPositionForSubscription(0).type());
}