Search in sources :

Example 1 with KinesisAsyncClient

use of software.amazon.awssdk.services.kinesis.KinesisAsyncClient in project flink by apache.

the class KinesisProxyV2Test method testSubscribeToShard.

@Test
public void testSubscribeToShard() {
    KinesisAsyncClient kinesis = mock(KinesisAsyncClient.class);
    KinesisProxyV2 proxy = new KinesisProxyV2(kinesis, mock(SdkAsyncHttpClient.class), createConfiguration(), mock(FullJitterBackoff.class));
    SubscribeToShardRequest request = SubscribeToShardRequest.builder().build();
    SubscribeToShardResponseHandler responseHandler = SubscribeToShardResponseHandler.builder().subscriber(event -> {
    }).build();
    proxy.subscribeToShard(request, responseHandler);
    verify(kinesis).subscribeToShard(eq(request), eq(responseHandler));
}
Also used : SubscribeToShardResponseHandler(software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler) ArgumentMatchers.any(org.mockito.ArgumentMatchers.any) STREAM_DESCRIBE_RETRIES(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.STREAM_DESCRIBE_RETRIES) SUBSCRIBE_TO_SHARD_BACKOFF_MAX(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.SUBSCRIBE_TO_SHARD_BACKOFF_MAX) ArgumentMatchers.anyLong(org.mockito.ArgumentMatchers.anyLong) ArgumentMatchers.eq(org.mockito.ArgumentMatchers.eq) DESCRIBE_STREAM_CONSUMER_BACKOFF_BASE(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.DESCRIBE_STREAM_CONSUMER_BACKOFF_BASE) DescribeStreamConsumerRequest(software.amazon.awssdk.services.kinesis.model.DescribeStreamConsumerRequest) SubscribeToShardRequest(software.amazon.awssdk.services.kinesis.model.SubscribeToShardRequest) CompletableFuture(java.util.concurrent.CompletableFuture) SUBSCRIBE_TO_SHARD_BACKOFF_EXPONENTIAL_CONSTANT(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.SUBSCRIBE_TO_SHARD_BACKOFF_EXPONENTIAL_CONSTANT) DESCRIBE_STREAM_CONSUMER_BACKOFF_MAX(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.DESCRIBE_STREAM_CONSUMER_BACKOFF_MAX) EFO_CONSUMER_NAME(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.EFO_CONSUMER_NAME) DescribeStreamSummaryRequest(software.amazon.awssdk.services.kinesis.model.DescribeStreamSummaryRequest) ArgumentCaptor(org.mockito.ArgumentCaptor) STREAM_DESCRIBE_BACKOFF_EXPONENTIAL_CONSTANT(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.STREAM_DESCRIBE_BACKOFF_EXPONENTIAL_CONSTANT) DeregisterStreamConsumerRequest(software.amazon.awssdk.services.kinesis.model.DeregisterStreamConsumerRequest) SUBSCRIBE_TO_SHARD_BACKOFF_BASE(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.SUBSCRIBE_TO_SHARD_BACKOFF_BASE) DEREGISTER_STREAM_BACKOFF_BASE(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.DEREGISTER_STREAM_BACKOFF_BASE) SdkAsyncHttpClient(software.amazon.awssdk.http.async.SdkAsyncHttpClient) ExpectedException(org.junit.rules.ExpectedException) RegisterStreamConsumerResponse(software.amazon.awssdk.services.kinesis.model.RegisterStreamConsumerResponse) DESCRIBE_STREAM_CONSUMER_BACKOFF_EXPONENTIAL_CONSTANT(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.DESCRIBE_STREAM_CONSUMER_BACKOFF_EXPONENTIAL_CONSTANT) LimitExceededException(software.amazon.awssdk.services.kinesis.model.LimitExceededException) KinesisAsyncClient(software.amazon.awssdk.services.kinesis.KinesisAsyncClient) RECORD_PUBLISHER_TYPE(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.RECORD_PUBLISHER_TYPE) Properties(java.util.Properties) Collections.emptyList(java.util.Collections.emptyList) REGISTER_STREAM_BACKOFF_BASE(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.REGISTER_STREAM_BACKOFF_BASE) SubscribeToShardResponseHandler(software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler) EFO(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.RecordPublisherType.EFO) Test(org.junit.Test) Mockito.when(org.mockito.Mockito.when) DEREGISTER_STREAM_BACKOFF_EXPONENTIAL_CONSTANT(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.DEREGISTER_STREAM_BACKOFF_EXPONENTIAL_CONSTANT) REGISTER_STREAM_BACKOFF_MAX(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.REGISTER_STREAM_BACKOFF_MAX) DEREGISTER_STREAM_BACKOFF_MAX(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.DEREGISTER_STREAM_BACKOFF_MAX) Mockito.verify(org.mockito.Mockito.verify) STREAM_DESCRIBE_BACKOFF_BASE(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.STREAM_DESCRIBE_BACKOFF_BASE) REGISTER_STREAM_BACKOFF_EXPONENTIAL_CONSTANT(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.REGISTER_STREAM_BACKOFF_EXPONENTIAL_CONSTANT) Rule(org.junit.Rule) DescribeStreamConsumerResponse(software.amazon.awssdk.services.kinesis.model.DescribeStreamConsumerResponse) FanOutRecordPublisherConfiguration(org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisherConfiguration) DeregisterStreamConsumerResponse(software.amazon.awssdk.services.kinesis.model.DeregisterStreamConsumerResponse) STREAM_DESCRIBE_BACKOFF_MAX(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.STREAM_DESCRIBE_BACKOFF_MAX) RegisterStreamConsumerRequest(software.amazon.awssdk.services.kinesis.model.RegisterStreamConsumerRequest) DescribeStreamSummaryResponse(software.amazon.awssdk.services.kinesis.model.DescribeStreamSummaryResponse) Assert.assertEquals(org.junit.Assert.assertEquals) Mockito.mock(org.mockito.Mockito.mock) KinesisAsyncClient(software.amazon.awssdk.services.kinesis.KinesisAsyncClient) SubscribeToShardRequest(software.amazon.awssdk.services.kinesis.model.SubscribeToShardRequest) SdkAsyncHttpClient(software.amazon.awssdk.http.async.SdkAsyncHttpClient) Test(org.junit.Test)

Example 2 with KinesisAsyncClient

use of software.amazon.awssdk.services.kinesis.KinesisAsyncClient in project flink by apache.

the class KinesisProxyV2Test method testDescribeStreamConsumerWithStreamConsumerArn.

@Test
public void testDescribeStreamConsumerWithStreamConsumerArn() throws Exception {
    KinesisAsyncClient client = mock(KinesisAsyncClient.class);
    KinesisProxyV2 proxy = new KinesisProxyV2(client, mock(SdkAsyncHttpClient.class), createConfiguration(), mock(FullJitterBackoff.class));
    DescribeStreamConsumerResponse expected = DescribeStreamConsumerResponse.builder().build();
    ArgumentCaptor<DescribeStreamConsumerRequest> requestCaptor = ArgumentCaptor.forClass(DescribeStreamConsumerRequest.class);
    when(client.describeStreamConsumer(requestCaptor.capture())).thenReturn(CompletableFuture.completedFuture(expected));
    DescribeStreamConsumerResponse actual = proxy.describeStreamConsumer("arn");
    assertEquals(expected, actual);
    DescribeStreamConsumerRequest request = requestCaptor.getValue();
    assertEquals("arn", request.consumerARN());
}
Also used : KinesisAsyncClient(software.amazon.awssdk.services.kinesis.KinesisAsyncClient) DescribeStreamConsumerResponse(software.amazon.awssdk.services.kinesis.model.DescribeStreamConsumerResponse) DescribeStreamConsumerRequest(software.amazon.awssdk.services.kinesis.model.DescribeStreamConsumerRequest) SdkAsyncHttpClient(software.amazon.awssdk.http.async.SdkAsyncHttpClient) Test(org.junit.Test)

Example 3 with KinesisAsyncClient

use of software.amazon.awssdk.services.kinesis.KinesisAsyncClient in project flink by apache.

the class KinesisProxyV2Test method testDescribeStreamSummaryFailsAfterMaxRetries.

@Test
public void testDescribeStreamSummaryFailsAfterMaxRetries() throws Exception {
    exception.expect(RuntimeException.class);
    exception.expectMessage("Retries exceeded - all 10 retry attempts failed.");
    FullJitterBackoff backoff = mock(FullJitterBackoff.class);
    KinesisAsyncClient client = mock(KinesisAsyncClient.class);
    KinesisProxyV2 proxy = new KinesisProxyV2(client, mock(SdkAsyncHttpClient.class), createConfiguration(), backoff);
    when(client.describeStreamSummary(any(DescribeStreamSummaryRequest.class))).thenThrow(new RuntimeException(LimitExceededException.builder().build()));
    proxy.describeStreamSummary("arn");
}
Also used : KinesisAsyncClient(software.amazon.awssdk.services.kinesis.KinesisAsyncClient) DescribeStreamSummaryRequest(software.amazon.awssdk.services.kinesis.model.DescribeStreamSummaryRequest) SdkAsyncHttpClient(software.amazon.awssdk.http.async.SdkAsyncHttpClient) Test(org.junit.Test)

Example 4 with KinesisAsyncClient

use of software.amazon.awssdk.services.kinesis.KinesisAsyncClient in project flink by apache.

the class KinesisProxyV2Test method testDescribeStreamSummary.

@Test
public void testDescribeStreamSummary() throws Exception {
    KinesisAsyncClient client = mock(KinesisAsyncClient.class);
    KinesisProxyV2 proxy = new KinesisProxyV2(client, mock(SdkAsyncHttpClient.class), createConfiguration(), mock(FullJitterBackoff.class));
    DescribeStreamSummaryResponse expected = DescribeStreamSummaryResponse.builder().build();
    ArgumentCaptor<DescribeStreamSummaryRequest> requestCaptor = ArgumentCaptor.forClass(DescribeStreamSummaryRequest.class);
    when(client.describeStreamSummary(requestCaptor.capture())).thenReturn(CompletableFuture.completedFuture(expected));
    DescribeStreamSummaryResponse actual = proxy.describeStreamSummary("stream");
    assertEquals(expected, actual);
    DescribeStreamSummaryRequest request = requestCaptor.getValue();
    assertEquals("stream", request.streamName());
}
Also used : KinesisAsyncClient(software.amazon.awssdk.services.kinesis.KinesisAsyncClient) DescribeStreamSummaryRequest(software.amazon.awssdk.services.kinesis.model.DescribeStreamSummaryRequest) SdkAsyncHttpClient(software.amazon.awssdk.http.async.SdkAsyncHttpClient) DescribeStreamSummaryResponse(software.amazon.awssdk.services.kinesis.model.DescribeStreamSummaryResponse) Test(org.junit.Test)

Example 5 with KinesisAsyncClient

use of software.amazon.awssdk.services.kinesis.KinesisAsyncClient in project flink by apache.

the class KinesisProxyV2Test method testDescribeStreamSummaryBackoffJitter.

@Test
public void testDescribeStreamSummaryBackoffJitter() throws Exception {
    FullJitterBackoff backoff = mock(FullJitterBackoff.class);
    KinesisAsyncClient client = mock(KinesisAsyncClient.class);
    KinesisProxyV2 proxy = new KinesisProxyV2(client, mock(SdkAsyncHttpClient.class), createConfiguration(), backoff);
    when(client.describeStreamSummary(any(DescribeStreamSummaryRequest.class))).thenThrow(new RuntimeException(LimitExceededException.builder().build())).thenReturn(CompletableFuture.completedFuture(DescribeStreamSummaryResponse.builder().build()));
    proxy.describeStreamSummary("arn");
    verify(backoff).sleep(anyLong());
    verify(backoff).calculateFullJitterBackoff(EXPECTED_DESCRIBE_STREAM_BASE, EXPECTED_DESCRIBE_STREAM_MAX, EXPECTED_DESCRIBE_STREAM_POW, 1);
}
Also used : KinesisAsyncClient(software.amazon.awssdk.services.kinesis.KinesisAsyncClient) SdkAsyncHttpClient(software.amazon.awssdk.http.async.SdkAsyncHttpClient) Test(org.junit.Test)

Aggregations

KinesisAsyncClient (software.amazon.awssdk.services.kinesis.KinesisAsyncClient)18 SdkAsyncHttpClient (software.amazon.awssdk.http.async.SdkAsyncHttpClient)14 Test (org.junit.Test)12 SubscribeToShardRequest (software.amazon.awssdk.services.kinesis.model.SubscribeToShardRequest)4 CompletableFuture (java.util.concurrent.CompletableFuture)3 DescribeStreamConsumerRequest (software.amazon.awssdk.services.kinesis.model.DescribeStreamConsumerRequest)3 DescribeStreamConsumerResponse (software.amazon.awssdk.services.kinesis.model.DescribeStreamConsumerResponse)3 DescribeStreamSummaryRequest (software.amazon.awssdk.services.kinesis.model.DescribeStreamSummaryRequest)3 Properties (java.util.Properties)2 AtomicInteger (java.util.concurrent.atomic.AtomicInteger)2 FanOutRecordPublisherConfiguration (org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisherConfiguration)2 Subscriber (org.reactivestreams.Subscriber)2 Subscription (org.reactivestreams.Subscription)2 SdkPublisher (software.amazon.awssdk.core.async.SdkPublisher)2 Region (software.amazon.awssdk.regions.Region)2 DescribeStreamSummaryResponse (software.amazon.awssdk.services.kinesis.model.DescribeStreamSummaryResponse)2 ShardIteratorType (software.amazon.awssdk.services.kinesis.model.ShardIteratorType)2 SubscribeToShardEvent (software.amazon.awssdk.services.kinesis.model.SubscribeToShardEvent)2 SubscribeToShardEventStream (software.amazon.awssdk.services.kinesis.model.SubscribeToShardEventStream)2 SubscribeToShardResponse (software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponse)2