Search in sources :

Example 1 with SubscribeToShardResponseHandler

use of software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler in project flink by apache.

the class KinesisProxyV2Test method testSubscribeToShard.

@Test
public void testSubscribeToShard() {
    KinesisAsyncClient kinesis = mock(KinesisAsyncClient.class);
    KinesisProxyV2 proxy = new KinesisProxyV2(kinesis, mock(SdkAsyncHttpClient.class), createConfiguration(), mock(FullJitterBackoff.class));
    SubscribeToShardRequest request = SubscribeToShardRequest.builder().build();
    SubscribeToShardResponseHandler responseHandler = SubscribeToShardResponseHandler.builder().subscriber(event -> {
    }).build();
    proxy.subscribeToShard(request, responseHandler);
    verify(kinesis).subscribeToShard(eq(request), eq(responseHandler));
}
Also used : SubscribeToShardResponseHandler(software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler) ArgumentMatchers.any(org.mockito.ArgumentMatchers.any) STREAM_DESCRIBE_RETRIES(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.STREAM_DESCRIBE_RETRIES) SUBSCRIBE_TO_SHARD_BACKOFF_MAX(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.SUBSCRIBE_TO_SHARD_BACKOFF_MAX) ArgumentMatchers.anyLong(org.mockito.ArgumentMatchers.anyLong) ArgumentMatchers.eq(org.mockito.ArgumentMatchers.eq) DESCRIBE_STREAM_CONSUMER_BACKOFF_BASE(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.DESCRIBE_STREAM_CONSUMER_BACKOFF_BASE) DescribeStreamConsumerRequest(software.amazon.awssdk.services.kinesis.model.DescribeStreamConsumerRequest) SubscribeToShardRequest(software.amazon.awssdk.services.kinesis.model.SubscribeToShardRequest) CompletableFuture(java.util.concurrent.CompletableFuture) SUBSCRIBE_TO_SHARD_BACKOFF_EXPONENTIAL_CONSTANT(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.SUBSCRIBE_TO_SHARD_BACKOFF_EXPONENTIAL_CONSTANT) DESCRIBE_STREAM_CONSUMER_BACKOFF_MAX(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.DESCRIBE_STREAM_CONSUMER_BACKOFF_MAX) EFO_CONSUMER_NAME(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.EFO_CONSUMER_NAME) DescribeStreamSummaryRequest(software.amazon.awssdk.services.kinesis.model.DescribeStreamSummaryRequest) ArgumentCaptor(org.mockito.ArgumentCaptor) STREAM_DESCRIBE_BACKOFF_EXPONENTIAL_CONSTANT(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.STREAM_DESCRIBE_BACKOFF_EXPONENTIAL_CONSTANT) DeregisterStreamConsumerRequest(software.amazon.awssdk.services.kinesis.model.DeregisterStreamConsumerRequest) SUBSCRIBE_TO_SHARD_BACKOFF_BASE(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.SUBSCRIBE_TO_SHARD_BACKOFF_BASE) DEREGISTER_STREAM_BACKOFF_BASE(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.DEREGISTER_STREAM_BACKOFF_BASE) SdkAsyncHttpClient(software.amazon.awssdk.http.async.SdkAsyncHttpClient) ExpectedException(org.junit.rules.ExpectedException) RegisterStreamConsumerResponse(software.amazon.awssdk.services.kinesis.model.RegisterStreamConsumerResponse) DESCRIBE_STREAM_CONSUMER_BACKOFF_EXPONENTIAL_CONSTANT(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.DESCRIBE_STREAM_CONSUMER_BACKOFF_EXPONENTIAL_CONSTANT) LimitExceededException(software.amazon.awssdk.services.kinesis.model.LimitExceededException) KinesisAsyncClient(software.amazon.awssdk.services.kinesis.KinesisAsyncClient) RECORD_PUBLISHER_TYPE(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.RECORD_PUBLISHER_TYPE) Properties(java.util.Properties) Collections.emptyList(java.util.Collections.emptyList) REGISTER_STREAM_BACKOFF_BASE(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.REGISTER_STREAM_BACKOFF_BASE) SubscribeToShardResponseHandler(software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler) EFO(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.RecordPublisherType.EFO) Test(org.junit.Test) Mockito.when(org.mockito.Mockito.when) DEREGISTER_STREAM_BACKOFF_EXPONENTIAL_CONSTANT(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.DEREGISTER_STREAM_BACKOFF_EXPONENTIAL_CONSTANT) REGISTER_STREAM_BACKOFF_MAX(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.REGISTER_STREAM_BACKOFF_MAX) DEREGISTER_STREAM_BACKOFF_MAX(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.DEREGISTER_STREAM_BACKOFF_MAX) Mockito.verify(org.mockito.Mockito.verify) STREAM_DESCRIBE_BACKOFF_BASE(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.STREAM_DESCRIBE_BACKOFF_BASE) REGISTER_STREAM_BACKOFF_EXPONENTIAL_CONSTANT(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.REGISTER_STREAM_BACKOFF_EXPONENTIAL_CONSTANT) Rule(org.junit.Rule) DescribeStreamConsumerResponse(software.amazon.awssdk.services.kinesis.model.DescribeStreamConsumerResponse) FanOutRecordPublisherConfiguration(org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisherConfiguration) DeregisterStreamConsumerResponse(software.amazon.awssdk.services.kinesis.model.DeregisterStreamConsumerResponse) STREAM_DESCRIBE_BACKOFF_MAX(org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.STREAM_DESCRIBE_BACKOFF_MAX) RegisterStreamConsumerRequest(software.amazon.awssdk.services.kinesis.model.RegisterStreamConsumerRequest) DescribeStreamSummaryResponse(software.amazon.awssdk.services.kinesis.model.DescribeStreamSummaryResponse) Assert.assertEquals(org.junit.Assert.assertEquals) Mockito.mock(org.mockito.Mockito.mock) KinesisAsyncClient(software.amazon.awssdk.services.kinesis.KinesisAsyncClient) SubscribeToShardRequest(software.amazon.awssdk.services.kinesis.model.SubscribeToShardRequest) SdkAsyncHttpClient(software.amazon.awssdk.http.async.SdkAsyncHttpClient) Test(org.junit.Test)

Example 2 with SubscribeToShardResponseHandler

use of software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler in project aws-doc-sdk-examples by awsdocs.

the class KinesisStreamEx method responseHandlerBuilderVisitor.

// snippet-end:[kinesis.java2.stream_example.visitor]
/**
 * Subscribes to the stream of events by implementing the SubscribeToShardResponseHandler.Visitor interface.
 */
private static CompletableFuture<Void> responseHandlerBuilderVisitor(KinesisAsyncClient client, SubscribeToShardRequest request) {
    SubscribeToShardResponseHandler.Visitor visitor = new SubscribeToShardResponseHandler.Visitor() {

        @Override
        public void visit(SubscribeToShardEvent event) {
            System.out.println("Received subscribe to shard event " + event);
        }
    };
    SubscribeToShardResponseHandler responseHandler = SubscribeToShardResponseHandler.builder().onError(t -> System.err.println("Error during stream - " + t.getMessage())).subscriber(visitor).build();
    return client.subscribeToShard(request, responseHandler);
}
Also used : SubscribeToShardResponseHandler(software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler) SubscribeToShardEvent(software.amazon.awssdk.services.kinesis.model.SubscribeToShardEvent)

Example 3 with SubscribeToShardResponseHandler

use of software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler in project aws-doc-sdk-examples by awsdocs.

the class KinesisStreamEx method main.

public static void main(String[] args) {
    // snippet-start:[kinesis.java2.stream_example.setup]
    Region region = Region.US_EAST_1;
    KinesisAsyncClient client = KinesisAsyncClient.builder().region(region).build();
    SubscribeToShardRequest request = SubscribeToShardRequest.builder().consumerARN(CONSUMER_ARN).shardId("arn:aws:kinesis:us-east-1:111122223333:stream/StockTradeStream").startingPosition(s -> s.type(ShardIteratorType.LATEST)).build();
    // snippet-end:[kinesis.java2.stream_example.setup]
    SubscribeToShardResponseHandler responseHandler = SubscribeToShardResponseHandler.builder().onError(t -> System.err.println("Error during stream - " + t.getMessage())).subscriber(MySubscriber::new).build();
    client.subscribeToShard(request, responseHandler);
    client.close();
}
Also used : KinesisAsyncClient(software.amazon.awssdk.services.kinesis.KinesisAsyncClient) ShardIteratorType(software.amazon.awssdk.services.kinesis.model.ShardIteratorType) SubscribeToShardResponseHandler(software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler) SubscribeToShardEventStream(software.amazon.awssdk.services.kinesis.model.SubscribeToShardEventStream) CompletableFuture(java.util.concurrent.CompletableFuture) SdkPublisher(software.amazon.awssdk.core.async.SdkPublisher) SubscribeToShardRequest(software.amazon.awssdk.services.kinesis.model.SubscribeToShardRequest) SubscribeToShardResponse(software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponse) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) Subscription(org.reactivestreams.Subscription) Region(software.amazon.awssdk.regions.Region) SubscribeToShardEvent(software.amazon.awssdk.services.kinesis.model.SubscribeToShardEvent) Subscriber(org.reactivestreams.Subscriber) SubscribeToShardResponseHandler(software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler) KinesisAsyncClient(software.amazon.awssdk.services.kinesis.KinesisAsyncClient) SubscribeToShardRequest(software.amazon.awssdk.services.kinesis.model.SubscribeToShardRequest) Region(software.amazon.awssdk.regions.Region)

Example 4 with SubscribeToShardResponseHandler

use of software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler in project aws-doc-sdk-examples by awsdocs.

the class KinesisStreamEx method responseHandlerBuilderVisitorBuilder.

// snippet-end:[kinesis.java2.stream_example.publish_transformer]
/**
 * Creates a SubscribeToShardResponseHandler.Visitor using the builder which lets you register an event handler for
 * all events you're interested in rather than implementing the interface.
 */
// snippet-start:[kinesis.java2.stream_example.visitor]
private static CompletableFuture<Void> responseHandlerBuilderVisitorBuilder(KinesisAsyncClient client, SubscribeToShardRequest request) {
    SubscribeToShardResponseHandler.Visitor visitor = SubscribeToShardResponseHandler.Visitor.builder().onSubscribeToShardEvent(e -> System.out.println("Received subscribe to shard event " + e)).build();
    SubscribeToShardResponseHandler responseHandler = SubscribeToShardResponseHandler.builder().onError(t -> System.err.println("Error during stream - " + t.getMessage())).subscriber(visitor).build();
    return client.subscribeToShard(request, responseHandler);
}
Also used : SubscribeToShardResponseHandler(software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler) KinesisAsyncClient(software.amazon.awssdk.services.kinesis.KinesisAsyncClient) ShardIteratorType(software.amazon.awssdk.services.kinesis.model.ShardIteratorType) SubscribeToShardResponseHandler(software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler) SubscribeToShardEventStream(software.amazon.awssdk.services.kinesis.model.SubscribeToShardEventStream) CompletableFuture(java.util.concurrent.CompletableFuture) SdkPublisher(software.amazon.awssdk.core.async.SdkPublisher) SubscribeToShardRequest(software.amazon.awssdk.services.kinesis.model.SubscribeToShardRequest) SubscribeToShardResponse(software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponse) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) Subscription(org.reactivestreams.Subscription) Region(software.amazon.awssdk.regions.Region) SubscribeToShardEvent(software.amazon.awssdk.services.kinesis.model.SubscribeToShardEvent) Subscriber(org.reactivestreams.Subscriber)

Example 5 with SubscribeToShardResponseHandler

use of software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler in project flink by apache.

the class FanOutShardSubscriber method openSubscriptionToShard.

/**
 * Calls {@link KinesisProxyV2#subscribeToShard} and waits to acquire a subscription. In the
 * event a non-recoverable error occurs this method will rethrow the exception. Once the
 * subscription is acquired the client signals to the producer that we are ready to receive
 * records.
 *
 * @param startingPosition the position in which to start consuming from
 * @throws FanOutSubscriberException when an exception is propagated from the networking stack
 */
private FanOutShardSubscription openSubscriptionToShard(final StartingPosition startingPosition) throws FanOutSubscriberException, InterruptedException {
    SubscribeToShardRequest request = SubscribeToShardRequest.builder().consumerARN(consumerArn).shardId(shardId).startingPosition(startingPosition).build();
    AtomicReference<Throwable> exception = new AtomicReference<>();
    CountDownLatch waitForSubscriptionLatch = new CountDownLatch(1);
    FanOutShardSubscription subscription = new FanOutShardSubscription(waitForSubscriptionLatch);
    SubscribeToShardResponseHandler responseHandler = SubscribeToShardResponseHandler.builder().onError(e -> {
        // (errors are ignored here once the subscription is open)
        if (waitForSubscriptionLatch.getCount() > 0) {
            exception.set(e);
            waitForSubscriptionLatch.countDown();
        }
    }).subscriber(() -> subscription).build();
    kinesis.subscribeToShard(request, responseHandler);
    boolean subscriptionEstablished = waitForSubscriptionLatch.await(subscribeToShardTimeout.toMillis(), TimeUnit.MILLISECONDS);
    if (!subscriptionEstablished) {
        final String errorMessage = "Timed out acquiring subscription - " + shardId + " (" + consumerArn + ")";
        LOG.error(errorMessage);
        subscription.cancelSubscription();
        handleError(new RecoverableFanOutSubscriberException(new TimeoutException(errorMessage)));
    }
    Throwable throwable = exception.get();
    if (throwable != null) {
        handleError(throwable);
    }
    LOG.debug("Acquired subscription - {} ({})", shardId, consumerArn);
    // Request the first record to kick off consumption
    // Following requests are made by the FanOutShardSubscription on the netty thread
    subscription.requestRecord();
    return subscription;
}
Also used : SubscribeToShardResponseHandler(software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler) SubscribeToShardRequest(software.amazon.awssdk.services.kinesis.model.SubscribeToShardRequest) AtomicReference(java.util.concurrent.atomic.AtomicReference) CountDownLatch(java.util.concurrent.CountDownLatch) TimeoutException(java.util.concurrent.TimeoutException) ReadTimeoutException(io.netty.handler.timeout.ReadTimeoutException)

Aggregations

SubscribeToShardResponseHandler (software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponseHandler)5 CompletableFuture (java.util.concurrent.CompletableFuture)3 KinesisAsyncClient (software.amazon.awssdk.services.kinesis.KinesisAsyncClient)3 SubscribeToShardRequest (software.amazon.awssdk.services.kinesis.model.SubscribeToShardRequest)3 AtomicInteger (java.util.concurrent.atomic.AtomicInteger)2 Subscriber (org.reactivestreams.Subscriber)2 Subscription (org.reactivestreams.Subscription)2 SdkPublisher (software.amazon.awssdk.core.async.SdkPublisher)2 Region (software.amazon.awssdk.regions.Region)2 ShardIteratorType (software.amazon.awssdk.services.kinesis.model.ShardIteratorType)2 SubscribeToShardEvent (software.amazon.awssdk.services.kinesis.model.SubscribeToShardEvent)2 SubscribeToShardEventStream (software.amazon.awssdk.services.kinesis.model.SubscribeToShardEventStream)2 SubscribeToShardResponse (software.amazon.awssdk.services.kinesis.model.SubscribeToShardResponse)2 ReadTimeoutException (io.netty.handler.timeout.ReadTimeoutException)1 Collections.emptyList (java.util.Collections.emptyList)1 Properties (java.util.Properties)1 CountDownLatch (java.util.concurrent.CountDownLatch)1 TimeoutException (java.util.concurrent.TimeoutException)1 AtomicReference (java.util.concurrent.atomic.AtomicReference)1 DEREGISTER_STREAM_BACKOFF_BASE (org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.DEREGISTER_STREAM_BACKOFF_BASE)1