Search in sources :

Example 6 with ListShardsRequest

use of com.amazonaws.services.kinesis.model.ListShardsRequest in project beam by apache.

the class SimplifiedKinesisClientTest method shouldListAllShardsForTimestampBeforeStreamCreationTimestamp.

@Test
public void shouldListAllShardsForTimestampBeforeStreamCreationTimestamp() throws Exception {
    Shard shard1 = new Shard().withShardId(SHARD_1);
    Shard shard2 = new Shard().withShardId(SHARD_2);
    Shard shard3 = new Shard().withShardId(SHARD_3);
    Instant startingPointTimestamp = Instant.parse("2000-01-01T15:00:00.000Z");
    Instant streamCreationTimestamp = startingPointTimestamp.plus(Duration.standardHours(1));
    DescribeStreamSummaryRequest describeStreamRequest = new DescribeStreamSummaryRequest().withStreamName(STREAM);
    when(kinesis.describeStreamSummary(describeStreamRequest)).thenReturn(new DescribeStreamSummaryResult().withStreamDescriptionSummary(new StreamDescriptionSummary().withStreamCreationTimestamp(streamCreationTimestamp.toDate())));
    ShardFilter shardFilter = new ShardFilter().withType(ShardFilterType.AT_TRIM_HORIZON);
    when(kinesis.listShards(new ListShardsRequest().withStreamName(STREAM).withShardFilter(shardFilter).withMaxResults(1_000))).thenReturn(new ListShardsResult().withShards(shard1, shard2, shard3).withNextToken(null));
    List<Shard> shards = underTest.listShardsAtPoint(STREAM, new StartingPoint(startingPointTimestamp));
    assertThat(shards).containsOnly(shard1, shard2, shard3);
}
Also used : StreamDescriptionSummary(com.amazonaws.services.kinesis.model.StreamDescriptionSummary) ListShardsResult(com.amazonaws.services.kinesis.model.ListShardsResult) ListShardsRequest(com.amazonaws.services.kinesis.model.ListShardsRequest) Instant(org.joda.time.Instant) DescribeStreamSummaryResult(com.amazonaws.services.kinesis.model.DescribeStreamSummaryResult) DescribeStreamSummaryRequest(com.amazonaws.services.kinesis.model.DescribeStreamSummaryRequest) ShardFilter(com.amazonaws.services.kinesis.model.ShardFilter) Shard(com.amazonaws.services.kinesis.model.Shard) Test(org.junit.Test)

Example 7 with ListShardsRequest

use of com.amazonaws.services.kinesis.model.ListShardsRequest in project beam by apache.

the class SimplifiedKinesisClientTest method shouldListAllShardsForLatest.

@Test
public void shouldListAllShardsForLatest() throws Exception {
    Shard shard1 = new Shard().withShardId(SHARD_1);
    Shard shard2 = new Shard().withShardId(SHARD_2);
    Shard shard3 = new Shard().withShardId(SHARD_3);
    when(kinesis.listShards(new ListShardsRequest().withStreamName(STREAM).withShardFilter(new ShardFilter().withType(ShardFilterType.AT_LATEST)).withMaxResults(1_000))).thenReturn(new ListShardsResult().withShards(shard1, shard2, shard3).withNextToken(null));
    List<Shard> shards = underTest.listShardsAtPoint(STREAM, new StartingPoint(InitialPositionInStream.LATEST));
    assertThat(shards).containsOnly(shard1, shard2, shard3);
}
Also used : ListShardsResult(com.amazonaws.services.kinesis.model.ListShardsResult) ListShardsRequest(com.amazonaws.services.kinesis.model.ListShardsRequest) ShardFilter(com.amazonaws.services.kinesis.model.ShardFilter) Shard(com.amazonaws.services.kinesis.model.Shard) Test(org.junit.Test)

Example 8 with ListShardsRequest

use of com.amazonaws.services.kinesis.model.ListShardsRequest in project beam by apache.

the class SimplifiedKinesisClientTest method shouldListAllShardsForTimestampWithinStreamRetentionAfterStreamCreationTimestamp.

@Test
public void shouldListAllShardsForTimestampWithinStreamRetentionAfterStreamCreationTimestamp() throws Exception {
    Shard shard1 = new Shard().withShardId(SHARD_1);
    Shard shard2 = new Shard().withShardId(SHARD_2);
    Shard shard3 = new Shard().withShardId(SHARD_3);
    int hoursDifference = 1;
    int retentionPeriodHours = hoursDifference * 3;
    Instant streamCreationTimestamp = CURRENT_TIMESTAMP.minus(Duration.standardHours(retentionPeriodHours));
    Instant startingPointTimestamp = streamCreationTimestamp.plus(Duration.standardHours(hoursDifference));
    when(currentInstantSupplier.get()).thenReturn(CURRENT_TIMESTAMP);
    when(kinesis.describeStreamSummary(new DescribeStreamSummaryRequest().withStreamName(STREAM))).thenReturn(new DescribeStreamSummaryResult().withStreamDescriptionSummary(new StreamDescriptionSummary().withRetentionPeriodHours(retentionPeriodHours).withStreamCreationTimestamp(streamCreationTimestamp.toDate())));
    ShardFilter shardFilter = new ShardFilter().withType(ShardFilterType.AT_TIMESTAMP).withTimestamp(startingPointTimestamp.toDate());
    when(kinesis.listShards(new ListShardsRequest().withStreamName(STREAM).withShardFilter(shardFilter).withMaxResults(1_000))).thenReturn(new ListShardsResult().withShards(shard1, shard2, shard3).withNextToken(null));
    List<Shard> shards = underTest.listShardsAtPoint(STREAM, new StartingPoint(startingPointTimestamp));
    assertThat(shards).containsOnly(shard1, shard2, shard3);
}
Also used : StreamDescriptionSummary(com.amazonaws.services.kinesis.model.StreamDescriptionSummary) ListShardsResult(com.amazonaws.services.kinesis.model.ListShardsResult) ListShardsRequest(com.amazonaws.services.kinesis.model.ListShardsRequest) Instant(org.joda.time.Instant) DescribeStreamSummaryResult(com.amazonaws.services.kinesis.model.DescribeStreamSummaryResult) DescribeStreamSummaryRequest(com.amazonaws.services.kinesis.model.DescribeStreamSummaryRequest) ShardFilter(com.amazonaws.services.kinesis.model.ShardFilter) Shard(com.amazonaws.services.kinesis.model.Shard) Datapoint(com.amazonaws.services.cloudwatch.model.Datapoint) Test(org.junit.Test)

Example 9 with ListShardsRequest

use of com.amazonaws.services.kinesis.model.ListShardsRequest in project beam by apache.

the class SimplifiedKinesisClientTest method shouldListAllShardsForTrimHorizon.

@Test
public void shouldListAllShardsForTrimHorizon() throws Exception {
    Shard shard1 = new Shard().withShardId(SHARD_1);
    Shard shard2 = new Shard().withShardId(SHARD_2);
    Shard shard3 = new Shard().withShardId(SHARD_3);
    ShardFilter shardFilter = new ShardFilter().withType(ShardFilterType.AT_TRIM_HORIZON);
    when(kinesis.listShards(new ListShardsRequest().withStreamName(STREAM).withShardFilter(shardFilter).withMaxResults(1_000))).thenReturn(new ListShardsResult().withShards(shard1, shard2, shard3).withNextToken(null));
    List<Shard> shards = underTest.listShardsAtPoint(STREAM, new StartingPoint(InitialPositionInStream.TRIM_HORIZON));
    assertThat(shards).containsOnly(shard1, shard2, shard3);
}
Also used : ListShardsResult(com.amazonaws.services.kinesis.model.ListShardsResult) ListShardsRequest(com.amazonaws.services.kinesis.model.ListShardsRequest) ShardFilter(com.amazonaws.services.kinesis.model.ShardFilter) Shard(com.amazonaws.services.kinesis.model.Shard) Test(org.junit.Test)

Example 10 with ListShardsRequest

use of com.amazonaws.services.kinesis.model.ListShardsRequest in project druid by druid-io.

the class KinesisAdminClient method listShards.

private Set<Shard> listShards(String streamName) {
    ListShardsRequest listShardsRequest = new ListShardsRequest().withStreamName(streamName);
    ImmutableSet.Builder<Shard> shards = ImmutableSet.builder();
    while (true) {
        ListShardsResult listShardsResult = amazonKinesis.listShards(listShardsRequest);
        shards.addAll(listShardsResult.getShards());
        String nextToken = listShardsResult.getNextToken();
        if (nextToken == null) {
            return shards.build();
        }
        listShardsRequest = new ListShardsRequest().withNextToken(nextToken);
    }
}
Also used : ListShardsResult(com.amazonaws.services.kinesis.model.ListShardsResult) ListShardsRequest(com.amazonaws.services.kinesis.model.ListShardsRequest) ImmutableSet(com.google.common.collect.ImmutableSet) Shard(com.amazonaws.services.kinesis.model.Shard)

Aggregations

ListShardsRequest (com.amazonaws.services.kinesis.model.ListShardsRequest)18 ListShardsResult (com.amazonaws.services.kinesis.model.ListShardsResult)15 Shard (com.amazonaws.services.kinesis.model.Shard)14 Test (org.junit.Test)10 ShardFilter (com.amazonaws.services.kinesis.model.ShardFilter)9 DescribeStreamSummaryRequest (com.amazonaws.services.kinesis.model.DescribeStreamSummaryRequest)5 DescribeStreamSummaryResult (com.amazonaws.services.kinesis.model.DescribeStreamSummaryResult)5 Instant (org.joda.time.Instant)5 StreamDescriptionSummary (com.amazonaws.services.kinesis.model.StreamDescriptionSummary)4 Datapoint (com.amazonaws.services.cloudwatch.model.Datapoint)3 LimitExceededException (com.amazonaws.services.kinesis.model.LimitExceededException)3 AmazonKinesis (com.amazonaws.services.kinesis.AmazonKinesis)2 ImmutableSet (com.google.common.collect.ImmutableSet)2 AmazonWebServiceRequest (com.amazonaws.AmazonWebServiceRequest)1 ResponseMetadata (com.amazonaws.ResponseMetadata)1 SdkClientException (com.amazonaws.SdkClientException)1 HttpResponse (com.amazonaws.http.HttpResponse)1 SdkHttpMetadata (com.amazonaws.http.SdkHttpMetadata)1 Region (com.amazonaws.regions.Region)1 AmazonCloudWatch (com.amazonaws.services.cloudwatch.AmazonCloudWatch)1