Examples with GetRecordsRequest com.amazonaws.services.kinesis.model.GetRecordsRequest used on opensource projects

Example 6 with GetRecordsRequest

use of com.amazonaws.services.kinesis.model.GetRecordsRequest in project apex-malhar by apache.

the class KinesisUtil method getRecords.

/**
 * Get the records from the particular shard
 * @param streamName Name of the stream from where the records to be accessed
 * @param recordsLimit Number of records to return from shard
 * @param shId Shard Id of the shard
 * @param iteratorType Shard iterator type
 * @param seqNo Record sequence number
 * @return the list of records from the given shard
 * @throws AmazonClientException
 */
public List<Record> getRecords(String streamName, Integer recordsLimit, String shId, ShardIteratorType iteratorType, String seqNo) throws AmazonClientException {
    assert client != null : "Illegal client";
    try {
        // Create the GetShardIteratorRequest instance and sets streamName, shardId and iteratorType to it
        GetShardIteratorRequest iteratorRequest = new GetShardIteratorRequest();
        iteratorRequest.setStreamName(streamName);
        iteratorRequest.setShardId(shId);
        iteratorRequest.setShardIteratorType(iteratorType);
        // If the iteratorType is AFTER_SEQUENCE_NUMBER, set the sequence No to the iteratorRequest
        if (ShardIteratorType.AFTER_SEQUENCE_NUMBER.equals(iteratorType) || ShardIteratorType.AT_SEQUENCE_NUMBER.equals(iteratorType)) {
            iteratorRequest.setStartingSequenceNumber(seqNo);
        }
        // Get the Response from the getShardIterator service method & get the shardIterator from that response
        GetShardIteratorResult iteratorResponse = client.getShardIterator(iteratorRequest);
        // getShardIterator() specifies the position in the shard
        String iterator = iteratorResponse.getShardIterator();
        // Create the GetRecordsRequest instance and set the recordsLimit and iterator
        GetRecordsRequest getRequest = new GetRecordsRequest();
        getRequest.setLimit(recordsLimit);
        getRequest.setShardIterator(iterator);
        // Get the Response from the getRecords service method and get the data records from that response.
        GetRecordsResult getResponse = client.getRecords(getRequest);
        return getResponse.getRecords();
    } catch (AmazonClientException e) {
        throw new RuntimeException(e);
    }
}

Example 7 with GetRecordsRequest

use of com.amazonaws.services.kinesis.model.GetRecordsRequest in project apex-malhar by apache.

the class KinesisTestConsumer method processNextIterator.

public String processNextIterator(String iterator) {
    GetRecordsRequest getRequest = new GetRecordsRequest();
    getRequest.setLimit(1000);
    getRequest.setShardIterator(iterator);
    // call "get" operation and get everything in this shard range
    GetRecordsResult getResponse = client.getRecords(getRequest);
    iterator = getResponse.getNextShardIterator();
    List<Record> records = getResponse.getRecords();
    processResponseRecords(records);
    return iterator;
}

Example 8 with GetRecordsRequest

use of com.amazonaws.services.kinesis.model.GetRecordsRequest in project beam by apache.

the class SimplifiedKinesisClientTest method shouldReturnLimitedNumberOfRecords.

@Test
public void shouldReturnLimitedNumberOfRecords() throws Exception {
    final Integer limit = 100;
    doAnswer((Answer<GetRecordsResult>) invocation -> {
        GetRecordsRequest request = (GetRecordsRequest) invocation.getArguments()[0];
        List<Record> records = generateRecords(request.getLimit());
        return new GetRecordsResult().withRecords(records).withMillisBehindLatest(1000L);
    }).when(kinesis).getRecords(any(GetRecordsRequest.class));
    GetKinesisRecordsResult result = underTest.getRecords(SHARD_ITERATOR, STREAM, SHARD_1, limit);
    assertThat(result.getRecords().size()).isEqualTo(limit);
}

Example 9 with GetRecordsRequest

use of com.amazonaws.services.kinesis.model.GetRecordsRequest in project hazelcast by hazelcast.

the class ShardReader method getRecordsAsync.

private Future<GetRecordsResult> getRecordsAsync(String shardIterator) {
    GetRecordsRequest request = new GetRecordsRequest();
    request.setShardIterator(shardIterator);
    return kinesis.getRecordsAsync(request);
}

Example 10 with GetRecordsRequest

use of com.amazonaws.services.kinesis.model.GetRecordsRequest in project storm by apache.

the class KinesisConnection method fetchRecords.

GetRecordsResult fetchRecords(String shardIterator) {
    GetRecordsRequest getRecordsRequest = new GetRecordsRequest();
    getRecordsRequest.setShardIterator(shardIterator);
    getRecordsRequest.setLimit(kinesisConnectionInfo.getRecordsLimit());
    GetRecordsResult getRecordsResult = kinesisClient.getRecords(getRecordsRequest);
    return getRecordsResult;
}