Search in sources :

Example 1 with WriteRequest

use of com.amazonaws.services.dynamodbv2.model.WriteRequest in project nifi by apache.

the class AbstractWriteDynamoDBProcessor method handleUnprocessedItems.

/**
 * Helper method to handle unprocessed items items
 * @param session process session
 * @param keysToFlowFileMap map of flow db primary key to flow file
 * @param table dynamodb table
 * @param hashKeyName the hash key name
 * @param hashKeyValueType the hash key value
 * @param rangeKeyName the range key name
 * @param rangeKeyValueType range key value
 * @param outcome the write outcome
 */
protected void handleUnprocessedItems(final ProcessSession session, Map<ItemKeys, FlowFile> keysToFlowFileMap, final String table, final String hashKeyName, final String hashKeyValueType, final String rangeKeyName, final String rangeKeyValueType, BatchWriteItemOutcome outcome) {
    BatchWriteItemResult result = outcome.getBatchWriteItemResult();
    // Handle unprocessed items
    List<WriteRequest> unprocessedItems = result.getUnprocessedItems().get(table);
    if (unprocessedItems != null && unprocessedItems.size() > 0) {
        for (WriteRequest request : unprocessedItems) {
            Map<String, AttributeValue> item = getRequestItem(request);
            Object hashKeyValue = getValue(item, hashKeyName, hashKeyValueType);
            Object rangeKeyValue = getValue(item, rangeKeyName, rangeKeyValueType);
            sendUnprocessedToUnprocessedRelationship(session, keysToFlowFileMap, hashKeyValue, rangeKeyValue);
        }
    }
}
Also used : BatchWriteItemResult(com.amazonaws.services.dynamodbv2.model.BatchWriteItemResult) AttributeValue(com.amazonaws.services.dynamodbv2.model.AttributeValue) WriteRequest(com.amazonaws.services.dynamodbv2.model.WriteRequest)

Example 2 with WriteRequest

use of com.amazonaws.services.dynamodbv2.model.WriteRequest in project nifi by apache.

the class PutDynamoDBTest method testStringHashStringRangePutSuccessfulWithMockOneUnprocessed.

@Test
public void testStringHashStringRangePutSuccessfulWithMockOneUnprocessed() {
    Map<String, List<WriteRequest>> unprocessed = new HashMap<String, List<WriteRequest>>();
    PutRequest put = new PutRequest();
    put.addItemEntry("hashS", new AttributeValue("h1"));
    put.addItemEntry("rangeS", new AttributeValue("r1"));
    WriteRequest write = new WriteRequest(put);
    List<WriteRequest> writes = new ArrayList<>();
    writes.add(write);
    unprocessed.put(stringHashStringRangeTableName, writes);
    result.setUnprocessedItems(unprocessed);
    final TestRunner putRunner = TestRunners.newTestRunner(putDynamoDB);
    putRunner.setProperty(AbstractDynamoDBProcessor.ACCESS_KEY, "abcd");
    putRunner.setProperty(AbstractDynamoDBProcessor.SECRET_KEY, "cdef");
    putRunner.setProperty(AbstractDynamoDBProcessor.REGION, REGION);
    putRunner.setProperty(AbstractDynamoDBProcessor.TABLE, stringHashStringRangeTableName);
    putRunner.setProperty(AbstractDynamoDBProcessor.HASH_KEY_NAME, "hashS");
    putRunner.setProperty(AbstractDynamoDBProcessor.HASH_KEY_VALUE, "h1");
    putRunner.setProperty(AbstractDynamoDBProcessor.RANGE_KEY_NAME, "rangeS");
    putRunner.setProperty(AbstractDynamoDBProcessor.RANGE_KEY_VALUE, "r1");
    putRunner.setProperty(AbstractDynamoDBProcessor.JSON_DOCUMENT, "j2");
    putRunner.enqueue("{\"hello\":\"world\"}".getBytes());
    putRunner.run(1);
    putRunner.assertAllFlowFilesTransferred(AbstractDynamoDBProcessor.REL_UNPROCESSED, 1);
}
Also used : AttributeValue(com.amazonaws.services.dynamodbv2.model.AttributeValue) HashMap(java.util.HashMap) WriteRequest(com.amazonaws.services.dynamodbv2.model.WriteRequest) TestRunner(org.apache.nifi.util.TestRunner) ArrayList(java.util.ArrayList) PutRequest(com.amazonaws.services.dynamodbv2.model.PutRequest) ArrayList(java.util.ArrayList) List(java.util.List) Test(org.junit.Test)

Example 3 with WriteRequest

use of com.amazonaws.services.dynamodbv2.model.WriteRequest in project beam by apache.

the class DynamoDBIOIT method buildWriteRequest.

private static KV<String, WriteRequest> buildWriteRequest(TestRow row) {
    AttributeValue id = new AttributeValue().withN(row.id().toString());
    AttributeValue name = new AttributeValue().withS(row.name());
    PutRequest req = new PutRequest(ImmutableMap.of(COL_ID, id, COL_NAME, name));
    return KV.of(env.options().getDynamoDBTable(), new WriteRequest().withPutRequest(req));
}
Also used : AttributeValue(com.amazonaws.services.dynamodbv2.model.AttributeValue) WriteRequest(com.amazonaws.services.dynamodbv2.model.WriteRequest) PutRequest(com.amazonaws.services.dynamodbv2.model.PutRequest)

Example 4 with WriteRequest

use of com.amazonaws.services.dynamodbv2.model.WriteRequest in project nifi by apache.

the class DeleteDynamoDBTest method setUp.

@Before
public void setUp() {
    outcome = new BatchWriteItemOutcome(result);
    result.setUnprocessedItems(new HashMap<String, List<WriteRequest>>());
    final DynamoDB mockDynamoDB = new DynamoDB(Regions.AP_NORTHEAST_1) {

        @Override
        public BatchWriteItemOutcome batchWriteItem(TableWriteItems... tableWriteItems) {
            return outcome;
        }
    };
    deleteDynamoDB = new DeleteDynamoDB() {

        @Override
        protected DynamoDB getDynamoDB() {
            return mockDynamoDB;
        }
    };
}
Also used : TableWriteItems(com.amazonaws.services.dynamodbv2.document.TableWriteItems) BatchWriteItemOutcome(com.amazonaws.services.dynamodbv2.document.BatchWriteItemOutcome) DynamoDB(com.amazonaws.services.dynamodbv2.document.DynamoDB) ArrayList(java.util.ArrayList) List(java.util.List) Before(org.junit.Before)

Example 5 with WriteRequest

use of com.amazonaws.services.dynamodbv2.model.WriteRequest in project nifi by apache.

the class DeleteDynamoDBTest method testStringHashStringRangeDeleteSuccessfulWithMockOneUnprocessed.

@Test
public void testStringHashStringRangeDeleteSuccessfulWithMockOneUnprocessed() {
    Map<String, List<WriteRequest>> unprocessed = new HashMap<String, List<WriteRequest>>();
    DeleteRequest delete = new DeleteRequest();
    delete.addKeyEntry("hashS", new AttributeValue("h1"));
    delete.addKeyEntry("rangeS", new AttributeValue("r1"));
    WriteRequest write = new WriteRequest(delete);
    List<WriteRequest> writes = new ArrayList<>();
    writes.add(write);
    unprocessed.put(stringHashStringRangeTableName, writes);
    result.setUnprocessedItems(unprocessed);
    final TestRunner deleteRunner = TestRunners.newTestRunner(deleteDynamoDB);
    deleteRunner.setProperty(AbstractDynamoDBProcessor.ACCESS_KEY, "abcd");
    deleteRunner.setProperty(AbstractDynamoDBProcessor.SECRET_KEY, "cdef");
    deleteRunner.setProperty(AbstractDynamoDBProcessor.REGION, REGION);
    deleteRunner.setProperty(AbstractDynamoDBProcessor.TABLE, stringHashStringRangeTableName);
    deleteRunner.setProperty(AbstractDynamoDBProcessor.HASH_KEY_NAME, "hashS");
    deleteRunner.setProperty(AbstractDynamoDBProcessor.HASH_KEY_VALUE, "h1");
    deleteRunner.setProperty(AbstractDynamoDBProcessor.RANGE_KEY_NAME, "rangeS");
    deleteRunner.setProperty(AbstractDynamoDBProcessor.RANGE_KEY_VALUE, "r1");
    deleteRunner.enqueue(new byte[] {});
    deleteRunner.run(1);
    deleteRunner.assertAllFlowFilesTransferred(AbstractDynamoDBProcessor.REL_UNPROCESSED, 1);
}
Also used : AttributeValue(com.amazonaws.services.dynamodbv2.model.AttributeValue) HashMap(java.util.HashMap) WriteRequest(com.amazonaws.services.dynamodbv2.model.WriteRequest) TestRunner(org.apache.nifi.util.TestRunner) ArrayList(java.util.ArrayList) ArrayList(java.util.ArrayList) List(java.util.List) DeleteRequest(com.amazonaws.services.dynamodbv2.model.DeleteRequest) Test(org.junit.Test)

Aggregations

WriteRequest (com.amazonaws.services.dynamodbv2.model.WriteRequest)7 List (java.util.List)7 AttributeValue (com.amazonaws.services.dynamodbv2.model.AttributeValue)6 ArrayList (java.util.ArrayList)6 PutRequest (com.amazonaws.services.dynamodbv2.model.PutRequest)4 HashMap (java.util.HashMap)4 BatchWriteItemOutcome (com.amazonaws.services.dynamodbv2.document.BatchWriteItemOutcome)3 TableWriteItems (com.amazonaws.services.dynamodbv2.document.TableWriteItems)3 BatchWriteItemRequest (com.amazonaws.services.dynamodbv2.model.BatchWriteItemRequest)3 BatchWriteItemResult (com.amazonaws.services.dynamodbv2.model.BatchWriteItemResult)3 DeleteRequest (com.amazonaws.services.dynamodbv2.model.DeleteRequest)3 Test (org.junit.Test)3 AmazonServiceException (com.amazonaws.AmazonServiceException)2 DynamoDB (com.amazonaws.services.dynamodbv2.document.DynamoDB)2 TestRunner (org.apache.nifi.util.TestRunner)2 Before (org.junit.Before)2 Item (com.amazonaws.services.dynamodbv2.document.Item)1 ConsumedCapacity (com.amazonaws.services.dynamodbv2.model.ConsumedCapacity)1 ReturnConsumedCapacity (com.amazonaws.services.dynamodbv2.model.ReturnConsumedCapacity)1 IOException (java.io.IOException)1