Search in sources :

Example 16 with S3EventNotification

use of com.amazonaws.services.s3.event.S3EventNotification in project bender by Nextdoor.

the class S3HandlerTest method testSourceRegex.

@Test
public void testSourceRegex() throws Throwable {
    BaseHandler.CONFIG_FILE = "/com/nextdoor/bender/handler/config_s3_source.json";
    TestContext ctx = new TestContext();
    ctx.setFunctionName("unittest");
    ctx.setInvokedFunctionArn("arn:aws:lambda:us-east-1:123:function:test-function:staging");
    BaseHandler<S3EventNotification> handler = (BaseHandler) getHandler();
    handler.init(ctx);
    handler.handler(getTestEvent(), ctx);
    assertEquals(1, DummyTransportHelper.BufferedTransporter.output.size());
}
Also used : S3EventNotification(com.amazonaws.services.s3.event.S3EventNotification) TestContext(com.nextdoor.bender.aws.TestContext) BaseHandler(com.nextdoor.bender.handler.BaseHandler) HandlerTest(com.nextdoor.bender.handler.HandlerTest) Test(org.junit.Test)

Example 17 with S3EventNotification

use of com.amazonaws.services.s3.event.S3EventNotification in project bender by Nextdoor.

the class S3HandlerTest method getTestEvent.

private S3EventNotification getTestEvent(String bucket, boolean doPut) throws Exception {
    /*
     * Upload a test resoruce to the mock S3
     */
    if (doPut) {
        String payload = IOUtils.toString(new InputStreamReader(this.getClass().getResourceAsStream("basic_input.log"), "UTF-8"));
        this.client.putObject(bucket, "basic_input.log", payload);
    }
    /*
     * Create a S3EventNotification event
     */
    S3ObjectEntity objEntity = new S3ObjectEntity("basic_input.log", 1L, null, null);
    S3BucketEntity bucketEntity = new S3BucketEntity(bucket, null, null);
    S3Entity entity = new S3Entity(null, bucketEntity, objEntity, null);
    S3EventNotificationRecord rec = new S3EventNotificationRecord(null, null, null, "1970-01-01T00:00:00.000Z", null, null, null, entity, null);
    List<S3EventNotificationRecord> notifications = new ArrayList<S3EventNotificationRecord>(2);
    notifications.add(rec);
    return new S3EventNotification(notifications);
}
Also used : S3EventNotification(com.amazonaws.services.s3.event.S3EventNotification) S3ObjectEntity(com.amazonaws.services.s3.event.S3EventNotification.S3ObjectEntity) S3Entity(com.amazonaws.services.s3.event.S3EventNotification.S3Entity) InputStreamReader(java.io.InputStreamReader) S3EventNotificationRecord(com.amazonaws.services.s3.event.S3EventNotification.S3EventNotificationRecord) ArrayList(java.util.ArrayList) S3BucketEntity(com.amazonaws.services.s3.event.S3EventNotification.S3BucketEntity)

Example 18 with S3EventNotification

use of com.amazonaws.services.s3.event.S3EventNotification in project bender by Nextdoor.

the class S3Handler method handler.

public void handler(S3EventNotification event, Context context) throws HandlerException {
    if (!initialized) {
        init(context);
        S3HandlerConfig handlerConfig = (S3HandlerConfig) this.config.getHandlerConfig();
        this.logTrigger = handlerConfig.getLogS3Trigger();
    }
    if (this.logTrigger) {
        logger.info("trigger: " + gson.toJson(event));
    }
    /*
     * Validate the S3 file matches the regex
     */
    List<S3EventNotificationRecord> toProcess = new ArrayList<S3EventNotificationRecord>(event.getRecords());
    for (S3EventNotificationRecord record : event.getRecords()) {
        String s3Path = String.format("s3://%s/%s", record.getS3().getBucket().getName(), record.getS3().getObject().getKey());
        try {
            this.source = SourceUtils.getSource(s3Path, this.sources);
        } catch (SourceNotFoundException e) {
            logger.warn("Skipping processing " + s3Path);
            toProcess.remove(record);
        }
    }
    if (toProcess.size() == 0) {
        logger.warn("Nothing to process");
        return;
    }
    this.recordIterator = new S3EventIterator(context, toProcess, s3ClientFactory);
    super.process(context);
}
Also used : SourceNotFoundException(com.nextdoor.bender.utils.SourceUtils.SourceNotFoundException) S3EventNotificationRecord(com.amazonaws.services.s3.event.S3EventNotification.S3EventNotificationRecord) ArrayList(java.util.ArrayList)

Aggregations

S3EventNotification (com.amazonaws.services.s3.event.S3EventNotification)17 S3EventNotificationRecord (com.amazonaws.services.s3.event.S3EventNotification.S3EventNotificationRecord)13 ArrayList (java.util.ArrayList)13 S3Entity (com.amazonaws.services.s3.event.S3EventNotification.S3Entity)11 S3ObjectEntity (com.amazonaws.services.s3.event.S3EventNotification.S3ObjectEntity)11 Test (org.junit.Test)8 AbstractServiceTest (org.finra.herd.service.AbstractServiceTest)6 S3BucketEntity (com.amazonaws.services.s3.event.S3EventNotification.S3BucketEntity)5 InputStreamReader (java.io.InputStreamReader)4 SNSRecord (com.amazonaws.services.lambda.runtime.events.SNSEvent.SNSRecord)3 Attribute (org.finra.herd.model.api.xml.Attribute)3 BusinessObjectDefinitionCreateRequest (org.finra.herd.model.api.xml.BusinessObjectDefinitionCreateRequest)3 BusinessObjectDefinitionKey (org.finra.herd.model.api.xml.BusinessObjectDefinitionKey)3 SNSEvent (com.amazonaws.services.lambda.runtime.events.SNSEvent)2 TestContext (com.nextdoor.bender.aws.TestContext)2 BaseHandler (com.nextdoor.bender.handler.BaseHandler)2 HandlerTest (com.nextdoor.bender.handler.HandlerTest)2 SourceNotFoundException (com.nextdoor.bender.utils.SourceUtils.SourceNotFoundException)2 BusinessObjectDefinition (org.finra.herd.model.api.xml.BusinessObjectDefinition)2 SampleDataFile (org.finra.herd.model.api.xml.SampleDataFile)2