Search in sources :

Example 1 with SQS

use of org.testcontainers.containers.localstack.LocalStackContainer.Service.SQS in project beam by apache.

the class SnsIOIT method testWriteThenRead.

@Test
public void testWriteThenRead() {
    ITOptions opts = env.options();
    int rows = opts.getNumberOfRows();
    // Write test dataset to SNS
    pipelineWrite.apply("Generate Sequence", GenerateSequence.from(0).to(rows)).apply("Prepare TestRows", ParDo.of(new DeterministicallyConstructTestRowFn())).apply("Write to SNS", SnsIO.<TestRow>write().withTopicArn(resources.snsTopic).withPublishRequestBuilder(r -> PublishRequest.builder().message(r.name())));
    // Read test dataset from SQS.
    PCollection<String> output = pipelineRead.apply("Read from SQS", SqsIO.read().withQueueUrl(resources.sqsQueue).withMaxNumRecords(rows)).apply("Extract message", MapElements.into(strings()).via(SnsIOIT::extractMessage));
    PAssert.thatSingleton(output.apply("Count All", Count.globally())).isEqualTo((long) rows);
    PAssert.that(output.apply(Combine.globally(new HashingFn()).withoutDefaults())).containsInAnyOrder(getExpectedHashForRowCount(rows));
    pipelineWrite.run();
    pipelineRead.run();
}
Also used : Count(org.apache.beam.sdk.transforms.Count) Combine(org.apache.beam.sdk.transforms.Combine) RunWith(org.junit.runner.RunWith) PublishRequest(software.amazon.awssdk.services.sns.model.PublishRequest) IOITHelper.executeWithRetry(org.apache.beam.sdk.io.common.IOITHelper.executeWithRetry) TestPipeline(org.apache.beam.sdk.testing.TestPipeline) TypeDescriptors.strings(org.apache.beam.sdk.values.TypeDescriptors.strings) Timeout(org.junit.rules.Timeout) TestRow.getExpectedHashForRowCount(org.apache.beam.sdk.io.common.TestRow.getExpectedHashForRowCount) ClassRule(org.junit.ClassRule) SqsIO(org.apache.beam.sdk.io.aws2.sqs.SqsIO) Service(org.testcontainers.containers.localstack.LocalStackContainer.Service) MapElements(org.apache.beam.sdk.transforms.MapElements) SNS(org.testcontainers.containers.localstack.LocalStackContainer.Service.SNS) HashingFn(org.apache.beam.sdk.io.common.HashingFn) DeterministicallyConstructTestRowFn(org.apache.beam.sdk.io.common.TestRow.DeterministicallyConstructTestRowFn) PAssert(org.apache.beam.sdk.testing.PAssert) TestRow(org.apache.beam.sdk.io.common.TestRow) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) ITEnvironment(org.apache.beam.sdk.io.aws2.ITEnvironment) SqsClient(software.amazon.awssdk.services.sqs.SqsClient) JsonProcessingException(com.fasterxml.jackson.core.JsonProcessingException) GenerateSequence(org.apache.beam.sdk.io.GenerateSequence) Test(org.junit.Test) JUnit4(org.junit.runners.JUnit4) PCollection(org.apache.beam.sdk.values.PCollection) Serializable(java.io.Serializable) Rule(org.junit.Rule) ExternalResource(org.junit.rules.ExternalResource) ParDo(org.apache.beam.sdk.transforms.ParDo) SnsClient(software.amazon.awssdk.services.sns.SnsClient) SqsMessage(org.apache.beam.sdk.io.aws2.sqs.SqsMessage) SQS(org.testcontainers.containers.localstack.LocalStackContainer.Service.SQS) DeterministicallyConstructTestRowFn(org.apache.beam.sdk.io.common.TestRow.DeterministicallyConstructTestRowFn) HashingFn(org.apache.beam.sdk.io.common.HashingFn) Test(org.junit.Test)

Aggregations

JsonProcessingException (com.fasterxml.jackson.core.JsonProcessingException)1 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)1 Serializable (java.io.Serializable)1 GenerateSequence (org.apache.beam.sdk.io.GenerateSequence)1 ITEnvironment (org.apache.beam.sdk.io.aws2.ITEnvironment)1 SqsIO (org.apache.beam.sdk.io.aws2.sqs.SqsIO)1 SqsMessage (org.apache.beam.sdk.io.aws2.sqs.SqsMessage)1 HashingFn (org.apache.beam.sdk.io.common.HashingFn)1 IOITHelper.executeWithRetry (org.apache.beam.sdk.io.common.IOITHelper.executeWithRetry)1 TestRow (org.apache.beam.sdk.io.common.TestRow)1 DeterministicallyConstructTestRowFn (org.apache.beam.sdk.io.common.TestRow.DeterministicallyConstructTestRowFn)1 TestRow.getExpectedHashForRowCount (org.apache.beam.sdk.io.common.TestRow.getExpectedHashForRowCount)1 PAssert (org.apache.beam.sdk.testing.PAssert)1 TestPipeline (org.apache.beam.sdk.testing.TestPipeline)1 Combine (org.apache.beam.sdk.transforms.Combine)1 Count (org.apache.beam.sdk.transforms.Count)1 MapElements (org.apache.beam.sdk.transforms.MapElements)1 ParDo (org.apache.beam.sdk.transforms.ParDo)1 PCollection (org.apache.beam.sdk.values.PCollection)1 TypeDescriptors.strings (org.apache.beam.sdk.values.TypeDescriptors.strings)1