Search in sources :

Example 41 with FailsafeElement

use of com.google.cloud.teleport.v2.values.FailsafeElement in project DataflowTemplates by GoogleCloudPlatform.

the class GCSToSplunkTest method testGCSToSplunkConvertWriteErrors.

@Test
public void testGCSToSplunkConvertWriteErrors() {
    // Arrange
    String stringifiedSplunkError = "Payload: test-payload. Error Message: test-message. Splunk write status code: 123.";
    String firstStringifiedFailsafeError = "Payload: world. Error Message: failed!.";
    String secondStringifiedFailsafeError = "Payload: one. Error Message: error!.";
    CoderRegistry coderRegistry = pipeline.getCoderRegistry();
    coderRegistry.registerCoderForType(FAILSAFE_ELEMENT_CODER.getEncodedTypeDescriptor(), FAILSAFE_ELEMENT_CODER);
    SplunkWriteError splunkWriteError = SplunkWriteError.newBuilder().withPayload("test-payload").withStatusCode(123).withStatusMessage("test-message").create();
    PCollection<SplunkWriteError> splunkErrorCollection = pipeline.apply("Add Splunk Errors", Create.of(splunkWriteError).withCoder(SplunkWriteErrorCoder.of()));
    FailsafeElement<String, String> firstFailsafeElement = FailsafeElement.of("hello", "world").setErrorMessage("failed!");
    PCollection<FailsafeElement<String, String>> firstFailsafeElementCollection = pipeline.apply("Add FailsafeElements to First", Create.of(firstFailsafeElement).withCoder(FAILSAFE_ELEMENT_CODER));
    FailsafeElement<String, String> secondFailsafeElement = FailsafeElement.of("another", "one").setErrorMessage("error!");
    PCollection<FailsafeElement<String, String>> secondFailsafeElementCollection = pipeline.apply("Add FailsafeElements to Second", Create.of(secondFailsafeElement).withCoder(FAILSAFE_ELEMENT_CODER));
    // Act
    PCollectionTuple stringifiedErrors = flattenErrorsAndConvertToString(firstFailsafeElementCollection, secondFailsafeElementCollection, splunkErrorCollection);
    // Assert
    PAssert.that(stringifiedErrors.get(COMBINED_ERRORS)).containsInAnyOrder(stringifiedSplunkError, firstStringifiedFailsafeError, secondStringifiedFailsafeError);
    // Execute pipeline
    pipeline.run();
}
Also used : CoderRegistry(org.apache.beam.sdk.coders.CoderRegistry) PCollectionTuple(org.apache.beam.sdk.values.PCollectionTuple) GCSToSplunk.flattenErrorsAndConvertToString(com.google.cloud.teleport.v2.templates.GCSToSplunk.flattenErrorsAndConvertToString) SplunkWriteError(org.apache.beam.sdk.io.splunk.SplunkWriteError) FailsafeElement(com.google.cloud.teleport.v2.values.FailsafeElement) Test(org.junit.Test)

Example 42 with FailsafeElement

use of com.google.cloud.teleport.v2.values.FailsafeElement in project DataflowTemplates by GoogleCloudPlatform.

the class GCSToSplunkTest method testGCSToSplunkReadJsonSchema.

@Test
public void testGCSToSplunkReadJsonSchema() {
    // Arrange
    String stringifiedJsonRecord = "{\"id\":\"007\",\"state\":\"CA\",\"price\":26.23}";
    SplunkEvent expectedSplunkEvent = SplunkEvent.newBuilder().withEvent(stringifiedJsonRecord).create();
    CoderRegistry coderRegistry = pipeline.getCoderRegistry();
    coderRegistry.registerCoderForClass(SplunkEvent.class, SplunkEventCoder.of());
    coderRegistry.registerCoderForType(FAILSAFE_ELEMENT_CODER.getEncodedTypeDescriptor(), FAILSAFE_ELEMENT_CODER);
    GCSToSplunkOptions options = PipelineOptionsFactory.create().as(GCSToSplunkOptions.class);
    options.setJsonSchemaPath(JSON_SCHEMA_FILE_PATH);
    options.setContainsHeaders(false);
    options.setInputFileSpec(NO_HEADER_CSV_FILE_PATH);
    // Act
    PCollectionTuple readCsvOut = pipeline.apply("Read CSV", readFromCsv(options));
    PCollectionTuple transformedLines = readCsvOut.apply("Convert to JSON", convertToFailsafeAndMaybeApplyUdf(options));
    PCollectionTuple splunkEventTuple = transformedLines.get(UDF_OUT).apply("Convert to Splunk Event", convertToSplunkEvent());
    // Assert
    PAssert.that(transformedLines.get(UDF_OUT)).satisfies(collection -> {
        FailsafeElement element = collection.iterator().next();
        assertThat(element.getPayload()).isEqualTo(stringifiedJsonRecord);
        return null;
    });
    PAssert.that(transformedLines.get(UDF_ERROR_OUT)).empty();
    PAssert.that(splunkEventTuple.get(SPLUNK_EVENT_OUT)).containsInAnyOrder(expectedSplunkEvent);
    PAssert.that(splunkEventTuple.get(SPLUNK_EVENT_ERROR_OUT)).empty();
    // Execute pipeline
    pipeline.run();
}
Also used : GCSToSplunk.convertToSplunkEvent(com.google.cloud.teleport.v2.templates.GCSToSplunk.convertToSplunkEvent) SplunkEvent(org.apache.beam.sdk.io.splunk.SplunkEvent) CoderRegistry(org.apache.beam.sdk.coders.CoderRegistry) PCollectionTuple(org.apache.beam.sdk.values.PCollectionTuple) GCSToSplunkOptions(com.google.cloud.teleport.v2.templates.GCSToSplunk.GCSToSplunkOptions) GCSToSplunk.flattenErrorsAndConvertToString(com.google.cloud.teleport.v2.templates.GCSToSplunk.flattenErrorsAndConvertToString) FailsafeElement(com.google.cloud.teleport.v2.values.FailsafeElement) Test(org.junit.Test)

Aggregations

FailsafeElement (com.google.cloud.teleport.v2.values.FailsafeElement)31 PCollectionTuple (org.apache.beam.sdk.values.PCollectionTuple)26 CoderRegistry (org.apache.beam.sdk.coders.CoderRegistry)21 Test (org.junit.Test)21 Pipeline (org.apache.beam.sdk.Pipeline)14 TableRow (com.google.api.services.bigquery.model.TableRow)8 PubsubMessage (org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage)6 DoFn (org.apache.beam.sdk.transforms.DoFn)6 PubSubToElasticsearchOptions (com.google.cloud.teleport.v2.elasticsearch.options.PubSubToElasticsearchOptions)5 IntegrationTest (com.google.cloud.teleport.v2.spanner.IntegrationTest)5 JSONObject (org.json.JSONObject)5 DeadLetterQueueManager (com.google.cloud.teleport.v2.cdc.dlq.DeadLetterQueueManager)4 StringDeadLetterQueueSanitizer (com.google.cloud.teleport.v2.cdc.dlq.StringDeadLetterQueueSanitizer)4 DataStreamIO (com.google.cloud.teleport.v2.cdc.sources.DataStreamIO)4 FailsafeElementCoder (com.google.cloud.teleport.v2.coders.FailsafeElementCoder)4 GCSToSplunk.flattenErrorsAndConvertToString (com.google.cloud.teleport.v2.templates.GCSToSplunk.flattenErrorsAndConvertToString)4 PipelineResult (org.apache.beam.sdk.PipelineResult)4 SpannerConfig (org.apache.beam.sdk.io.gcp.spanner.SpannerConfig)4 KV (org.apache.beam.sdk.values.KV)4 GCSToElasticsearchOptions (com.google.cloud.teleport.v2.elasticsearch.options.GCSToElasticsearchOptions)3