Search in sources :

Example 1 with ChangeEventContext

use of com.google.cloud.teleport.v2.templates.datastream.ChangeEventContext in project DataflowTemplates by GoogleCloudPlatform.

the class SpannerTransactionWriterDoFn method processElement.

@ProcessElement
public void processElement(ProcessContext c) {
    FailsafeElement<String, String> msg = c.element();
    Ddl ddl = c.sideInput(ddlView);
    processedEvents.inc();
    /*
     * Try Catch block to capture any exceptions that might occur while processing
     * DataStream events while writing to Cloud Spanner. All Exceptions that are caught
     * can be retried based on the exception type.
     */
    try {
        JsonNode changeEvent = mapper.readTree(msg.getPayload());
        ChangeEventContext changeEventContext = ChangeEventContextFactory.createChangeEventContext(changeEvent, ddl, shadowTablePrefix, sourceType);
        // Sequence information for the current change event.
        ChangeEventSequence currentChangeEventSequence = ChangeEventSequenceFactory.createChangeEventSequenceFromChangeEventContext(changeEventContext);
        // Start transaction
        spannerAccessor.getDatabaseClient().readWriteTransaction().run((TransactionCallable<Void>) transaction -> {
            ChangeEventSequence previousChangeEventSequence = ChangeEventSequenceFactory.createChangeEventSequenceFromShadowTable(transaction, changeEventContext);
            if (previousChangeEventSequence != null && previousChangeEventSequence.compareTo(currentChangeEventSequence) >= 0) {
                return null;
            }
            transaction.buffer(changeEventContext.getMutations());
            return null;
        });
        com.google.cloud.Timestamp timestamp = com.google.cloud.Timestamp.now();
        c.output(timestamp);
        sucessfulEvents.inc();
    } catch (InvalidChangeEventException e) {
        // Errors that result from invalid change events.
        outputWithErrorTag(c, msg, e, SpannerTransactionWriter.PERMANENT_ERROR_TAG);
        skippedEvents.inc();
    } catch (ChangeEventConvertorException e) {
        // Errors that result during Event conversions are not retryable.
        outputWithErrorTag(c, msg, e, SpannerTransactionWriter.PERMANENT_ERROR_TAG);
        conversionErrors.inc();
    } catch (SpannerException se) {
        /* Errors that happen when writing to Cloud Spanner are considered retryable.
       * Since all event convertion errors are caught beforehand as permanent errors,
       * any other errors encountered while writing to Cloud Spanner can be retried.
       * Examples include:
       * 1. Deadline exceeded errors from Cloud Spanner.
       * 2. Failures due to foreign key/interleaved table constraints.
       * 3. Any transient errors in Cloud Spanner.
       */
        outputWithErrorTag(c, msg, se, SpannerTransactionWriter.RETRYABLE_ERROR_TAG);
        retryableErrors.inc();
    } catch (Exception e) {
        // Any other errors are considered severe and not retryable.
        outputWithErrorTag(c, msg, e, SpannerTransactionWriter.PERMANENT_ERROR_TAG);
        failedEvents.inc();
    }
}
Also used : TransactionCallable(com.google.cloud.spanner.TransactionRunner.TransactionCallable) InvalidChangeEventException(com.google.cloud.teleport.v2.templates.datastream.InvalidChangeEventException) Ddl(com.google.cloud.teleport.v2.templates.spanner.ddl.Ddl) LoggerFactory(org.slf4j.LoggerFactory) Timestamp(com.google.cloud.Timestamp) DeserializationFeature(com.fasterxml.jackson.databind.DeserializationFeature) Metrics(org.apache.beam.sdk.metrics.Metrics) ChangeEventConvertorException(com.google.cloud.teleport.v2.templates.datastream.ChangeEventConvertorException) TupleTag(org.apache.beam.sdk.values.TupleTag) ChangeEventSequenceFactory(com.google.cloud.teleport.v2.templates.datastream.ChangeEventSequenceFactory) JsonNode(com.fasterxml.jackson.databind.JsonNode) ExposedSpannerAccessor(org.apache.beam.sdk.io.gcp.spanner.ExposedSpannerAccessor) ChangeEventContext(com.google.cloud.teleport.v2.templates.datastream.ChangeEventContext) ChangeEventSequence(com.google.cloud.teleport.v2.templates.datastream.ChangeEventSequence) PrintWriter(java.io.PrintWriter) DoFn(org.apache.beam.sdk.transforms.DoFn) Logger(org.slf4j.Logger) StringWriter(java.io.StringWriter) Counter(org.apache.beam.sdk.metrics.Counter) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) Serializable(java.io.Serializable) SpannerConfig(org.apache.beam.sdk.io.gcp.spanner.SpannerConfig) SpannerException(com.google.cloud.spanner.SpannerException) PCollectionView(org.apache.beam.sdk.values.PCollectionView) FailsafeElement(com.google.cloud.teleport.v2.values.FailsafeElement) Preconditions(com.google.common.base.Preconditions) ChangeEventContextFactory(com.google.cloud.teleport.v2.templates.datastream.ChangeEventContextFactory) JsonNode(com.fasterxml.jackson.databind.JsonNode) Timestamp(com.google.cloud.Timestamp) Ddl(com.google.cloud.teleport.v2.templates.spanner.ddl.Ddl) InvalidChangeEventException(com.google.cloud.teleport.v2.templates.datastream.InvalidChangeEventException) ChangeEventConvertorException(com.google.cloud.teleport.v2.templates.datastream.ChangeEventConvertorException) SpannerException(com.google.cloud.spanner.SpannerException) ChangeEventConvertorException(com.google.cloud.teleport.v2.templates.datastream.ChangeEventConvertorException) InvalidChangeEventException(com.google.cloud.teleport.v2.templates.datastream.InvalidChangeEventException) ChangeEventContext(com.google.cloud.teleport.v2.templates.datastream.ChangeEventContext) SpannerException(com.google.cloud.spanner.SpannerException) ChangeEventSequence(com.google.cloud.teleport.v2.templates.datastream.ChangeEventSequence)

Example 2 with ChangeEventContext

use of com.google.cloud.teleport.v2.templates.datastream.ChangeEventContext in project DataflowTemplates by GoogleCloudPlatform.

the class OracleChangeEventContextTest method canGenerateShadowTableMutation.

@Test
public void canGenerateShadowTableMutation() throws Exception {
    // Test Ddl
    Ddl ddl = ChangeEventConvertorTest.getTestDdl();
    // Test Change Event
    JSONObject changeEvent = ChangeEventConvertorTest.getTestChangeEvent("Users2");
    changeEvent.put(DatastreamConstants.ORACLE_TIMESTAMP_KEY, eventTimestamp);
    changeEvent.put(DatastreamConstants.ORACLE_SCN_KEY, "1");
    changeEvent.put(DatastreamConstants.EVENT_SOURCE_TYPE_KEY, DatastreamConstants.ORACLE_SOURCE_TYPE);
    ChangeEventContext changeEventContext = ChangeEventContextFactory.createChangeEventContext(getJsonNode(changeEvent.toString()), ddl, "shadow_", DatastreamConstants.ORACLE_SOURCE_TYPE);
    Mutation shadowMutation = changeEventContext.getShadowTableMutation();
    Map<String, Value> actual = shadowMutation.asMap();
    // Expected result
    Map<String, Value> expected = ChangeEventConvertorTest.getExpectedMapForTestChangeEvent();
    expected.put(DatastreamConstants.ORACLE_TIMESTAMP_SHADOW_INFO.getLeft(), Value.int64(eventTimestamp));
    expected.put(DatastreamConstants.ORACLE_SCN_SHADOW_INFO.getLeft(), Value.int64(1));
    // Verify if OracleChangeEventContext was actually created.
    assertThat(changeEventContext, instanceOf(OracleChangeEventContext.class));
    // Verify shadow mutation
    assertThat(actual, is(expected));
    assertEquals(shadowMutation.getTable(), "shadow_Users2");
    assertEquals(shadowMutation.getOperation(), Mutation.Op.INSERT_OR_UPDATE);
}
Also used : JSONObject(org.json.JSONObject) Value(com.google.cloud.spanner.Value) Mutation(com.google.cloud.spanner.Mutation) Ddl(com.google.cloud.teleport.v2.templates.spanner.ddl.Ddl) Test(org.junit.Test)

Example 3 with ChangeEventContext

use of com.google.cloud.teleport.v2.templates.datastream.ChangeEventContext in project DataflowTemplates by GoogleCloudPlatform.

the class MySqlChangeEventContextTest method canGenerateShadowTableMutation.

@Test
public void canGenerateShadowTableMutation() throws Exception {
    long eventTimestamp = 1615159728L;
    // Test Ddl
    Ddl ddl = ChangeEventConvertorTest.getTestDdl();
    // Test Change Event
    JSONObject changeEvent = ChangeEventConvertorTest.getTestChangeEvent("Users2");
    changeEvent.put(DatastreamConstants.MYSQL_TIMESTAMP_KEY, eventTimestamp);
    changeEvent.put(DatastreamConstants.MYSQL_LOGFILE_KEY, "file1.log");
    changeEvent.put(DatastreamConstants.MYSQL_LOGPOSITION_KEY, 1L);
    changeEvent.put(DatastreamConstants.EVENT_SOURCE_TYPE_KEY, DatastreamConstants.MYSQL_SOURCE_TYPE);
    ChangeEventContext changeEventContext = ChangeEventContextFactory.createChangeEventContext(getJsonNode(changeEvent.toString()), ddl, "shadow_", DatastreamConstants.MYSQL_SOURCE_TYPE);
    Mutation shadowMutation = changeEventContext.getShadowTableMutation();
    Map<String, Value> actual = shadowMutation.asMap();
    // Expected result
    Map<String, Value> expected = ChangeEventConvertorTest.getExpectedMapForTestChangeEvent();
    expected.put(DatastreamConstants.MYSQL_TIMESTAMP_SHADOW_INFO.getLeft(), Value.int64(eventTimestamp));
    expected.put(DatastreamConstants.MYSQL_LOGFILE_SHADOW_INFO.getLeft(), Value.string("file1.log"));
    expected.put(DatastreamConstants.MYSQL_LOGPOSITION_SHADOW_INFO.getLeft(), Value.int64(1));
    // Verify if MySqlChangeEventContext was actually created.
    assertThat(changeEventContext, instanceOf(MySqlChangeEventContext.class));
    // Verify shadow mutation
    assertThat(actual, is(expected));
    assertEquals(shadowMutation.getTable(), "shadow_Users2");
    assertEquals(shadowMutation.getOperation(), Mutation.Op.INSERT_OR_UPDATE);
}
Also used : JSONObject(org.json.JSONObject) Value(com.google.cloud.spanner.Value) Mutation(com.google.cloud.spanner.Mutation) Ddl(com.google.cloud.teleport.v2.templates.spanner.ddl.Ddl) Test(org.junit.Test)

Example 4 with ChangeEventContext

use of com.google.cloud.teleport.v2.templates.datastream.ChangeEventContext in project DataflowTemplates by GoogleCloudPlatform.

the class MySqlChangeEventContextTest method canGenerateShadowTableMutationForBackfillEventsWithMissingSortOrderKeys.

@Test
public void canGenerateShadowTableMutationForBackfillEventsWithMissingSortOrderKeys() throws Exception {
    long eventTimestamp = 1615159728L;
    // Test Ddl
    Ddl ddl = ChangeEventConvertorTest.getTestDdl();
    // Test Change Event which does not contain sort order fields like log file and log position.
    JSONObject changeEvent = ChangeEventConvertorTest.getTestChangeEvent("Users2");
    changeEvent.put(DatastreamConstants.MYSQL_TIMESTAMP_KEY, eventTimestamp);
    changeEvent.put(DatastreamConstants.EVENT_SOURCE_TYPE_KEY, DatastreamConstants.MYSQL_SOURCE_TYPE);
    ChangeEventContext changeEventContext = ChangeEventContextFactory.createChangeEventContext(getJsonNode(changeEvent.toString()), ddl, "shadow_", DatastreamConstants.MYSQL_SOURCE_TYPE);
    Mutation shadowMutation = changeEventContext.getShadowTableMutation();
    Map<String, Value> actual = shadowMutation.asMap();
    // Expected result
    Map<String, Value> expected = ChangeEventConvertorTest.getExpectedMapForTestChangeEvent();
    expected.put(DatastreamConstants.MYSQL_TIMESTAMP_SHADOW_INFO.getLeft(), Value.int64(eventTimestamp));
    expected.put(DatastreamConstants.MYSQL_LOGFILE_SHADOW_INFO.getLeft(), Value.string(""));
    expected.put(DatastreamConstants.MYSQL_LOGPOSITION_SHADOW_INFO.getLeft(), Value.int64(-1));
    // Verify if MySqlChangeEventContext was actually created.
    assertThat(changeEventContext, instanceOf(MySqlChangeEventContext.class));
    // Verify shadow mutation
    assertThat(actual, is(expected));
    assertEquals(shadowMutation.getTable(), "shadow_Users2");
    assertEquals(shadowMutation.getOperation(), Mutation.Op.INSERT_OR_UPDATE);
}
Also used : JSONObject(org.json.JSONObject) Value(com.google.cloud.spanner.Value) Mutation(com.google.cloud.spanner.Mutation) Ddl(com.google.cloud.teleport.v2.templates.spanner.ddl.Ddl) Test(org.junit.Test)

Example 5 with ChangeEventContext

use of com.google.cloud.teleport.v2.templates.datastream.ChangeEventContext in project DataflowTemplates by GoogleCloudPlatform.

the class MySqlChangeEventContextTest method canGenerateShadowTableMutationForBackfillEvents.

@Test
public void canGenerateShadowTableMutationForBackfillEvents() throws Exception {
    long eventTimestamp = 1615159728L;
    // Test Ddl
    Ddl ddl = ChangeEventConvertorTest.getTestDdl();
    // Test Change Event
    JSONObject changeEvent = ChangeEventConvertorTest.getTestChangeEvent("Users2");
    changeEvent.put(DatastreamConstants.MYSQL_TIMESTAMP_KEY, eventTimestamp);
    changeEvent.put(DatastreamConstants.MYSQL_LOGFILE_KEY, JSONObject.NULL);
    changeEvent.put(DatastreamConstants.MYSQL_LOGPOSITION_KEY, JSONObject.NULL);
    changeEvent.put(DatastreamConstants.EVENT_SOURCE_TYPE_KEY, DatastreamConstants.MYSQL_SOURCE_TYPE);
    ChangeEventContext changeEventContext = ChangeEventContextFactory.createChangeEventContext(getJsonNode(changeEvent.toString()), ddl, "shadow_", DatastreamConstants.MYSQL_SOURCE_TYPE);
    Mutation shadowMutation = changeEventContext.getShadowTableMutation();
    Map<String, Value> actual = shadowMutation.asMap();
    // Expected result
    Map<String, Value> expected = ChangeEventConvertorTest.getExpectedMapForTestChangeEvent();
    expected.put(DatastreamConstants.MYSQL_TIMESTAMP_SHADOW_INFO.getLeft(), Value.int64(eventTimestamp));
    expected.put(DatastreamConstants.MYSQL_LOGFILE_SHADOW_INFO.getLeft(), Value.string(""));
    expected.put(DatastreamConstants.MYSQL_LOGPOSITION_SHADOW_INFO.getLeft(), Value.int64(-1));
    // Verify if MySqlChangeEventContext was actually created.
    assertThat(changeEventContext, instanceOf(MySqlChangeEventContext.class));
    // Verify shadow mutation
    assertThat(actual, is(expected));
    assertEquals(shadowMutation.getTable(), "shadow_Users2");
    assertEquals(shadowMutation.getOperation(), Mutation.Op.INSERT_OR_UPDATE);
}
Also used : JSONObject(org.json.JSONObject) Value(com.google.cloud.spanner.Value) Mutation(com.google.cloud.spanner.Mutation) Ddl(com.google.cloud.teleport.v2.templates.spanner.ddl.Ddl) Test(org.junit.Test)

Aggregations

Ddl (com.google.cloud.teleport.v2.templates.spanner.ddl.Ddl)7 Mutation (com.google.cloud.spanner.Mutation)6 Value (com.google.cloud.spanner.Value)6 JSONObject (org.json.JSONObject)6 Test (org.junit.Test)6 DeserializationFeature (com.fasterxml.jackson.databind.DeserializationFeature)1 JsonNode (com.fasterxml.jackson.databind.JsonNode)1 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)1 Timestamp (com.google.cloud.Timestamp)1 SpannerException (com.google.cloud.spanner.SpannerException)1 TransactionCallable (com.google.cloud.spanner.TransactionRunner.TransactionCallable)1 ChangeEventContext (com.google.cloud.teleport.v2.templates.datastream.ChangeEventContext)1 ChangeEventContextFactory (com.google.cloud.teleport.v2.templates.datastream.ChangeEventContextFactory)1 ChangeEventConvertorException (com.google.cloud.teleport.v2.templates.datastream.ChangeEventConvertorException)1 ChangeEventSequence (com.google.cloud.teleport.v2.templates.datastream.ChangeEventSequence)1 ChangeEventSequenceFactory (com.google.cloud.teleport.v2.templates.datastream.ChangeEventSequenceFactory)1 InvalidChangeEventException (com.google.cloud.teleport.v2.templates.datastream.InvalidChangeEventException)1 FailsafeElement (com.google.cloud.teleport.v2.values.FailsafeElement)1 Preconditions (com.google.common.base.Preconditions)1 PrintWriter (java.io.PrintWriter)1