Search in sources :

Example 1 with PARTITION_ID_ATTRIBUTE_LABEL

use of org.apache.beam.sdk.io.gcp.spanner.changestreams.ChangeStreamMetrics.PARTITION_ID_ATTRIBUTE_LABEL in project beam by apache.

the class ChildPartitionsRecordAction method processChildPartition.

// Unboxing of runInTransaction result will not produce a null value, we can ignore it
@SuppressWarnings("nullness")
private void processChildPartition(PartitionMetadata partition, ChildPartitionsRecord record, ChildPartition childPartition) {
    try (Scope scope = TRACER.spanBuilder("ChildPartitionsRecordAction.processChildPartition").setRecordEvents(true).startScopedSpan()) {
        TRACER.getCurrentSpan().putAttribute(PARTITION_ID_ATTRIBUTE_LABEL, AttributeValue.stringAttributeValue(partition.getPartitionToken()));
        final String partitionToken = partition.getPartitionToken();
        final String childPartitionToken = childPartition.getToken();
        final boolean isSplit = isSplit(childPartition);
        LOG.debug("[" + partitionToken + "] Processing child partition" + (isSplit ? " split" : " merge") + " event");
        final PartitionMetadata row = toPartitionMetadata(record.getStartTimestamp(), partition.getEndTimestamp(), partition.getHeartbeatMillis(), childPartition);
        LOG.debug("[" + partitionToken + "] Inserting child partition token " + childPartitionToken);
        final Boolean insertedRow = partitionMetadataDao.runInTransaction(transaction -> {
            if (transaction.getPartition(childPartitionToken) == null) {
                transaction.insert(row);
                return true;
            } else {
                return false;
            }
        }).getResult();
        if (insertedRow && isSplit) {
            metrics.incPartitionRecordSplitCount();
        } else if (insertedRow) {
            metrics.incPartitionRecordMergeCount();
        } else {
            LOG.debug("[" + partitionToken + "] Child token " + childPartitionToken + " already exists, skipping...");
        }
    }
}
Also used : Tracer(io.opencensus.trace.Tracer) Logger(org.slf4j.Logger) AttributeValue(io.opencensus.trace.AttributeValue) ChangeStreamMetrics(org.apache.beam.sdk.io.gcp.spanner.changestreams.ChangeStreamMetrics) ChildPartition(org.apache.beam.sdk.io.gcp.spanner.changestreams.model.ChildPartition) PartitionMetadataDao(org.apache.beam.sdk.io.gcp.spanner.changestreams.dao.PartitionMetadataDao) Scope(io.opencensus.common.Scope) LoggerFactory(org.slf4j.LoggerFactory) ProcessContinuation(org.apache.beam.sdk.transforms.DoFn.ProcessContinuation) PARTITION_ID_ATTRIBUTE_LABEL(org.apache.beam.sdk.io.gcp.spanner.changestreams.ChangeStreamMetrics.PARTITION_ID_ATTRIBUTE_LABEL) Timestamp(com.google.cloud.Timestamp) TimestampConverter(org.apache.beam.sdk.io.gcp.spanner.changestreams.TimestampConverter) ManualWatermarkEstimator(org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator) PartitionMetadata(org.apache.beam.sdk.io.gcp.spanner.changestreams.model.PartitionMetadata) ChildPartitionsRecord(org.apache.beam.sdk.io.gcp.spanner.changestreams.model.ChildPartitionsRecord) VisibleForTesting(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting) CREATED(org.apache.beam.sdk.io.gcp.spanner.changestreams.model.PartitionMetadata.State.CREATED) Instant(org.joda.time.Instant) Optional(java.util.Optional) Tracing(io.opencensus.trace.Tracing) OffsetRange(org.apache.beam.sdk.io.range.OffsetRange) RestrictionTracker(org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker) Nullable(javax.annotation.Nullable) Scope(io.opencensus.common.Scope) PartitionMetadata(org.apache.beam.sdk.io.gcp.spanner.changestreams.model.PartitionMetadata)

Aggregations

Timestamp (com.google.cloud.Timestamp)1 Scope (io.opencensus.common.Scope)1 AttributeValue (io.opencensus.trace.AttributeValue)1 Tracer (io.opencensus.trace.Tracer)1 Tracing (io.opencensus.trace.Tracing)1 Optional (java.util.Optional)1 Nullable (javax.annotation.Nullable)1 ChangeStreamMetrics (org.apache.beam.sdk.io.gcp.spanner.changestreams.ChangeStreamMetrics)1 PARTITION_ID_ATTRIBUTE_LABEL (org.apache.beam.sdk.io.gcp.spanner.changestreams.ChangeStreamMetrics.PARTITION_ID_ATTRIBUTE_LABEL)1 TimestampConverter (org.apache.beam.sdk.io.gcp.spanner.changestreams.TimestampConverter)1 PartitionMetadataDao (org.apache.beam.sdk.io.gcp.spanner.changestreams.dao.PartitionMetadataDao)1 ChildPartition (org.apache.beam.sdk.io.gcp.spanner.changestreams.model.ChildPartition)1 ChildPartitionsRecord (org.apache.beam.sdk.io.gcp.spanner.changestreams.model.ChildPartitionsRecord)1 PartitionMetadata (org.apache.beam.sdk.io.gcp.spanner.changestreams.model.PartitionMetadata)1 CREATED (org.apache.beam.sdk.io.gcp.spanner.changestreams.model.PartitionMetadata.State.CREATED)1 OffsetRange (org.apache.beam.sdk.io.range.OffsetRange)1 ProcessContinuation (org.apache.beam.sdk.transforms.DoFn.ProcessContinuation)1 ManualWatermarkEstimator (org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator)1 RestrictionTracker (org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker)1 VisibleForTesting (org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting)1