Search in sources :

Example 11 with CounterMetadata

use of com.google.api.services.dataflow.model.CounterMetadata in project beam by apache.

the class ExecutionTimeMonitoringInfoToCounterUpdateTransformer method transform.

@Override
@Nullable
public CounterUpdate transform(MonitoringInfo monitoringInfo) {
    Optional<String> validationResult = validate(monitoringInfo);
    if (validationResult.isPresent()) {
        LOG.debug(validationResult.get());
        return null;
    }
    long value = decodeInt64Counter(monitoringInfo.getPayload());
    String urn = monitoringInfo.getUrn();
    final String ptransform = monitoringInfo.getLabelsMap().get(MonitoringInfoConstants.Labels.PTRANSFORM);
    DataflowStepContext stepContext = transformIdMapping.get(ptransform);
    String counterName = URN_TO_COUNTER_NAME_MAPPING.get(urn);
    CounterStructuredNameAndMetadata name = new CounterStructuredNameAndMetadata();
    name.setName(new CounterStructuredName().setOrigin("SYSTEM").setName(counterName).setOriginalStepName(stepContext.getNameContext().originalName()).setExecutionStepName(stepContext.getNameContext().stageName())).setMetadata(new CounterMetadata().setKind(Kind.SUM.toString()));
    return new CounterUpdate().setStructuredNameAndMetadata(name).setCumulative(true).setInteger(DataflowCounterUpdateExtractor.longToSplitInt(value));
}
Also used : CounterMetadata(com.google.api.services.dataflow.model.CounterMetadata) CounterStructuredName(com.google.api.services.dataflow.model.CounterStructuredName) DataflowStepContext(org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext) CounterStructuredNameAndMetadata(com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata) CounterUpdate(com.google.api.services.dataflow.model.CounterUpdate) Nullable(org.checkerframework.checker.nullness.qual.Nullable)

Example 12 with CounterMetadata

use of com.google.api.services.dataflow.model.CounterMetadata in project beam by apache.

the class UserDistributionMonitoringInfoToCounterUpdateTransformer method transform.

/**
 * Transforms user counter MonitoringInfo to relevant CounterUpdate.
 *
 * @return Relevant CounterUpdate or null if transformation failed.
 */
@Override
@Nullable
public CounterUpdate transform(MonitoringInfo monitoringInfo) {
    Optional<String> validationResult = validate(monitoringInfo);
    if (validationResult.isPresent()) {
        LOG.debug(validationResult.get());
        return null;
    }
    DistributionData data = decodeInt64Distribution(monitoringInfo.getPayload());
    Map<String, String> miLabels = monitoringInfo.getLabelsMap();
    final String ptransform = miLabels.get(MonitoringInfoConstants.Labels.PTRANSFORM);
    final String counterName = miLabels.get(MonitoringInfoConstants.Labels.NAME);
    final String counterNamespace = miLabels.get(MonitoringInfoConstants.Labels.NAMESPACE);
    CounterStructuredNameAndMetadata name = new CounterStructuredNameAndMetadata();
    DataflowStepContext stepContext = transformIdMapping.get(ptransform);
    name.setName(new CounterStructuredName().setOrigin(Origin.USER.toString()).setName(counterName).setOriginalStepName(stepContext.getNameContext().originalName()).setOriginNamespace(counterNamespace)).setMetadata(new CounterMetadata().setKind(Kind.DISTRIBUTION.toString()));
    return new CounterUpdate().setStructuredNameAndMetadata(name).setCumulative(true).setDistribution(new DistributionUpdate().setMax(DataflowCounterUpdateExtractor.longToSplitInt(data.max())).setMin(DataflowCounterUpdateExtractor.longToSplitInt(data.min())).setSum(DataflowCounterUpdateExtractor.longToSplitInt(data.sum())).setCount(DataflowCounterUpdateExtractor.longToSplitInt(data.count())));
}
Also used : CounterMetadata(com.google.api.services.dataflow.model.CounterMetadata) DistributionData(org.apache.beam.runners.core.metrics.DistributionData) CounterStructuredName(com.google.api.services.dataflow.model.CounterStructuredName) DistributionUpdate(com.google.api.services.dataflow.model.DistributionUpdate) CounterStructuredNameAndMetadata(com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata) DataflowStepContext(org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext) CounterUpdate(com.google.api.services.dataflow.model.CounterUpdate) Nullable(org.checkerframework.checker.nullness.qual.Nullable)

Example 13 with CounterMetadata

use of com.google.api.services.dataflow.model.CounterMetadata in project beam by apache.

the class BatchModeExecutionContextTest method extractMetricUpdatesDistribution.

@Test
public void extractMetricUpdatesDistribution() {
    BatchModeExecutionContext executionContext = BatchModeExecutionContext.forTesting(PipelineOptionsFactory.create(), "testStage");
    DataflowOperationContext operationContext = executionContext.createOperationContext(NameContextsForTests.nameContextForTest());
    Distribution distribution = operationContext.metricsContainer().getDistribution(MetricName.named("namespace", "some-distribution"));
    distribution.update(2);
    distribution.update(8);
    final CounterUpdate expected = new CounterUpdate().setStructuredNameAndMetadata(new CounterStructuredNameAndMetadata().setName(new CounterStructuredName().setOrigin("USER").setOriginNamespace("namespace").setName("some-distribution").setOriginalStepName("originalName")).setMetadata(new CounterMetadata().setKind(Kind.DISTRIBUTION.toString()))).setCumulative(true).setDistribution(new DistributionUpdate().setCount(longToSplitInt(2)).setMax(longToSplitInt(8)).setMin(longToSplitInt(2)).setSum(longToSplitInt(10)));
    assertThat(executionContext.extractMetricUpdates(false), containsInAnyOrder(expected));
}
Also used : CounterMetadata(com.google.api.services.dataflow.model.CounterMetadata) CounterStructuredName(com.google.api.services.dataflow.model.CounterStructuredName) Distribution(org.apache.beam.sdk.metrics.Distribution) DistributionUpdate(com.google.api.services.dataflow.model.DistributionUpdate) CounterStructuredNameAndMetadata(com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata) CounterUpdate(com.google.api.services.dataflow.model.CounterUpdate) Test(org.junit.Test)

Example 14 with CounterMetadata

use of com.google.api.services.dataflow.model.CounterMetadata in project beam by apache.

the class StreamingStepMetricsContainerTest method testCounterUpdateExtraction.

@Test
public void testCounterUpdateExtraction() {
    c1.getCounter(name1).inc(5);
    c2.getCounter(name1).inc(8);
    c2.getCounter(name2).inc(12);
    Iterable<CounterUpdate> updates = StreamingStepMetricsContainer.extractMetricUpdates(registry);
    assertThat(updates, containsInAnyOrder(new CounterUpdate().setStructuredNameAndMetadata(new CounterStructuredNameAndMetadata().setName(new CounterStructuredName().setOrigin(Origin.USER.toString()).setOriginNamespace("ns").setName("name1").setOriginalStepName("s1")).setMetadata(new CounterMetadata().setKind(Kind.SUM.toString()))).setCumulative(false).setInteger(longToSplitInt(5)), new CounterUpdate().setStructuredNameAndMetadata(new CounterStructuredNameAndMetadata().setName(new CounterStructuredName().setOrigin(Origin.USER.toString()).setOriginNamespace("ns").setName("name1").setOriginalStepName("s2")).setMetadata(new CounterMetadata().setKind(Kind.SUM.toString()))).setCumulative(false).setInteger(longToSplitInt(8)), new CounterUpdate().setStructuredNameAndMetadata(new CounterStructuredNameAndMetadata().setName(new CounterStructuredName().setOrigin(Origin.USER.toString()).setOriginNamespace("ns").setName("name2").setOriginalStepName("s2")).setMetadata(new CounterMetadata().setKind(Kind.SUM.toString()))).setCumulative(false).setInteger(longToSplitInt(12))));
    c2.getCounter(name1).inc(7);
    updates = StreamingStepMetricsContainer.extractMetricUpdates(registry);
    assertThat(updates, containsInAnyOrder(new CounterUpdate().setStructuredNameAndMetadata(new CounterStructuredNameAndMetadata().setName(new CounterStructuredName().setOrigin(Origin.USER.toString()).setOriginNamespace("ns").setName("name1").setOriginalStepName("s2")).setMetadata(new CounterMetadata().setKind(Kind.SUM.toString()))).setCumulative(false).setInteger(longToSplitInt(7))));
}
Also used : CounterMetadata(com.google.api.services.dataflow.model.CounterMetadata) CounterStructuredName(com.google.api.services.dataflow.model.CounterStructuredName) CounterStructuredNameAndMetadata(com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata) CounterUpdate(com.google.api.services.dataflow.model.CounterUpdate) Test(org.junit.Test)

Example 15 with CounterMetadata

use of com.google.api.services.dataflow.model.CounterMetadata in project beam by apache.

the class CounterUpdateAggregatorsTest method testAggregateDistribution.

@Test
public void testAggregateDistribution() {
    List<CounterUpdate> distributionUpdates = new ArrayList<>();
    for (int i = 0; i < 10; i++) {
        distributionUpdates.add(new CounterUpdate().setStructuredNameAndMetadata(new CounterStructuredNameAndMetadata().setMetadata(new CounterMetadata().setKind(Kind.DISTRIBUTION.toString()))).setDistribution(new DistributionUpdate().setSum(longToSplitInt((long) i)).setMax(longToSplitInt((long) i)).setMin(longToSplitInt((long) i)).setCount(longToSplitInt((long) 1))));
    }
    List<CounterUpdate> aggregated = CounterUpdateAggregators.aggregate(distributionUpdates);
    assertEquals(1, aggregated.size());
    CounterUpdate combined = aggregated.get(0);
    assertEquals(45L, splitIntToLong(combined.getDistribution().getSum()));
    assertEquals(10L, splitIntToLong(combined.getDistribution().getCount()));
    assertEquals(9L, splitIntToLong(combined.getDistribution().getMax()));
    assertEquals(0L, splitIntToLong(combined.getDistribution().getMin()));
}
Also used : CounterMetadata(com.google.api.services.dataflow.model.CounterMetadata) ArrayList(java.util.ArrayList) DistributionUpdate(com.google.api.services.dataflow.model.DistributionUpdate) CounterStructuredNameAndMetadata(com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata) CounterUpdate(com.google.api.services.dataflow.model.CounterUpdate) Test(org.junit.Test)

Aggregations

CounterMetadata (com.google.api.services.dataflow.model.CounterMetadata)18 CounterStructuredNameAndMetadata (com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata)18 CounterUpdate (com.google.api.services.dataflow.model.CounterUpdate)16 CounterStructuredName (com.google.api.services.dataflow.model.CounterStructuredName)12 Test (org.junit.Test)10 DistributionUpdate (com.google.api.services.dataflow.model.DistributionUpdate)5 ArrayList (java.util.ArrayList)4 DataflowStepContext (org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext)3 Nullable (org.checkerframework.checker.nullness.qual.Nullable)3 Before (org.junit.Before)3 IntegerMean (com.google.api.services.dataflow.model.IntegerMean)2 WorkItemStatus (com.google.api.services.dataflow.model.WorkItemStatus)2 Distribution (org.apache.beam.sdk.metrics.Distribution)2 NameAndKind (com.google.api.services.dataflow.model.NameAndKind)1 Source (com.google.api.services.dataflow.model.Source)1 SplitInt64 (com.google.api.services.dataflow.model.SplitInt64)1 Closeable (java.io.Closeable)1 CounterCell (org.apache.beam.runners.core.metrics.CounterCell)1 DistributionData (org.apache.beam.runners.core.metrics.DistributionData)1 MetricsContainerImpl (org.apache.beam.runners.core.metrics.MetricsContainerImpl)1