Search in sources :

Example 1 with Distribution

use of org.apache.beam.sdk.metrics.Distribution in project beam by apache.

the class StreamingStepMetricsContainerTest method testDistributionUpdateExtraction.

@Test
public void testDistributionUpdateExtraction() {
    Distribution distribution = c1.getDistribution(name1);
    distribution.update(5);
    distribution.update(6);
    distribution.update(7);
    Iterable<CounterUpdate> updates = StreamingStepMetricsContainer.extractMetricUpdates(registry);
    assertThat(updates, containsInAnyOrder(new CounterUpdate().setStructuredNameAndMetadata(new CounterStructuredNameAndMetadata().setName(new CounterStructuredName().setOrigin(Origin.USER.toString()).setOriginNamespace("ns").setName("name1").setOriginalStepName("s1")).setMetadata(new CounterMetadata().setKind(Kind.DISTRIBUTION.toString()))).setCumulative(false).setDistribution(new DistributionUpdate().setCount(longToSplitInt(3)).setMax(longToSplitInt(7)).setMin(longToSplitInt(5)).setSum(longToSplitInt(18)))));
    c1.getDistribution(name1).update(3);
    updates = StreamingStepMetricsContainer.extractMetricUpdates(registry);
    assertThat(updates, containsInAnyOrder(new CounterUpdate().setStructuredNameAndMetadata(new CounterStructuredNameAndMetadata().setName(new CounterStructuredName().setOrigin(Origin.USER.toString()).setOriginNamespace("ns").setName("name1").setOriginalStepName("s1")).setMetadata(new CounterMetadata().setKind(Kind.DISTRIBUTION.toString()))).setCumulative(false).setDistribution(new DistributionUpdate().setCount(longToSplitInt(1)).setMax(longToSplitInt(3)).setMin(longToSplitInt(3)).setSum(longToSplitInt(3)))));
}
Also used : CounterMetadata(com.google.api.services.dataflow.model.CounterMetadata) CounterStructuredName(com.google.api.services.dataflow.model.CounterStructuredName) Distribution(org.apache.beam.sdk.metrics.Distribution) DistributionUpdate(com.google.api.services.dataflow.model.DistributionUpdate) CounterStructuredNameAndMetadata(com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata) CounterUpdate(com.google.api.services.dataflow.model.CounterUpdate) Test(org.junit.Test)

Example 2 with Distribution

use of org.apache.beam.sdk.metrics.Distribution in project beam by apache.

the class FlinkMetricContainerTest method testDistribution.

@Test
public void testDistribution() {
    FlinkMetricContainer.FlinkDistributionGauge flinkGauge = new FlinkMetricContainer.FlinkDistributionGauge(DistributionResult.IDENTITY_ELEMENT);
    when(metricGroup.gauge(eq("namespace.name"), anyObject())).thenReturn(flinkGauge);
    MetricsContainer step = container.getMetricsContainer("step");
    MetricName metricName = MetricName.named("namespace", "name");
    Distribution distribution = step.getDistribution(metricName);
    assertThat(flinkGauge.getValue(), is(DistributionResult.IDENTITY_ELEMENT));
    // first set will install the mocked distribution
    container.updateMetrics("step");
    distribution.update(42);
    distribution.update(-23);
    distribution.update(0);
    distribution.update(1);
    container.updateMetrics("step");
    assertThat(flinkGauge.getValue().getMax(), is(42L));
    assertThat(flinkGauge.getValue().getMin(), is(-23L));
    assertThat(flinkGauge.getValue().getCount(), is(4L));
    assertThat(flinkGauge.getValue().getSum(), is(20L));
    assertThat(flinkGauge.getValue().getMean(), is(5.0));
}
Also used : MetricName(org.apache.beam.sdk.metrics.MetricName) MonitoringInfoMetricName(org.apache.beam.runners.core.metrics.MonitoringInfoMetricName) MetricsContainer(org.apache.beam.sdk.metrics.MetricsContainer) Distribution(org.apache.beam.sdk.metrics.Distribution) FlinkDistributionGauge(org.apache.beam.runners.flink.metrics.FlinkMetricContainer.FlinkDistributionGauge) FlinkDistributionGauge(org.apache.beam.runners.flink.metrics.FlinkMetricContainer.FlinkDistributionGauge) Test(org.junit.Test)

Example 3 with Distribution

use of org.apache.beam.sdk.metrics.Distribution in project beam by apache.

the class MetricsContainerImpl method updateForDistributionInt64Type.

private void updateForDistributionInt64Type(MonitoringInfo monitoringInfo) {
    MetricName metricName = MonitoringInfoMetricName.of(monitoringInfo);
    Distribution distribution = getDistribution(metricName);
    DistributionData data = decodeInt64Distribution(monitoringInfo.getPayload());
    distribution.update(data.sum(), data.count(), data.min(), data.max());
}
Also used : MetricName(org.apache.beam.sdk.metrics.MetricName) MonitoringInfoEncodings.encodeInt64Distribution(org.apache.beam.runners.core.metrics.MonitoringInfoEncodings.encodeInt64Distribution) MonitoringInfoEncodings.decodeInt64Distribution(org.apache.beam.runners.core.metrics.MonitoringInfoEncodings.decodeInt64Distribution) Distribution(org.apache.beam.sdk.metrics.Distribution)

Example 4 with Distribution

use of org.apache.beam.sdk.metrics.Distribution in project beam by apache.

the class BatchModeExecutionContextTest method extractMetricUpdatesDistribution.

@Test
public void extractMetricUpdatesDistribution() {
    BatchModeExecutionContext executionContext = BatchModeExecutionContext.forTesting(PipelineOptionsFactory.create(), "testStage");
    DataflowOperationContext operationContext = executionContext.createOperationContext(NameContextsForTests.nameContextForTest());
    Distribution distribution = operationContext.metricsContainer().getDistribution(MetricName.named("namespace", "some-distribution"));
    distribution.update(2);
    distribution.update(8);
    final CounterUpdate expected = new CounterUpdate().setStructuredNameAndMetadata(new CounterStructuredNameAndMetadata().setName(new CounterStructuredName().setOrigin("USER").setOriginNamespace("namespace").setName("some-distribution").setOriginalStepName("originalName")).setMetadata(new CounterMetadata().setKind(Kind.DISTRIBUTION.toString()))).setCumulative(true).setDistribution(new DistributionUpdate().setCount(longToSplitInt(2)).setMax(longToSplitInt(8)).setMin(longToSplitInt(2)).setSum(longToSplitInt(10)));
    assertThat(executionContext.extractMetricUpdates(false), containsInAnyOrder(expected));
}
Also used : CounterMetadata(com.google.api.services.dataflow.model.CounterMetadata) CounterStructuredName(com.google.api.services.dataflow.model.CounterStructuredName) Distribution(org.apache.beam.sdk.metrics.Distribution) DistributionUpdate(com.google.api.services.dataflow.model.DistributionUpdate) CounterStructuredNameAndMetadata(com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata) CounterUpdate(com.google.api.services.dataflow.model.CounterUpdate) Test(org.junit.Test)

Aggregations

Distribution (org.apache.beam.sdk.metrics.Distribution)4 Test (org.junit.Test)3 CounterMetadata (com.google.api.services.dataflow.model.CounterMetadata)2 CounterStructuredName (com.google.api.services.dataflow.model.CounterStructuredName)2 CounterStructuredNameAndMetadata (com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata)2 CounterUpdate (com.google.api.services.dataflow.model.CounterUpdate)2 DistributionUpdate (com.google.api.services.dataflow.model.DistributionUpdate)2 MetricName (org.apache.beam.sdk.metrics.MetricName)2 MonitoringInfoEncodings.decodeInt64Distribution (org.apache.beam.runners.core.metrics.MonitoringInfoEncodings.decodeInt64Distribution)1 MonitoringInfoEncodings.encodeInt64Distribution (org.apache.beam.runners.core.metrics.MonitoringInfoEncodings.encodeInt64Distribution)1 MonitoringInfoMetricName (org.apache.beam.runners.core.metrics.MonitoringInfoMetricName)1 FlinkDistributionGauge (org.apache.beam.runners.flink.metrics.FlinkMetricContainer.FlinkDistributionGauge)1 MetricsContainer (org.apache.beam.sdk.metrics.MetricsContainer)1