Examples with InterceptingTaskMetricGroup org.apache.flink.runtime.metrics.util.InterceptingTaskMetricGroup used on opensource projects

Search in sources :

Example 1 with InterceptingTaskMetricGroup

use of org.apache.flink.runtime.metrics.util.InterceptingTaskMetricGroup in project flink by apache.

the class MultipleInputStreamTaskTest method testWatermarkMetrics.

@Test
@SuppressWarnings("unchecked")
public void testWatermarkMetrics() throws Exception {
    OperatorID mainOperatorId = new OperatorID();
    OperatorID chainedOperatorId = new OperatorID();
    InterceptingOperatorMetricGroup mainOperatorMetricGroup = new InterceptingOperatorMetricGroup();
    InterceptingOperatorMetricGroup chainedOperatorMetricGroup = new InterceptingOperatorMetricGroup();
    InterceptingTaskMetricGroup taskMetricGroup = new InterceptingTaskMetricGroup() {

        @Override
        public InternalOperatorMetricGroup getOrAddOperator(OperatorID id, String name) {
            if (id.equals(mainOperatorId)) {
                return mainOperatorMetricGroup;
            } else if (id.equals(chainedOperatorId)) {
                return chainedOperatorMetricGroup;
            } else {
                return super.getOrAddOperator(id, name);
            }
        }
    };
    try (StreamTaskMailboxTestHarness<String> testHarness = new StreamTaskMailboxTestHarnessBuilder<>(MultipleInputStreamTask::new, BasicTypeInfo.STRING_TYPE_INFO).modifyExecutionConfig(applyObjectReuse(objectReuse)).addInput(BasicTypeInfo.STRING_TYPE_INFO).addSourceInput(new SourceOperatorFactory<>(new MockSource(Boundedness.CONTINUOUS_UNBOUNDED, 2, true, false), WatermarkStrategy.forGenerator(ctx -> new RecordToWatermarkGenerator())), BasicTypeInfo.INT_TYPE_INFO).addInput(BasicTypeInfo.DOUBLE_TYPE_INFO).setupOperatorChain(mainOperatorId, new MapToStringMultipleInputOperatorFactory(3)).chain(chainedOperatorId, new WatermarkMetricOperator(), BasicTypeInfo.STRING_TYPE_INFO.createSerializer(new ExecutionConfig())).finish().setTaskMetricGroup(taskMetricGroup).build()) {
        Gauge<Long> taskInputWatermarkGauge = (Gauge<Long>) taskMetricGroup.get(MetricNames.IO_CURRENT_INPUT_WATERMARK);
        Gauge<Long> mainInput1WatermarkGauge = (Gauge<Long>) mainOperatorMetricGroup.get(MetricNames.currentInputWatermarkName(1));
        Gauge<Long> mainInput2WatermarkGauge = (Gauge<Long>) mainOperatorMetricGroup.get(MetricNames.currentInputWatermarkName(2));
        Gauge<Long> mainInput3WatermarkGauge = (Gauge<Long>) mainOperatorMetricGroup.get(MetricNames.currentInputWatermarkName(3));
        Gauge<Long> mainInputWatermarkGauge = (Gauge<Long>) mainOperatorMetricGroup.get(MetricNames.IO_CURRENT_INPUT_WATERMARK);
        Gauge<Long> mainOutputWatermarkGauge = (Gauge<Long>) mainOperatorMetricGroup.get(MetricNames.IO_CURRENT_OUTPUT_WATERMARK);
        Gauge<Long> chainedInputWatermarkGauge = (Gauge<Long>) chainedOperatorMetricGroup.get(MetricNames.IO_CURRENT_INPUT_WATERMARK);
        Gauge<Long> chainedOutputWatermarkGauge = (Gauge<Long>) chainedOperatorMetricGroup.get(MetricNames.IO_CURRENT_OUTPUT_WATERMARK);
        assertEquals(Long.MIN_VALUE, taskInputWatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, mainInputWatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, mainInput1WatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, mainInput2WatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, mainInput3WatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, mainOutputWatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, chainedInputWatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, chainedOutputWatermarkGauge.getValue().longValue());
        testHarness.processElement(new Watermark(1L), 0);
        assertEquals(Long.MIN_VALUE, taskInputWatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, mainInputWatermarkGauge.getValue().longValue());
        assertEquals(1L, mainInput1WatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, mainInput2WatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, mainInput3WatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, mainOutputWatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, chainedInputWatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, chainedOutputWatermarkGauge.getValue().longValue());
        addSourceRecords(testHarness, 1, 2);
        testHarness.processAll();
        assertEquals(Long.MIN_VALUE, taskInputWatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, mainInputWatermarkGauge.getValue().longValue());
        assertEquals(1L, mainInput1WatermarkGauge.getValue().longValue());
        assertEquals(2L, mainInput2WatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, mainInput3WatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, mainOutputWatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, chainedInputWatermarkGauge.getValue().longValue());
        assertEquals(Long.MIN_VALUE, chainedOutputWatermarkGauge.getValue().longValue());
        testHarness.processElement(new Watermark(2L), 1);
        assertEquals(1L, taskInputWatermarkGauge.getValue().longValue());
        assertEquals(1L, mainInputWatermarkGauge.getValue().longValue());
        assertEquals(1L, mainInput1WatermarkGauge.getValue().longValue());
        assertEquals(2L, mainInput2WatermarkGauge.getValue().longValue());
        assertEquals(2L, mainInput3WatermarkGauge.getValue().longValue());
        assertEquals(1L, mainOutputWatermarkGauge.getValue().longValue());
        assertEquals(1L, chainedInputWatermarkGauge.getValue().longValue());
        assertEquals(2L, chainedOutputWatermarkGauge.getValue().longValue());
        testHarness.processElement(new Watermark(4L), 0);
        addSourceRecords(testHarness, 1, 3);
        testHarness.processAll();
        assertEquals(2L, taskInputWatermarkGauge.getValue().longValue());
        assertEquals(2L, mainInputWatermarkGauge.getValue().longValue());
        assertEquals(4L, mainInput1WatermarkGauge.getValue().longValue());
        assertEquals(3L, mainInput2WatermarkGauge.getValue().longValue());
        assertEquals(2L, mainInput3WatermarkGauge.getValue().longValue());
        assertEquals(2L, mainOutputWatermarkGauge.getValue().longValue());
        assertEquals(2L, chainedInputWatermarkGauge.getValue().longValue());
        assertEquals(4L, chainedOutputWatermarkGauge.getValue().longValue());
        finishAddingRecords(testHarness, 1);
        testHarness.endInput();
        testHarness.waitForTaskCompletion();
        testHarness.finishProcessing();
    }
}
Also used : MockSource(org.apache.flink.api.connector.source.mocks.MockSource) OperatorID(org.apache.flink.runtime.jobgraph.OperatorID) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) Gauge(org.apache.flink.metrics.Gauge) InterceptingOperatorMetricGroup(org.apache.flink.runtime.metrics.util.InterceptingOperatorMetricGroup) WatermarkMetricOperator(org.apache.flink.streaming.runtime.tasks.OneInputStreamTaskTest.WatermarkMetricOperator) InterceptingTaskMetricGroup(org.apache.flink.runtime.metrics.util.InterceptingTaskMetricGroup) Watermark(org.apache.flink.streaming.api.watermark.Watermark) Test(org.junit.Test)

Example 2 with InterceptingTaskMetricGroup

use of org.apache.flink.runtime.metrics.util.InterceptingTaskMetricGroup in project flink by apache.

the class TwoInputStreamTaskTest method testWatermarkMetrics.

@Test
@SuppressWarnings("unchecked")
public void testWatermarkMetrics() throws Exception {
    final TwoInputStreamTaskTestHarness<String, Integer, String> testHarness = new TwoInputStreamTaskTestHarness<>(TwoInputStreamTask::new, BasicTypeInfo.STRING_TYPE_INFO, BasicTypeInfo.INT_TYPE_INFO, BasicTypeInfo.STRING_TYPE_INFO);
    CoStreamMap<String, Integer, String> headOperator = new CoStreamMap<>(new IdentityMap());
    final OperatorID headOperatorId = new OperatorID();
    OneInputStreamTaskTest.WatermarkMetricOperator chainedOperator = new OneInputStreamTaskTest.WatermarkMetricOperator();
    OperatorID chainedOperatorId = new OperatorID();
    testHarness.setupOperatorChain(headOperatorId, headOperator).chain(chainedOperatorId, chainedOperator, BasicTypeInfo.STRING_TYPE_INFO.createSerializer(new ExecutionConfig())).finish();
    InterceptingOperatorMetricGroup headOperatorMetricGroup = new InterceptingOperatorMetricGroup();
    InterceptingOperatorMetricGroup chainedOperatorMetricGroup = new InterceptingOperatorMetricGroup();
    InterceptingTaskMetricGroup taskMetricGroup = new InterceptingTaskMetricGroup() {

        @Override
        public InternalOperatorMetricGroup getOrAddOperator(OperatorID id, String name) {
            if (id.equals(headOperatorId)) {
                return headOperatorMetricGroup;
            } else if (id.equals(chainedOperatorId)) {
                return chainedOperatorMetricGroup;
            } else {
                return super.getOrAddOperator(id, name);
            }
        }
    };
    StreamMockEnvironment env = new StreamMockEnvironment(testHarness.jobConfig, testHarness.taskConfig, testHarness.memorySize, new MockInputSplitProvider(), testHarness.bufferSize, new TestTaskStateManager()) {

        @Override
        public TaskMetricGroup getMetricGroup() {
            return taskMetricGroup;
        }
    };
    testHarness.invoke(env);
    testHarness.waitForTaskRunning();
    Gauge<Long> taskInputWatermarkGauge = (Gauge<Long>) taskMetricGroup.get(MetricNames.IO_CURRENT_INPUT_WATERMARK);
    Gauge<Long> headInput1WatermarkGauge = (Gauge<Long>) headOperatorMetricGroup.get(MetricNames.IO_CURRENT_INPUT_1_WATERMARK);
    Gauge<Long> headInput2WatermarkGauge = (Gauge<Long>) headOperatorMetricGroup.get(MetricNames.IO_CURRENT_INPUT_2_WATERMARK);
    Gauge<Long> headInputWatermarkGauge = (Gauge<Long>) headOperatorMetricGroup.get(MetricNames.IO_CURRENT_INPUT_WATERMARK);
    Gauge<Long> headOutputWatermarkGauge = (Gauge<Long>) headOperatorMetricGroup.get(MetricNames.IO_CURRENT_OUTPUT_WATERMARK);
    Gauge<Long> chainedInputWatermarkGauge = (Gauge<Long>) chainedOperatorMetricGroup.get(MetricNames.IO_CURRENT_INPUT_WATERMARK);
    Gauge<Long> chainedOutputWatermarkGauge = (Gauge<Long>) chainedOperatorMetricGroup.get(MetricNames.IO_CURRENT_OUTPUT_WATERMARK);
    Assert.assertEquals("A metric was registered multiple times.", 7, new HashSet<>(Arrays.asList(taskInputWatermarkGauge, headInput1WatermarkGauge, headInput2WatermarkGauge, headInputWatermarkGauge, headOutputWatermarkGauge, chainedInputWatermarkGauge, chainedOutputWatermarkGauge)).size());
    Assert.assertEquals(Long.MIN_VALUE, taskInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, headInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, headInput1WatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, headInput2WatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, headOutputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, chainedInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, chainedOutputWatermarkGauge.getValue().longValue());
    testHarness.processElement(new Watermark(1L), 0, 0);
    testHarness.waitForInputProcessing();
    Assert.assertEquals(Long.MIN_VALUE, taskInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, headInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(1L, headInput1WatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, headInput2WatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, headOutputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, chainedInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, chainedOutputWatermarkGauge.getValue().longValue());
    testHarness.processElement(new Watermark(2L), 1, 0);
    testHarness.waitForInputProcessing();
    Assert.assertEquals(1L, taskInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(1L, headInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(1L, headInput1WatermarkGauge.getValue().longValue());
    Assert.assertEquals(2L, headInput2WatermarkGauge.getValue().longValue());
    Assert.assertEquals(1L, headOutputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(1L, chainedInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(2L, chainedOutputWatermarkGauge.getValue().longValue());
    testHarness.processElement(new Watermark(3L), 0, 0);
    testHarness.waitForInputProcessing();
    Assert.assertEquals(2L, taskInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(2L, headInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(3L, headInput1WatermarkGauge.getValue().longValue());
    Assert.assertEquals(2L, headInput2WatermarkGauge.getValue().longValue());
    Assert.assertEquals(2L, headOutputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(2L, chainedInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(4L, chainedOutputWatermarkGauge.getValue().longValue());
    testHarness.endInput();
    testHarness.waitForTaskCompletion();
}
Also used : CoStreamMap(org.apache.flink.streaming.api.operators.co.CoStreamMap) OperatorID(org.apache.flink.runtime.jobgraph.OperatorID) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) Gauge(org.apache.flink.metrics.Gauge) TestTaskStateManager(org.apache.flink.runtime.state.TestTaskStateManager) InterceptingOperatorMetricGroup(org.apache.flink.runtime.metrics.util.InterceptingOperatorMetricGroup) InterceptingTaskMetricGroup(org.apache.flink.runtime.metrics.util.InterceptingTaskMetricGroup) MockInputSplitProvider(org.apache.flink.runtime.operators.testutils.MockInputSplitProvider) Watermark(org.apache.flink.streaming.api.watermark.Watermark) HashSet(java.util.HashSet) Test(org.junit.Test)

Example 3 with InterceptingTaskMetricGroup

use of org.apache.flink.runtime.metrics.util.InterceptingTaskMetricGroup in project flink by apache.

the class OneInputStreamTaskTest method testWatermarkMetrics.

@Test
@SuppressWarnings("unchecked")
public void testWatermarkMetrics() throws Exception {
    final OneInputStreamTaskTestHarness<String, String> testHarness = new OneInputStreamTaskTestHarness<>(OneInputStreamTask::new, BasicTypeInfo.STRING_TYPE_INFO, BasicTypeInfo.STRING_TYPE_INFO);
    OneInputStreamOperator<String, String> headOperator = new WatermarkMetricOperator();
    OperatorID headOperatorId = new OperatorID();
    OneInputStreamOperator<String, String> chainedOperator = new WatermarkMetricOperator();
    OperatorID chainedOperatorId = new OperatorID();
    testHarness.setupOperatorChain(headOperatorId, headOperator).chain(chainedOperatorId, chainedOperator, BasicTypeInfo.STRING_TYPE_INFO.createSerializer(new ExecutionConfig())).finish();
    InterceptingOperatorMetricGroup headOperatorMetricGroup = new InterceptingOperatorMetricGroup();
    InterceptingOperatorMetricGroup chainedOperatorMetricGroup = new InterceptingOperatorMetricGroup();
    InterceptingTaskMetricGroup taskMetricGroup = new InterceptingTaskMetricGroup() {

        @Override
        public InternalOperatorMetricGroup getOrAddOperator(OperatorID id, String name) {
            if (id.equals(headOperatorId)) {
                return headOperatorMetricGroup;
            } else if (id.equals(chainedOperatorId)) {
                return chainedOperatorMetricGroup;
            } else {
                return super.getOrAddOperator(id, name);
            }
        }
    };
    StreamMockEnvironment env = new StreamMockEnvironment(testHarness.jobConfig, testHarness.taskConfig, testHarness.memorySize, new MockInputSplitProvider(), testHarness.bufferSize, new TestTaskStateManager()) {

        @Override
        public TaskMetricGroup getMetricGroup() {
            return taskMetricGroup;
        }
    };
    testHarness.invoke(env);
    testHarness.waitForTaskRunning();
    Gauge<Long> taskInputWatermarkGauge = (Gauge<Long>) taskMetricGroup.get(MetricNames.IO_CURRENT_INPUT_WATERMARK);
    Gauge<Long> headInputWatermarkGauge = (Gauge<Long>) headOperatorMetricGroup.get(MetricNames.IO_CURRENT_INPUT_WATERMARK);
    Gauge<Long> headOutputWatermarkGauge = (Gauge<Long>) headOperatorMetricGroup.get(MetricNames.IO_CURRENT_OUTPUT_WATERMARK);
    Gauge<Long> chainedInputWatermarkGauge = (Gauge<Long>) chainedOperatorMetricGroup.get(MetricNames.IO_CURRENT_INPUT_WATERMARK);
    Gauge<Long> chainedOutputWatermarkGauge = (Gauge<Long>) chainedOperatorMetricGroup.get(MetricNames.IO_CURRENT_OUTPUT_WATERMARK);
    Assert.assertEquals("A metric was registered multiple times.", 5, new HashSet<>(Arrays.asList(taskInputWatermarkGauge, headInputWatermarkGauge, headOutputWatermarkGauge, chainedInputWatermarkGauge, chainedOutputWatermarkGauge)).size());
    Assert.assertEquals(Long.MIN_VALUE, taskInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, headInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, headOutputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, chainedInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(Long.MIN_VALUE, chainedOutputWatermarkGauge.getValue().longValue());
    testHarness.processElement(new Watermark(1L));
    testHarness.waitForInputProcessing();
    Assert.assertEquals(1L, taskInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(1L, headInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(2L, headOutputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(2L, chainedInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(4L, chainedOutputWatermarkGauge.getValue().longValue());
    testHarness.processElement(new Watermark(2L));
    testHarness.waitForInputProcessing();
    Assert.assertEquals(2L, taskInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(2L, headInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(4L, headOutputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(4L, chainedInputWatermarkGauge.getValue().longValue());
    Assert.assertEquals(8L, chainedOutputWatermarkGauge.getValue().longValue());
    testHarness.endInput();
    testHarness.waitForTaskCompletion();
}
Also used : OperatorID(org.apache.flink.runtime.jobgraph.OperatorID) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) Gauge(org.apache.flink.metrics.Gauge) TestTaskStateManager(org.apache.flink.runtime.state.TestTaskStateManager) InterceptingOperatorMetricGroup(org.apache.flink.runtime.metrics.util.InterceptingOperatorMetricGroup) InterceptingTaskMetricGroup(org.apache.flink.runtime.metrics.util.InterceptingTaskMetricGroup) MockInputSplitProvider(org.apache.flink.runtime.operators.testutils.MockInputSplitProvider) Watermark(org.apache.flink.streaming.api.watermark.Watermark) HashSet(java.util.HashSet) Test(org.junit.Test)

Aggregations

ExecutionConfig (org.apache.flink.api.common.ExecutionConfig)3 Gauge (org.apache.flink.metrics.Gauge)3 OperatorID (org.apache.flink.runtime.jobgraph.OperatorID)3 InterceptingOperatorMetricGroup (org.apache.flink.runtime.metrics.util.InterceptingOperatorMetricGroup)3 InterceptingTaskMetricGroup (org.apache.flink.runtime.metrics.util.InterceptingTaskMetricGroup)3 Watermark (org.apache.flink.streaming.api.watermark.Watermark)3 Test (org.junit.Test)3 HashSet (java.util.HashSet)2 MockInputSplitProvider (org.apache.flink.runtime.operators.testutils.MockInputSplitProvider)2 TestTaskStateManager (org.apache.flink.runtime.state.TestTaskStateManager)2 MockSource (org.apache.flink.api.connector.source.mocks.MockSource)1 CoStreamMap (org.apache.flink.streaming.api.operators.co.CoStreamMap)1 WatermarkMetricOperator (org.apache.flink.streaming.runtime.tasks.OneInputStreamTaskTest.WatermarkMetricOperator)1
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial