Examples with Watermark org.apache.flink.streaming.api.watermark.Watermark used on opensource projects

Example 31 with Watermark

use of org.apache.flink.streaming.api.watermark.Watermark in project flink by apache.

the class IngestionTimeExtractorTest method testMonotonousTimestamps.

@Test
public void testMonotonousTimestamps() {
    AssignerWithPeriodicWatermarks<String> assigner = new IngestionTimeExtractor<>();
    long maxRecordSoFar = 0L;
    long maxWatermarkSoFar = 0L;
    for (int i = 0; i < 1343; i++) {
        if (i % 7 == 1) {
            Watermark mark = assigner.getCurrentWatermark();
            assertNotNull(mark);
            // increasing watermarks
            assertTrue(mark.getTimestamp() >= maxWatermarkSoFar);
            maxWatermarkSoFar = mark.getTimestamp();
            // tight watermarks
            assertTrue(mark.getTimestamp() >= maxRecordSoFar - 1);
        } else {
            long next = assigner.extractTimestamp("a", Long.MIN_VALUE);
            // increasing timestamps
            assertTrue(next >= maxRecordSoFar);
            // timestamps are never below or at the watermark
            assertTrue(next > maxWatermarkSoFar);
            maxRecordSoFar = next;
        }
        if (i % 9 == 0) {
            try {
                Thread.sleep(1);
            } catch (InterruptedException ignored) {
            }
        }
    }
}

Example 32 with Watermark

use of org.apache.flink.streaming.api.watermark.Watermark in project flink by apache.

the class ExtractTimestampsOperator method processElement.

@Override
public void processElement(StreamRecord<T> element) throws Exception {
    long newTimestamp = userFunction.extractTimestamp(element.getValue(), element.getTimestamp());
    output.collect(element.replace(element.getValue(), newTimestamp));
    long watermark = userFunction.extractWatermark(element.getValue(), newTimestamp);
    if (watermark > currentWatermark) {
        currentWatermark = watermark;
        output.emitWatermark(new Watermark(currentWatermark));
    }
}

Example 33 with Watermark

use of org.apache.flink.streaming.api.watermark.Watermark in project flink by apache.

the class TimestampsAndPeriodicWatermarksOperator method onProcessingTime.

@Override
public void onProcessingTime(long timestamp) throws Exception {
    // register next timer
    Watermark newWatermark = userFunction.getCurrentWatermark();
    if (newWatermark != null && newWatermark.getTimestamp() > currentWatermark) {
        currentWatermark = newWatermark.getTimestamp();
        // emit watermark
        output.emitWatermark(newWatermark);
    }
    long now = getProcessingTimeService().getCurrentProcessingTime();
    getProcessingTimeService().registerTimer(now + watermarkInterval, this);
}

Example 34 with Watermark

use of org.apache.flink.streaming.api.watermark.Watermark in project flink by apache.

the class TimestampsAndPeriodicWatermarksOperator method close.

@Override
public void close() throws Exception {
    super.close();
    // emit a final watermark
    Watermark newWatermark = userFunction.getCurrentWatermark();
    if (newWatermark != null && newWatermark.getTimestamp() > currentWatermark) {
        currentWatermark = newWatermark.getTimestamp();
        // emit watermark
        output.emitWatermark(newWatermark);
    }
}

Example 35 with Watermark

use of org.apache.flink.streaming.api.watermark.Watermark in project flink by apache.

the class TimestampsAndPunctuatedWatermarksOperator method processElement.

@Override
public void processElement(StreamRecord<T> element) throws Exception {
    final T value = element.getValue();
    final long newTimestamp = userFunction.extractTimestamp(value, element.hasTimestamp() ? element.getTimestamp() : Long.MIN_VALUE);
    output.collect(element.replace(element.getValue(), newTimestamp));
    final Watermark nextWatermark = userFunction.checkAndGetNextWatermark(value, newTimestamp);
    if (nextWatermark != null && nextWatermark.getTimestamp() > currentWatermark) {
        currentWatermark = nextWatermark.getTimestamp();
        output.emitWatermark(nextWatermark);
    }
}