Examples with SegmentWithRange io.pravega.shared.watermarks.SegmentWithRange used on opensource projects

Example 1 with SegmentWithRange

use of io.pravega.shared.watermarks.SegmentWithRange in project pravega by pravega.

the class WatermarkSerializerTest method testWatermark.

@Test
public void testWatermark() {
    SegmentWithRange segmentWithRange1 = new SegmentWithRange(0L, 0.0, 0.5);
    SegmentWithRange segmentWithRange2 = new SegmentWithRange(1L, 0.5, 1.0);
    ImmutableMap<SegmentWithRange, Long> map = ImmutableMap.of(segmentWithRange1, 1L, segmentWithRange2, 1L);
    Watermark watermark = new Watermark(0L, 1L, map);
    WatermarkSerializer serializer = new WatermarkSerializer();
    ByteBuffer serialized = serializer.serialize(watermark);
    Watermark deserialized = serializer.deserialize(serialized);
    assertEquals(watermark, deserialized);
}

Example 2 with SegmentWithRange

use of io.pravega.shared.watermarks.SegmentWithRange in project pravega by pravega.

the class PeriodicWatermarking method computeWatermark.

/**
 * This method takes marks (time + position) of active writers and finds greatest lower bound on time and
 * least upper bound on positions and returns the watermark object composed of the two.
 * The least upper bound computed from positions may not result in a consistent and complete stream cut.
 * So, a positional upper bound is then converted into a stream cut by including segments from higher epoch.
 * Also, it is possible that in an effort to fill missing range, we may end up creating an upper bound that
 * is composed of segments from highest epoch. In next iteration, from new writer positions, we may be able to
 * compute a tighter upper bound. But since watermark has to advance position and time, we will take the upper bound
 * of previous stream cut and new stream cut.
 *
 * @param scope scope
 * @param streamName stream name
 * @param context operation context
 * @param activeWriters marks for all active writers.
 * @param previousWatermark previous watermark that was emitted.
 * @return CompletableFuture which when completed will contain watermark to be emitted.
 */
private CompletableFuture<Watermark> computeWatermark(String scope, String streamName, OperationContext context, List<Map.Entry<String, WriterMark>> activeWriters, Watermark previousWatermark) {
    long requestId = context.getRequestId();
    Watermark.WatermarkBuilder builder = Watermark.builder();
    ConcurrentHashMap<SegmentWithRange, Long> upperBound = new ConcurrentHashMap<>();
    // We are deliberately making two passes over writers - first to find lowest time. Second loop will convert writer
    // positions to StreamSegmentRecord objects by retrieving ranges from store. And then perform computation on those
    // objects.
    LongSummaryStatistics summarized = activeWriters.stream().collect(Collectors.summarizingLong(x -> x.getValue().getTimestamp()));
    long lowerBoundOnTime = summarized.getMin();
    long upperBoundOnTime = summarized.getMax();
    if (lowerBoundOnTime > previousWatermark.getLowerTimeBound()) {
        CompletableFuture<List<Map<SegmentWithRange, Long>>> positionsFuture = Futures.allOfWithResults(activeWriters.stream().map(x -> {
            return Futures.keysAllOfWithResults(x.getValue().getPosition().entrySet().stream().collect(Collectors.toMap(y -> getSegmentWithRange(scope, streamName, context, y.getKey()), Entry::getValue)));
        }).collect(Collectors.toList()));
        log.debug(requestId, "Emitting watermark for stream {}/{} with time {}", scope, streamName, lowerBoundOnTime);
        return positionsFuture.thenAccept(listOfPositions -> listOfPositions.forEach(position -> {
            // add writer positions to upperBound map.
            addToUpperBound(position, upperBound);
        })).thenCompose(v -> computeStreamCut(scope, streamName, context, upperBound, previousWatermark).thenApply(streamCut -> builder.lowerTimeBound(lowerBoundOnTime).upperTimeBound(upperBoundOnTime).streamCut(ImmutableMap.copyOf(streamCut)).build()));
    } else {
        // new time is not advanced. No watermark to be emitted.
        return CompletableFuture.completedFuture(null);
    }
}

Example 3 with SegmentWithRange

use of io.pravega.shared.watermarks.SegmentWithRange in project pravega by pravega.

the class PeriodicWatermarking method addToUpperBound.

/**
 * Method that updates the supplied upperBound by comparing it with supplied position such that resultant upperbound
 * is an upper bound on current position and all previously considered positions.
 * This method should be called with each writer's position iteratively and it will update the upperBound accordingly.
 * Note: This method is not thread safe, even though upperBound is a concurrent hash map.
 * This method looks at the state in the map and then either adds or removes entries from the map.
 * If this was called concurrently, then the behaviour is unpredictable.
 * @param position position be included while computing new upper bound
 * @param upperBound existing upper bound
 */
private void addToUpperBound(Map<SegmentWithRange, Long> position, Map<SegmentWithRange, Long> upperBound) {
    for (Map.Entry<SegmentWithRange, Long> writerPos : position.entrySet()) {
        SegmentWithRange segment = writerPos.getKey();
        long offset = writerPos.getValue();
        if (upperBound.containsKey(segment)) {
            // update offset if the segment is already present.
            long newOffset = Math.max(offset, upperBound.get(segment));
            upperBound.put(segment, newOffset);
        } else if (!hasSuccessors(segment, upperBound.keySet())) {
            // only include segment if it doesnt have a successor already included in the set.
            Set<SegmentWithRange> included = upperBound.keySet();
            included.forEach(x -> {
                // remove all predecessors of `segment` from upper bound.
                if (segment.overlaps(x) && segment.getSegmentId() > x.getSegmentId()) {
                    upperBound.remove(x);
                }
            });
            // add segment to upperBound.
            upperBound.put(segment, offset);
        }
    }
}