Examples with TransformWatermarks org.apache.beam.runners.direct.WatermarkManager.TransformWatermarks used on opensource projects

Example 21 with TransformWatermarks

use of org.apache.beam.runners.direct.WatermarkManager.TransformWatermarks in project beam by apache.

the class EvaluationContext method commitBundles.

private Iterable<? extends CommittedBundle<?>> commitBundles(Iterable<? extends UncommittedBundle<?>> bundles) {
    ImmutableList.Builder<CommittedBundle<?>> completed = ImmutableList.builder();
    for (UncommittedBundle<?> inProgress : bundles) {
        AppliedPTransform<?, ?, ?> producing = graph.getProducer(inProgress.getPCollection());
        TransformWatermarks watermarks = watermarkManager.getWatermarks(producing);
        CommittedBundle<?> committed = inProgress.commit(watermarks.getSynchronizedProcessingOutputTime());
        // filter them out
        if (!Iterables.isEmpty(committed.getElements())) {
            completed.add(committed);
        }
    }
    return completed.build();
}

Example 22 with TransformWatermarks

use of org.apache.beam.runners.direct.WatermarkManager.TransformWatermarks in project beam by apache.

the class EvaluationContext method fireAvailableCallbacks.

private void fireAvailableCallbacks(AppliedPTransform<?, ?, ?> producingTransform) {
    TransformWatermarks watermarks = watermarkManager.getWatermarks(producingTransform);
    Instant outputWatermark = watermarks.getOutputWatermark();
    callbackExecutor.fireForWatermark(producingTransform, outputWatermark);
}