Search in sources :

Example 11 with FixedWindows

use of org.apache.beam.sdk.transforms.windowing.FixedWindows in project beam by apache.

the class ParDoTest method testWindowingInStartAndFinishBundle.

@Test
@Category(ValidatesRunner.class)
public void testWindowingInStartAndFinishBundle() {
    final FixedWindows windowFn = FixedWindows.of(Duration.millis(1));
    PCollection<String> output = pipeline.apply(Create.timestamped(TimestampedValue.of("elem", new Instant(1)))).apply(Window.<String>into(windowFn)).apply(ParDo.of(new DoFn<String, String>() {

        @ProcessElement
        public void processElement(ProcessContext c) {
            c.output(c.element());
            System.out.println("Process: " + c.element() + ":" + c.timestamp().getMillis());
        }

        @FinishBundle
        public void finishBundle(FinishBundleContext c) {
            Instant ts = new Instant(3);
            c.output("finish", ts, windowFn.assignWindow(ts));
            System.out.println("Finish: 3");
        }
    })).apply(ParDo.of(new PrintingDoFn()));
    PAssert.that(output).satisfies(new Checker());
    pipeline.run();
}
Also used : FixedWindows(org.apache.beam.sdk.transforms.windowing.FixedWindows) Instant(org.joda.time.Instant) ProcessElement(org.apache.beam.sdk.transforms.DoFn.ProcessElement) StringUtils.byteArrayToJsonString(org.apache.beam.sdk.util.StringUtils.byteArrayToJsonString) Matchers.containsString(org.hamcrest.Matchers.containsString) Category(org.junit.experimental.categories.Category) Test(org.junit.Test)

Aggregations

FixedWindows (org.apache.beam.sdk.transforms.windowing.FixedWindows)11 Instant (org.joda.time.Instant)11 Test (org.junit.Test)11 Duration (org.joda.time.Duration)6 IntervalWindow (org.apache.beam.sdk.transforms.windowing.IntervalWindow)5 Category (org.junit.experimental.categories.Category)5 StreamingTest (org.apache.beam.runners.spark.StreamingTest)3 ReadableInstant (org.joda.time.ReadableInstant)3 ProcessElement (org.apache.beam.sdk.transforms.DoFn.ProcessElement)1 StringUtils.byteArrayToJsonString (org.apache.beam.sdk.util.StringUtils.byteArrayToJsonString)1 Matchers.containsString (org.hamcrest.Matchers.containsString)1