Search in sources :

Example 66 with Instant

use of org.joda.time.Instant in project beam by apache.

the class AfterDelayFromFirstElementStateMachine method onMerge.

@Override
public void onMerge(OnMergeContext c) throws Exception {
    // If the trigger is already finished, there is no way it will become re-activated
    if (c.trigger().isFinished()) {
        StateMerging.clear(c.state(), DELAYED_UNTIL_TAG);
        // NOTE: We do not attempt to delete  the timers.
        return;
    }
    // Determine the earliest point across all the windows, and delay to that.
    StateMerging.mergeCombiningValues(c.state(), DELAYED_UNTIL_TAG);
    Instant earliestTargetTime = c.state().access(DELAYED_UNTIL_TAG).read();
    if (earliestTargetTime != null) {
        c.setTimer(earliestTargetTime, timeDomain);
    }
}
Also used : Instant(org.joda.time.Instant)

Example 67 with Instant

use of org.joda.time.Instant in project beam by apache.

the class AfterDelayFromFirstElementStateMachine method onElement.

@Override
public void onElement(OnElementContext c) throws Exception {
    GroupingState<Instant, Instant> delayUntilState = c.state().access(DELAYED_UNTIL_TAG);
    Instant oldDelayUntil = delayUntilState.read();
    // ignore anyhow, we don't bother with it if it is already set.
    if (oldDelayUntil != null) {
        return;
    }
    Instant targetTimestamp = getTargetTimestamp(c);
    delayUntilState.add(targetTimestamp);
    c.setTimer(targetTimestamp, timeDomain);
}
Also used : Instant(org.joda.time.Instant)

Example 68 with Instant

use of org.joda.time.Instant in project beam by apache.

the class InMemoryStateInternalsTest method testMergeEarliestWatermarkIntoSource.

@Test
public void testMergeEarliestWatermarkIntoSource() throws Exception {
    WatermarkHoldState value1 = underTest.state(NAMESPACE_1, WATERMARK_EARLIEST_ADDR);
    WatermarkHoldState value2 = underTest.state(NAMESPACE_2, WATERMARK_EARLIEST_ADDR);
    value1.add(new Instant(3000));
    value2.add(new Instant(5000));
    value1.add(new Instant(4000));
    value2.add(new Instant(2000));
    // Merging clears the old values and updates the merged value.
    StateMerging.mergeWatermarks(Arrays.asList(value1, value2), value1, WINDOW_1);
    assertThat(value1.read(), equalTo(new Instant(2000)));
    assertThat(value2.read(), equalTo(null));
}
Also used : Instant(org.joda.time.Instant) WatermarkHoldState(org.apache.beam.sdk.state.WatermarkHoldState) Test(org.junit.Test)

Example 69 with Instant

use of org.joda.time.Instant in project beam by apache.

the class InMemoryStateInternalsTest method testWatermarkLatestState.

@Test
public void testWatermarkLatestState() throws Exception {
    WatermarkHoldState value = underTest.state(NAMESPACE_1, WATERMARK_LATEST_ADDR);
    // State instances are cached, but depend on the namespace.
    assertEquals(value, underTest.state(NAMESPACE_1, WATERMARK_LATEST_ADDR));
    assertFalse(value.equals(underTest.state(NAMESPACE_2, WATERMARK_LATEST_ADDR)));
    assertThat(value.read(), Matchers.nullValue());
    value.add(new Instant(2000));
    assertThat(value.read(), equalTo(new Instant(2000)));
    value.add(new Instant(3000));
    assertThat(value.read(), equalTo(new Instant(3000)));
    value.add(new Instant(1000));
    assertThat(value.read(), equalTo(new Instant(3000)));
    value.clear();
    assertThat(value.read(), equalTo(null));
    assertThat(underTest.state(NAMESPACE_1, WATERMARK_LATEST_ADDR), Matchers.sameInstance(value));
}
Also used : Instant(org.joda.time.Instant) WatermarkHoldState(org.apache.beam.sdk.state.WatermarkHoldState) Test(org.junit.Test)

Example 70 with Instant

use of org.joda.time.Instant in project beam by apache.

the class InMemoryStateInternalsTest method testWatermarkEarliestState.

@Test
public void testWatermarkEarliestState() throws Exception {
    WatermarkHoldState value = underTest.state(NAMESPACE_1, WATERMARK_EARLIEST_ADDR);
    // State instances are cached, but depend on the namespace.
    assertEquals(value, underTest.state(NAMESPACE_1, WATERMARK_EARLIEST_ADDR));
    assertFalse(value.equals(underTest.state(NAMESPACE_2, WATERMARK_EARLIEST_ADDR)));
    assertThat(value.read(), Matchers.nullValue());
    value.add(new Instant(2000));
    assertThat(value.read(), equalTo(new Instant(2000)));
    value.add(new Instant(3000));
    assertThat(value.read(), equalTo(new Instant(2000)));
    value.add(new Instant(1000));
    assertThat(value.read(), equalTo(new Instant(1000)));
    value.clear();
    assertThat(value.read(), equalTo(null));
    assertThat(underTest.state(NAMESPACE_1, WATERMARK_EARLIEST_ADDR), Matchers.sameInstance(value));
}
Also used : Instant(org.joda.time.Instant) WatermarkHoldState(org.apache.beam.sdk.state.WatermarkHoldState) Test(org.junit.Test)

Aggregations

Instant (org.joda.time.Instant)410 Test (org.junit.Test)326 IntervalWindow (org.apache.beam.sdk.transforms.windowing.IntervalWindow)135 KV (org.apache.beam.sdk.values.KV)64 Category (org.junit.experimental.categories.Category)60 WindowedValue (org.apache.beam.sdk.util.WindowedValue)47 Duration (org.joda.time.Duration)44 ReadableInstant (org.joda.time.ReadableInstant)36 BoundedWindow (org.apache.beam.sdk.transforms.windowing.BoundedWindow)30 WatermarkHoldState (org.apache.beam.sdk.state.WatermarkHoldState)24 TimerData (org.apache.beam.runners.core.TimerInternals.TimerData)22 Matchers.emptyIterable (org.hamcrest.Matchers.emptyIterable)22 HashMap (java.util.HashMap)19 TransformWatermarks (org.apache.beam.runners.direct.WatermarkManager.TransformWatermarks)18 StateNamespaceForTest (org.apache.beam.runners.core.StateNamespaceForTest)17 WindowMatchers.isSingleWindowedValue (org.apache.beam.runners.core.WindowMatchers.isSingleWindowedValue)17 WindowMatchers.isWindowedValue (org.apache.beam.runners.core.WindowMatchers.isWindowedValue)17 TupleTag (org.apache.beam.sdk.values.TupleTag)14 ArrayList (java.util.ArrayList)13 Map (java.util.Map)12