Search in sources :

Example 1 with InternalTimerService

use of org.apache.flink.streaming.api.operators.InternalTimerService in project flink by apache.

the class BatchExecutionInternalTimeServiceTest method testProcessingTimeTimers.

@Test
public void testProcessingTimeTimers() {
    BatchExecutionKeyedStateBackend<Integer> keyedStatedBackend = new BatchExecutionKeyedStateBackend<>(KEY_SERIALIZER, new KeyGroupRange(0, 1));
    TestProcessingTimeService processingTimeService = new TestProcessingTimeService();
    InternalTimeServiceManager<Integer> timeServiceManager = BatchExecutionInternalTimeServiceManager.create(keyedStatedBackend, this.getClass().getClassLoader(), new DummyKeyContext(), processingTimeService, Collections.emptyList());
    List<Long> timers = new ArrayList<>();
    InternalTimerService<VoidNamespace> timerService = timeServiceManager.getInternalTimerService("test", KEY_SERIALIZER, new VoidNamespaceSerializer(), LambdaTrigger.processingTimeTrigger(timer -> timers.add(timer.getTimestamp())));
    keyedStatedBackend.setCurrentKey(1);
    timerService.registerProcessingTimeTimer(VoidNamespace.INSTANCE, 150);
    // we should never register physical timers
    assertThat(processingTimeService.getNumActiveTimers(), equalTo(0));
    // changing the current key fires all timers
    keyedStatedBackend.setCurrentKey(2);
    assertThat(timers, equalTo(Collections.singletonList(150L)));
}
Also used : VoidNamespace(org.apache.flink.runtime.state.VoidNamespace) InternalTimeServiceManager(org.apache.flink.streaming.api.operators.InternalTimeServiceManager) Arrays(java.util.Arrays) UnregisteredMetricsGroup(org.apache.flink.metrics.groups.UnregisteredMetricsGroup) CoreMatchers.equalTo(org.hamcrest.CoreMatchers.equalTo) Watermark(org.apache.flink.streaming.api.watermark.Watermark) ArrayList(java.util.ArrayList) Assert.assertThat(org.junit.Assert.assertThat) IntSerializer(org.apache.flink.api.common.typeutils.base.IntSerializer) VoidNamespaceSerializer(org.apache.flink.runtime.state.VoidNamespaceSerializer) CloseableRegistry(org.apache.flink.core.fs.CloseableRegistry) InternalTimerService(org.apache.flink.streaming.api.operators.InternalTimerService) TestLogger(org.apache.flink.util.TestLogger) BiConsumer(java.util.function.BiConsumer) Assert.fail(org.junit.Assert.fail) MockEnvironment(org.apache.flink.runtime.operators.testutils.MockEnvironment) ExpectedException(org.junit.rules.ExpectedException) KeyGroupRange(org.apache.flink.runtime.state.KeyGroupRange) KeyContext(org.apache.flink.streaming.api.operators.KeyContext) Triggerable(org.apache.flink.streaming.api.operators.Triggerable) InternalTimer(org.apache.flink.streaming.api.operators.InternalTimer) Test(org.junit.Test) MemoryStateBackend(org.apache.flink.runtime.state.memory.MemoryStateBackend) TestProcessingTimeService(org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService) Consumer(java.util.function.Consumer) TtlTimeProvider(org.apache.flink.runtime.state.ttl.TtlTimeProvider) List(java.util.List) JobID(org.apache.flink.api.common.JobID) Rule(org.junit.Rule) AbstractKeyedStateBackend(org.apache.flink.runtime.state.AbstractKeyedStateBackend) Assert(org.junit.Assert) Collections(java.util.Collections) KeyGroupRange(org.apache.flink.runtime.state.KeyGroupRange) ArrayList(java.util.ArrayList) TestProcessingTimeService(org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService) VoidNamespace(org.apache.flink.runtime.state.VoidNamespace) VoidNamespaceSerializer(org.apache.flink.runtime.state.VoidNamespaceSerializer) Test(org.junit.Test)

Example 2 with InternalTimerService

use of org.apache.flink.streaming.api.operators.InternalTimerService in project flink by apache.

the class LegacyKeyedCoProcessOperator method open.

@Override
public void open() throws Exception {
    super.open();
    collector = new TimestampedCollector<>(output);
    InternalTimerService<VoidNamespace> internalTimerService = getInternalTimerService("user-timers", VoidNamespaceSerializer.INSTANCE, this);
    TimerService timerService = new SimpleTimerService(internalTimerService);
    context = new ContextImpl<>(userFunction, timerService);
    onTimerContext = new OnTimerContextImpl<>(userFunction, timerService);
}
Also used : SimpleTimerService(org.apache.flink.streaming.api.SimpleTimerService) VoidNamespace(org.apache.flink.runtime.state.VoidNamespace) TimerService(org.apache.flink.streaming.api.TimerService) SimpleTimerService(org.apache.flink.streaming.api.SimpleTimerService) InternalTimerService(org.apache.flink.streaming.api.operators.InternalTimerService)

Example 3 with InternalTimerService

use of org.apache.flink.streaming.api.operators.InternalTimerService in project flink by apache.

the class KeyedStateBootstrapOperator method open.

@Override
public void open() throws Exception {
    super.open();
    Supplier<InternalTimerService<VoidNamespace>> internalTimerService = () -> getInternalTimerService("user-timers", VoidNamespaceSerializer.INSTANCE, VoidTriggerable.instance());
    TimerService timerService = new LazyTimerService(internalTimerService, getProcessingTimeService());
    context = new KeyedStateBootstrapOperator<K, IN>.ContextImpl(userFunction, timerService);
}
Also used : InternalTimerService(org.apache.flink.streaming.api.operators.InternalTimerService) TimerService(org.apache.flink.streaming.api.TimerService) InternalTimerService(org.apache.flink.streaming.api.operators.InternalTimerService)

Example 4 with InternalTimerService

use of org.apache.flink.streaming.api.operators.InternalTimerService in project flink by apache.

the class BatchExecutionInternalTimeServiceTest method testSettingSameKeyDoesNotFireTimers.

@Test
public void testSettingSameKeyDoesNotFireTimers() {
    BatchExecutionKeyedStateBackend<Integer> keyedStatedBackend = new BatchExecutionKeyedStateBackend<>(KEY_SERIALIZER, new KeyGroupRange(0, 1));
    InternalTimeServiceManager<Integer> timeServiceManager = BatchExecutionInternalTimeServiceManager.create(keyedStatedBackend, this.getClass().getClassLoader(), new DummyKeyContext(), new TestProcessingTimeService(), Collections.emptyList());
    List<Long> timers = new ArrayList<>();
    InternalTimerService<VoidNamespace> timerService = timeServiceManager.getInternalTimerService("test", KEY_SERIALIZER, new VoidNamespaceSerializer(), LambdaTrigger.eventTimeTrigger(timer -> timers.add(timer.getTimestamp())));
    keyedStatedBackend.setCurrentKey(1);
    timerService.registerEventTimeTimer(VoidNamespace.INSTANCE, 123);
    keyedStatedBackend.setCurrentKey(1);
    assertThat(timers, equalTo(Collections.emptyList()));
}
Also used : VoidNamespace(org.apache.flink.runtime.state.VoidNamespace) InternalTimeServiceManager(org.apache.flink.streaming.api.operators.InternalTimeServiceManager) Arrays(java.util.Arrays) UnregisteredMetricsGroup(org.apache.flink.metrics.groups.UnregisteredMetricsGroup) CoreMatchers.equalTo(org.hamcrest.CoreMatchers.equalTo) Watermark(org.apache.flink.streaming.api.watermark.Watermark) ArrayList(java.util.ArrayList) Assert.assertThat(org.junit.Assert.assertThat) IntSerializer(org.apache.flink.api.common.typeutils.base.IntSerializer) VoidNamespaceSerializer(org.apache.flink.runtime.state.VoidNamespaceSerializer) CloseableRegistry(org.apache.flink.core.fs.CloseableRegistry) InternalTimerService(org.apache.flink.streaming.api.operators.InternalTimerService) TestLogger(org.apache.flink.util.TestLogger) BiConsumer(java.util.function.BiConsumer) Assert.fail(org.junit.Assert.fail) MockEnvironment(org.apache.flink.runtime.operators.testutils.MockEnvironment) ExpectedException(org.junit.rules.ExpectedException) KeyGroupRange(org.apache.flink.runtime.state.KeyGroupRange) KeyContext(org.apache.flink.streaming.api.operators.KeyContext) Triggerable(org.apache.flink.streaming.api.operators.Triggerable) InternalTimer(org.apache.flink.streaming.api.operators.InternalTimer) Test(org.junit.Test) MemoryStateBackend(org.apache.flink.runtime.state.memory.MemoryStateBackend) TestProcessingTimeService(org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService) Consumer(java.util.function.Consumer) TtlTimeProvider(org.apache.flink.runtime.state.ttl.TtlTimeProvider) List(java.util.List) JobID(org.apache.flink.api.common.JobID) Rule(org.junit.Rule) AbstractKeyedStateBackend(org.apache.flink.runtime.state.AbstractKeyedStateBackend) Assert(org.junit.Assert) Collections(java.util.Collections) KeyGroupRange(org.apache.flink.runtime.state.KeyGroupRange) ArrayList(java.util.ArrayList) TestProcessingTimeService(org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService) VoidNamespace(org.apache.flink.runtime.state.VoidNamespace) VoidNamespaceSerializer(org.apache.flink.runtime.state.VoidNamespaceSerializer) Test(org.junit.Test)

Example 5 with InternalTimerService

use of org.apache.flink.streaming.api.operators.InternalTimerService in project flink by apache.

the class BatchExecutionInternalTimeServiceTest method testFiringEventTimeTimers.

@Test
public void testFiringEventTimeTimers() throws Exception {
    BatchExecutionKeyedStateBackend<Integer> keyedStatedBackend = new BatchExecutionKeyedStateBackend<>(KEY_SERIALIZER, new KeyGroupRange(0, 1));
    InternalTimeServiceManager<Integer> timeServiceManager = BatchExecutionInternalTimeServiceManager.create(keyedStatedBackend, this.getClass().getClassLoader(), new DummyKeyContext(), new TestProcessingTimeService(), Collections.emptyList());
    List<Long> timers = new ArrayList<>();
    InternalTimerService<VoidNamespace> timerService = timeServiceManager.getInternalTimerService("test", KEY_SERIALIZER, new VoidNamespaceSerializer(), LambdaTrigger.eventTimeTrigger(timer -> timers.add(timer.getTimestamp())));
    keyedStatedBackend.setCurrentKey(1);
    timerService.registerEventTimeTimer(VoidNamespace.INSTANCE, 123);
    // advancing the watermark should not fire timers
    timeServiceManager.advanceWatermark(new Watermark(1000));
    timerService.deleteEventTimeTimer(VoidNamespace.INSTANCE, 123);
    timerService.registerEventTimeTimer(VoidNamespace.INSTANCE, 150);
    // changing the current key fires all timers
    keyedStatedBackend.setCurrentKey(2);
    assertThat(timers, equalTo(Collections.singletonList(150L)));
}
Also used : VoidNamespace(org.apache.flink.runtime.state.VoidNamespace) InternalTimeServiceManager(org.apache.flink.streaming.api.operators.InternalTimeServiceManager) Arrays(java.util.Arrays) UnregisteredMetricsGroup(org.apache.flink.metrics.groups.UnregisteredMetricsGroup) CoreMatchers.equalTo(org.hamcrest.CoreMatchers.equalTo) Watermark(org.apache.flink.streaming.api.watermark.Watermark) ArrayList(java.util.ArrayList) Assert.assertThat(org.junit.Assert.assertThat) IntSerializer(org.apache.flink.api.common.typeutils.base.IntSerializer) VoidNamespaceSerializer(org.apache.flink.runtime.state.VoidNamespaceSerializer) CloseableRegistry(org.apache.flink.core.fs.CloseableRegistry) InternalTimerService(org.apache.flink.streaming.api.operators.InternalTimerService) TestLogger(org.apache.flink.util.TestLogger) BiConsumer(java.util.function.BiConsumer) Assert.fail(org.junit.Assert.fail) MockEnvironment(org.apache.flink.runtime.operators.testutils.MockEnvironment) ExpectedException(org.junit.rules.ExpectedException) KeyGroupRange(org.apache.flink.runtime.state.KeyGroupRange) KeyContext(org.apache.flink.streaming.api.operators.KeyContext) Triggerable(org.apache.flink.streaming.api.operators.Triggerable) InternalTimer(org.apache.flink.streaming.api.operators.InternalTimer) Test(org.junit.Test) MemoryStateBackend(org.apache.flink.runtime.state.memory.MemoryStateBackend) TestProcessingTimeService(org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService) Consumer(java.util.function.Consumer) TtlTimeProvider(org.apache.flink.runtime.state.ttl.TtlTimeProvider) List(java.util.List) JobID(org.apache.flink.api.common.JobID) Rule(org.junit.Rule) AbstractKeyedStateBackend(org.apache.flink.runtime.state.AbstractKeyedStateBackend) Assert(org.junit.Assert) Collections(java.util.Collections) KeyGroupRange(org.apache.flink.runtime.state.KeyGroupRange) ArrayList(java.util.ArrayList) TestProcessingTimeService(org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService) VoidNamespace(org.apache.flink.runtime.state.VoidNamespace) VoidNamespaceSerializer(org.apache.flink.runtime.state.VoidNamespaceSerializer) Watermark(org.apache.flink.streaming.api.watermark.Watermark) Test(org.junit.Test)

Aggregations

InternalTimerService (org.apache.flink.streaming.api.operators.InternalTimerService)7 VoidNamespace (org.apache.flink.runtime.state.VoidNamespace)6 TimerService (org.apache.flink.streaming.api.TimerService)4 ArrayList (java.util.ArrayList)3 Arrays (java.util.Arrays)3 Collections (java.util.Collections)3 List (java.util.List)3 BiConsumer (java.util.function.BiConsumer)3 Consumer (java.util.function.Consumer)3 JobID (org.apache.flink.api.common.JobID)3 IntSerializer (org.apache.flink.api.common.typeutils.base.IntSerializer)3 CloseableRegistry (org.apache.flink.core.fs.CloseableRegistry)3 UnregisteredMetricsGroup (org.apache.flink.metrics.groups.UnregisteredMetricsGroup)3 MockEnvironment (org.apache.flink.runtime.operators.testutils.MockEnvironment)3 AbstractKeyedStateBackend (org.apache.flink.runtime.state.AbstractKeyedStateBackend)3 KeyGroupRange (org.apache.flink.runtime.state.KeyGroupRange)3 VoidNamespaceSerializer (org.apache.flink.runtime.state.VoidNamespaceSerializer)3 MemoryStateBackend (org.apache.flink.runtime.state.memory.MemoryStateBackend)3 TtlTimeProvider (org.apache.flink.runtime.state.ttl.TtlTimeProvider)3 SimpleTimerService (org.apache.flink.streaming.api.SimpleTimerService)3