Search in sources :

Example 6 with Evictor

use of org.apache.flink.streaming.api.windowing.evictors.Evictor in project flink by apache.

the class DataSetSavepointWindowReaderITCase method testProcessEvictorWindowStateReader.

@Test
public void testProcessEvictorWindowStateReader() throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    env.setStateBackend(getStateBackend());
    env.setParallelism(4);
    env.addSource(createSource(numbers)).rebalance().assignTimestampsAndWatermarks(WatermarkStrategy.<Integer>noWatermarks().withTimestampAssigner((event, timestamp) -> 0)).keyBy(id -> id).window(TumblingEventTimeWindows.of(Time.milliseconds(10))).evictor(new NoOpEvictor<>()).process(new NoOpProcessWindowFunction()).uid(uid).addSink(new DiscardingSink<>());
    String savepointPath = takeSavepoint(env);
    ExecutionEnvironment batchEnv = ExecutionEnvironment.getExecutionEnvironment();
    ExistingSavepoint savepoint = Savepoint.load(batchEnv, savepointPath, getStateBackend());
    List<Integer> results = savepoint.window(TumblingEventTimeWindows.of(Time.milliseconds(10))).evictor().process(uid, new BasicReaderFunction(), Types.INT, Types.INT, Types.INT).collect();
    Assert.assertThat("Unexpected results from keyed state", results, Matchers.containsInAnyOrder(numbers));
}
Also used : TimestampedValue(org.apache.flink.streaming.runtime.operators.windowing.TimestampedValue) CountTrigger(org.apache.flink.streaming.api.windowing.triggers.CountTrigger) ReduceSum(org.apache.flink.state.api.utils.ReduceSum) ReducingStateDescriptor(org.apache.flink.api.common.state.ReducingStateDescriptor) PurgingTrigger(org.apache.flink.streaming.api.windowing.triggers.PurgingTrigger) SavepointTestBase(org.apache.flink.state.api.utils.SavepointTestBase) Window(org.apache.flink.streaming.api.windowing.windows.Window) WindowReaderFunction(org.apache.flink.state.api.functions.WindowReaderFunction) StateBackend(org.apache.flink.runtime.state.StateBackend) ReducingState(org.apache.flink.api.common.state.ReducingState) Collector(org.apache.flink.util.Collector) TimeWindow(org.apache.flink.streaming.api.windowing.windows.TimeWindow) LongSerializer(org.apache.flink.api.common.typeutils.base.LongSerializer) ReduceFunction(org.apache.flink.api.common.functions.ReduceFunction) GlobalWindow(org.apache.flink.streaming.api.windowing.windows.GlobalWindow) Types(org.apache.flink.api.common.typeinfo.Types) Time(org.apache.flink.streaming.api.windowing.time.Time) DiscardingSink(org.apache.flink.streaming.api.functions.sink.DiscardingSink) Evictor(org.apache.flink.streaming.api.windowing.evictors.Evictor) Matchers(org.hamcrest.Matchers) WatermarkStrategy(org.apache.flink.api.common.eventtime.WatermarkStrategy) Test(org.junit.Test) AggregateSum(org.apache.flink.state.api.utils.AggregateSum) ProcessWindowFunction(org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction) WindowFunction(org.apache.flink.streaming.api.functions.windowing.WindowFunction) List(java.util.List) ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) TumblingEventTimeWindows(org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows) Assert(org.junit.Assert) GlobalWindows(org.apache.flink.streaming.api.windowing.assigners.GlobalWindows) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) Test(org.junit.Test)

Example 7 with Evictor

use of org.apache.flink.streaming.api.windowing.evictors.Evictor in project flink by apache.

the class DataSetSavepointWindowReaderITCase method testApplyEvictorWindowStateReader.

@Test
public void testApplyEvictorWindowStateReader() throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    env.setStateBackend(getStateBackend());
    env.setParallelism(4);
    env.addSource(createSource(numbers)).rebalance().assignTimestampsAndWatermarks(WatermarkStrategy.<Integer>noWatermarks().withTimestampAssigner((event, timestamp) -> 0)).keyBy(id -> id).window(TumblingEventTimeWindows.of(Time.milliseconds(10))).evictor(new NoOpEvictor<>()).apply(new NoOpWindowFunction()).uid(uid).addSink(new DiscardingSink<>());
    String savepointPath = takeSavepoint(env);
    ExecutionEnvironment batchEnv = ExecutionEnvironment.getExecutionEnvironment();
    ExistingSavepoint savepoint = Savepoint.load(batchEnv, savepointPath, getStateBackend());
    List<Integer> results = savepoint.window(TumblingEventTimeWindows.of(Time.milliseconds(1))).evictor().process(uid, new BasicReaderFunction(), Types.INT, Types.INT, Types.INT).collect();
    Assert.assertThat("Unexpected results from keyed state", results, Matchers.containsInAnyOrder(numbers));
}
Also used : TimestampedValue(org.apache.flink.streaming.runtime.operators.windowing.TimestampedValue) CountTrigger(org.apache.flink.streaming.api.windowing.triggers.CountTrigger) ReduceSum(org.apache.flink.state.api.utils.ReduceSum) ReducingStateDescriptor(org.apache.flink.api.common.state.ReducingStateDescriptor) PurgingTrigger(org.apache.flink.streaming.api.windowing.triggers.PurgingTrigger) SavepointTestBase(org.apache.flink.state.api.utils.SavepointTestBase) Window(org.apache.flink.streaming.api.windowing.windows.Window) WindowReaderFunction(org.apache.flink.state.api.functions.WindowReaderFunction) StateBackend(org.apache.flink.runtime.state.StateBackend) ReducingState(org.apache.flink.api.common.state.ReducingState) Collector(org.apache.flink.util.Collector) TimeWindow(org.apache.flink.streaming.api.windowing.windows.TimeWindow) LongSerializer(org.apache.flink.api.common.typeutils.base.LongSerializer) ReduceFunction(org.apache.flink.api.common.functions.ReduceFunction) GlobalWindow(org.apache.flink.streaming.api.windowing.windows.GlobalWindow) Types(org.apache.flink.api.common.typeinfo.Types) Time(org.apache.flink.streaming.api.windowing.time.Time) DiscardingSink(org.apache.flink.streaming.api.functions.sink.DiscardingSink) Evictor(org.apache.flink.streaming.api.windowing.evictors.Evictor) Matchers(org.hamcrest.Matchers) WatermarkStrategy(org.apache.flink.api.common.eventtime.WatermarkStrategy) Test(org.junit.Test) AggregateSum(org.apache.flink.state.api.utils.AggregateSum) ProcessWindowFunction(org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction) WindowFunction(org.apache.flink.streaming.api.functions.windowing.WindowFunction) List(java.util.List) ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) TumblingEventTimeWindows(org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows) Assert(org.junit.Assert) GlobalWindows(org.apache.flink.streaming.api.windowing.assigners.GlobalWindows) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) Test(org.junit.Test)

Example 8 with Evictor

use of org.apache.flink.streaming.api.windowing.evictors.Evictor in project flink by apache.

the class SavepointWindowReaderITCase method testReduceEvictorWindowStateReader.

@Test
public void testReduceEvictorWindowStateReader() throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    env.setStateBackend(getStateBackend());
    env.setParallelism(4);
    env.addSource(createSource(numbers)).rebalance().assignTimestampsAndWatermarks(WatermarkStrategy.<Integer>noWatermarks().withTimestampAssigner((event, timestamp) -> 0)).keyBy(id -> id).window(TumblingEventTimeWindows.of(Time.milliseconds(10))).evictor(new NoOpEvictor<>()).reduce(new ReduceSum()).uid(uid).addSink(new DiscardingSink<>());
    String savepointPath = takeSavepoint(env);
    SavepointReader savepoint = SavepointReader.read(env, savepointPath, getStateBackend());
    List<Integer> results = JobResultRetriever.collect(savepoint.window(TumblingEventTimeWindows.of(Time.milliseconds(10))).evictor().reduce(uid, new ReduceSum(), Types.INT, Types.INT));
    Assert.assertThat("Unexpected results from keyed state", results, Matchers.containsInAnyOrder(numbers));
}
Also used : TimestampedValue(org.apache.flink.streaming.runtime.operators.windowing.TimestampedValue) CountTrigger(org.apache.flink.streaming.api.windowing.triggers.CountTrigger) ReduceSum(org.apache.flink.state.api.utils.ReduceSum) ReducingStateDescriptor(org.apache.flink.api.common.state.ReducingStateDescriptor) PurgingTrigger(org.apache.flink.streaming.api.windowing.triggers.PurgingTrigger) JobResultRetriever(org.apache.flink.state.api.utils.JobResultRetriever) SavepointTestBase(org.apache.flink.state.api.utils.SavepointTestBase) Window(org.apache.flink.streaming.api.windowing.windows.Window) WindowReaderFunction(org.apache.flink.state.api.functions.WindowReaderFunction) StateBackend(org.apache.flink.runtime.state.StateBackend) ReducingState(org.apache.flink.api.common.state.ReducingState) Collector(org.apache.flink.util.Collector) TimeWindow(org.apache.flink.streaming.api.windowing.windows.TimeWindow) LongSerializer(org.apache.flink.api.common.typeutils.base.LongSerializer) ReduceFunction(org.apache.flink.api.common.functions.ReduceFunction) GlobalWindow(org.apache.flink.streaming.api.windowing.windows.GlobalWindow) Types(org.apache.flink.api.common.typeinfo.Types) Time(org.apache.flink.streaming.api.windowing.time.Time) DiscardingSink(org.apache.flink.streaming.api.functions.sink.DiscardingSink) Evictor(org.apache.flink.streaming.api.windowing.evictors.Evictor) Matchers(org.hamcrest.Matchers) WatermarkStrategy(org.apache.flink.api.common.eventtime.WatermarkStrategy) Test(org.junit.Test) AggregateSum(org.apache.flink.state.api.utils.AggregateSum) ProcessWindowFunction(org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction) WindowFunction(org.apache.flink.streaming.api.functions.windowing.WindowFunction) List(java.util.List) TumblingEventTimeWindows(org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows) Assert(org.junit.Assert) GlobalWindows(org.apache.flink.streaming.api.windowing.assigners.GlobalWindows) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) ReduceSum(org.apache.flink.state.api.utils.ReduceSum) Test(org.junit.Test)

Aggregations

List (java.util.List)8 WatermarkStrategy (org.apache.flink.api.common.eventtime.WatermarkStrategy)8 ReduceFunction (org.apache.flink.api.common.functions.ReduceFunction)8 ReducingState (org.apache.flink.api.common.state.ReducingState)8 ReducingStateDescriptor (org.apache.flink.api.common.state.ReducingStateDescriptor)8 Types (org.apache.flink.api.common.typeinfo.Types)8 LongSerializer (org.apache.flink.api.common.typeutils.base.LongSerializer)8 StateBackend (org.apache.flink.runtime.state.StateBackend)8 WindowReaderFunction (org.apache.flink.state.api.functions.WindowReaderFunction)8 AggregateSum (org.apache.flink.state.api.utils.AggregateSum)8 ReduceSum (org.apache.flink.state.api.utils.ReduceSum)8 SavepointTestBase (org.apache.flink.state.api.utils.SavepointTestBase)8 StreamExecutionEnvironment (org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)8 DiscardingSink (org.apache.flink.streaming.api.functions.sink.DiscardingSink)8 ProcessWindowFunction (org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction)8 WindowFunction (org.apache.flink.streaming.api.functions.windowing.WindowFunction)8 GlobalWindows (org.apache.flink.streaming.api.windowing.assigners.GlobalWindows)8 TumblingEventTimeWindows (org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows)8 Evictor (org.apache.flink.streaming.api.windowing.evictors.Evictor)8 Time (org.apache.flink.streaming.api.windowing.time.Time)8