Search in sources :

Example 1 with InternalAggregateProcessWindowFunction

use of org.apache.flink.streaming.runtime.operators.windowing.functions.InternalAggregateProcessWindowFunction in project flink by apache.

the class WindowedStream method aggregate.

/**
	 * Applies the given window function to each window. The window function is called for each
	 * evaluation of the window for each key individually. The output of the window function is
	 * interpreted as a regular non-windowed stream.
	 *
	 * <p>Arriving data is incrementally aggregated using the given aggregate function. This means
	 * that the window function typically has only a single value to process when called.
	 *
	 * @param aggregateFunction The aggregation function that is used for incremental aggregation.
	 * @param windowFunction The window function.
	 * @param accumulatorType Type information for the internal accumulator type of the aggregation function
	 * @param resultType Type information for the result type of the window function
	 *
	 * @return The data stream that is the result of applying the window function to the window.
	 *
	 * @param <ACC> The type of the AggregateFunction's accumulator
	 * @param <V> The type of AggregateFunction's result, and the WindowFunction's input
	 * @param <R> The type of the elements in the resulting stream, equal to the
	 *            WindowFunction's result type
	 */
@PublicEvolving
public <ACC, V, R> SingleOutputStreamOperator<R> aggregate(AggregateFunction<T, ACC, V> aggregateFunction, ProcessWindowFunction<V, R, K, W> windowFunction, TypeInformation<ACC> accumulatorType, TypeInformation<V> aggregateResultType, TypeInformation<R> resultType) {
    checkNotNull(aggregateFunction, "aggregateFunction");
    checkNotNull(windowFunction, "windowFunction");
    checkNotNull(accumulatorType, "accumulatorType");
    checkNotNull(aggregateResultType, "aggregateResultType");
    checkNotNull(resultType, "resultType");
    if (aggregateFunction instanceof RichFunction) {
        throw new UnsupportedOperationException("This aggregate function cannot be a RichFunction.");
    }
    //clean the closures
    windowFunction = input.getExecutionEnvironment().clean(windowFunction);
    aggregateFunction = input.getExecutionEnvironment().clean(aggregateFunction);
    String callLocation = Utils.getCallLocationName();
    String udfName = "WindowedStream." + callLocation;
    String opName;
    KeySelector<T, K> keySel = input.getKeySelector();
    OneInputStreamOperator<T, R> operator;
    if (evictor != null) {
        @SuppressWarnings({ "unchecked", "rawtypes" }) TypeSerializer<StreamRecord<T>> streamRecordSerializer = (TypeSerializer<StreamRecord<T>>) new StreamElementSerializer(input.getType().createSerializer(getExecutionEnvironment().getConfig()));
        ListStateDescriptor<StreamRecord<T>> stateDesc = new ListStateDescriptor<>("window-contents", streamRecordSerializer);
        opName = "TriggerWindow(" + windowAssigner + ", " + stateDesc + ", " + trigger + ", " + evictor + ", " + udfName + ")";
        operator = new EvictingWindowOperator<>(windowAssigner, windowAssigner.getWindowSerializer(getExecutionEnvironment().getConfig()), keySel, input.getKeyType().createSerializer(getExecutionEnvironment().getConfig()), stateDesc, new InternalAggregateProcessWindowFunction<>(aggregateFunction, windowFunction), trigger, evictor, allowedLateness, lateDataOutputTag);
    } else {
        AggregatingStateDescriptor<T, ACC, V> stateDesc = new AggregatingStateDescriptor<>("window-contents", aggregateFunction, accumulatorType.createSerializer(getExecutionEnvironment().getConfig()));
        opName = "TriggerWindow(" + windowAssigner + ", " + stateDesc + ", " + trigger + ", " + udfName + ")";
        operator = new WindowOperator<>(windowAssigner, windowAssigner.getWindowSerializer(getExecutionEnvironment().getConfig()), keySel, input.getKeyType().createSerializer(getExecutionEnvironment().getConfig()), stateDesc, new InternalSingleValueProcessWindowFunction<>(windowFunction), trigger, allowedLateness, lateDataOutputTag);
    }
    return input.transform(opName, resultType, operator);
}
Also used : StreamRecord(org.apache.flink.streaming.runtime.streamrecord.StreamRecord) RichFunction(org.apache.flink.api.common.functions.RichFunction) AggregatingStateDescriptor(org.apache.flink.api.common.state.AggregatingStateDescriptor) ListStateDescriptor(org.apache.flink.api.common.state.ListStateDescriptor) InternalAggregateProcessWindowFunction(org.apache.flink.streaming.runtime.operators.windowing.functions.InternalAggregateProcessWindowFunction) TypeSerializer(org.apache.flink.api.common.typeutils.TypeSerializer) InternalSingleValueProcessWindowFunction(org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueProcessWindowFunction) StreamElementSerializer(org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer) PublicEvolving(org.apache.flink.annotation.PublicEvolving)

Example 2 with InternalAggregateProcessWindowFunction

use of org.apache.flink.streaming.runtime.operators.windowing.functions.InternalAggregateProcessWindowFunction in project flink by apache.

the class InternalWindowFunctionTest method testInternalAggregateProcessWindowFunction.

@SuppressWarnings("unchecked")
@Test
public void testInternalAggregateProcessWindowFunction() throws Exception {
    AggregateProcessWindowFunctionMock mock = mock(AggregateProcessWindowFunctionMock.class);
    InternalAggregateProcessWindowFunction<Long, Set<Long>, Map<Long, Long>, String, Long, TimeWindow> windowFunction = new InternalAggregateProcessWindowFunction<>(new AggregateFunction<Long, Set<Long>, Map<Long, Long>>() {

        private static final long serialVersionUID = 1L;

        @Override
        public Set<Long> createAccumulator() {
            return new HashSet<>();
        }

        @Override
        public Set<Long> add(Long value, Set<Long> accumulator) {
            accumulator.add(value);
            return accumulator;
        }

        @Override
        public Map<Long, Long> getResult(Set<Long> accumulator) {
            Map<Long, Long> result = new HashMap<>();
            for (Long in : accumulator) {
                result.put(in, in);
            }
            return result;
        }

        @Override
        public Set<Long> merge(Set<Long> a, Set<Long> b) {
            a.addAll(b);
            return a;
        }
    }, mock);
    // check setOutputType
    TypeInformation<String> stringType = BasicTypeInfo.STRING_TYPE_INFO;
    ExecutionConfig execConf = new ExecutionConfig();
    execConf.setParallelism(42);
    StreamingFunctionUtils.setOutputType(windowFunction, stringType, execConf);
    verify(mock).setOutputType(stringType, execConf);
    // check open
    Configuration config = new Configuration();
    windowFunction.open(config);
    verify(mock).open(config);
    // check setRuntimeContext
    RuntimeContext rCtx = mock(RuntimeContext.class);
    windowFunction.setRuntimeContext(rCtx);
    verify(mock).setRuntimeContext(rCtx);
    // check apply
    TimeWindow w = mock(TimeWindow.class);
    Collector<String> c = (Collector<String>) mock(Collector.class);
    List<Long> args = new LinkedList<>();
    args.add(23L);
    args.add(24L);
    InternalWindowFunction.InternalWindowContext ctx = mock(InternalWindowFunction.InternalWindowContext.class);
    doAnswer(new Answer() {

        @Override
        public Object answer(InvocationOnMock invocationOnMock) throws Throwable {
            ProcessWindowFunctionMock.Context c = (ProcessWindowFunction.Context) invocationOnMock.getArguments()[1];
            c.currentProcessingTime();
            c.currentWatermark();
            c.windowState();
            c.globalState();
            return null;
        }
    }).when(mock).process(eq(42L), (AggregateProcessWindowFunctionMock.Context) anyObject(), (Iterable) argThat(containsInAnyOrder(allOf(hasEntry(is(23L), is(23L)), hasEntry(is(24L), is(24L))))), eq(c));
    windowFunction.process(42L, w, ctx, args, c);
    verify(ctx).currentProcessingTime();
    verify(ctx).currentWatermark();
    verify(ctx).windowState();
    verify(ctx).globalState();
    // check close
    windowFunction.close();
    verify(mock).close();
}
Also used : HashSet(java.util.HashSet) Set(java.util.Set) Configuration(org.apache.flink.configuration.Configuration) InternalAggregateProcessWindowFunction(org.apache.flink.streaming.runtime.operators.windowing.functions.InternalAggregateProcessWindowFunction) InternalIterableProcessWindowFunction(org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessWindowFunction) ProcessWindowFunction(org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction) InternalSingleValueProcessWindowFunction(org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueProcessWindowFunction) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) InternalAggregateProcessWindowFunction(org.apache.flink.streaming.runtime.operators.windowing.functions.InternalAggregateProcessWindowFunction) Collector(org.apache.flink.util.Collector) RuntimeContext(org.apache.flink.api.common.functions.RuntimeContext) InternalWindowFunction(org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction) TimeWindow(org.apache.flink.streaming.api.windowing.windows.TimeWindow) LinkedList(java.util.LinkedList) Answer(org.mockito.stubbing.Answer) Mockito.doAnswer(org.mockito.Mockito.doAnswer) InvocationOnMock(org.mockito.invocation.InvocationOnMock) Mockito.anyObject(org.mockito.Mockito.anyObject) RuntimeContext(org.apache.flink.api.common.functions.RuntimeContext) HashMap(java.util.HashMap) Map(java.util.Map) Test(org.junit.Test)

Aggregations

InternalAggregateProcessWindowFunction (org.apache.flink.streaming.runtime.operators.windowing.functions.InternalAggregateProcessWindowFunction)2 InternalSingleValueProcessWindowFunction (org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueProcessWindowFunction)2 HashMap (java.util.HashMap)1 HashSet (java.util.HashSet)1 LinkedList (java.util.LinkedList)1 Map (java.util.Map)1 Set (java.util.Set)1 PublicEvolving (org.apache.flink.annotation.PublicEvolving)1 ExecutionConfig (org.apache.flink.api.common.ExecutionConfig)1 RichFunction (org.apache.flink.api.common.functions.RichFunction)1 RuntimeContext (org.apache.flink.api.common.functions.RuntimeContext)1 AggregatingStateDescriptor (org.apache.flink.api.common.state.AggregatingStateDescriptor)1 ListStateDescriptor (org.apache.flink.api.common.state.ListStateDescriptor)1 TypeSerializer (org.apache.flink.api.common.typeutils.TypeSerializer)1 Configuration (org.apache.flink.configuration.Configuration)1 ProcessWindowFunction (org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction)1 TimeWindow (org.apache.flink.streaming.api.windowing.windows.TimeWindow)1 InternalIterableProcessWindowFunction (org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessWindowFunction)1 InternalWindowFunction (org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction)1 StreamElementSerializer (org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer)1