Search in sources :

Example 96 with Configuration

use of org.apache.flink.configuration.Configuration in project flink by apache.

the class InternalWindowFunctionTest method testInternalAggregateProcessWindowFunction.

@SuppressWarnings("unchecked")
@Test
public void testInternalAggregateProcessWindowFunction() throws Exception {
    AggregateProcessWindowFunctionMock mock = mock(AggregateProcessWindowFunctionMock.class);
    InternalAggregateProcessWindowFunction<Long, Set<Long>, Map<Long, Long>, String, Long, TimeWindow> windowFunction = new InternalAggregateProcessWindowFunction<>(new AggregateFunction<Long, Set<Long>, Map<Long, Long>>() {

        private static final long serialVersionUID = 1L;

        @Override
        public Set<Long> createAccumulator() {
            return new HashSet<>();
        }

        @Override
        public void add(Long value, Set<Long> accumulator) {
            accumulator.add(value);
        }

        @Override
        public Map<Long, Long> getResult(Set<Long> accumulator) {
            Map<Long, Long> result = new HashMap<>();
            for (Long in : accumulator) {
                result.put(in, in);
            }
            return result;
        }

        @Override
        public Set<Long> merge(Set<Long> a, Set<Long> b) {
            a.addAll(b);
            return a;
        }
    }, mock);
    // check setOutputType
    TypeInformation<String> stringType = BasicTypeInfo.STRING_TYPE_INFO;
    ExecutionConfig execConf = new ExecutionConfig();
    execConf.setParallelism(42);
    StreamingFunctionUtils.setOutputType(windowFunction, stringType, execConf);
    verify(mock).setOutputType(stringType, execConf);
    // check open
    Configuration config = new Configuration();
    windowFunction.open(config);
    verify(mock).open(config);
    // check setRuntimeContext
    RuntimeContext rCtx = mock(RuntimeContext.class);
    windowFunction.setRuntimeContext(rCtx);
    verify(mock).setRuntimeContext(rCtx);
    // check apply
    TimeWindow w = mock(TimeWindow.class);
    Collector<String> c = (Collector<String>) mock(Collector.class);
    List<Long> args = new LinkedList<>();
    args.add(23L);
    args.add(24L);
    windowFunction.apply(42L, w, args, c);
    verify(mock).process(eq(42L), (AggregateProcessWindowFunctionMock.Context) anyObject(), (Iterable) argThat(containsInAnyOrder(allOf(hasEntry(is(23L), is(23L)), hasEntry(is(24L), is(24L))))), eq(c));
    // check close
    windowFunction.close();
    verify(mock).close();
}
Also used : HashSet(java.util.HashSet) Set(java.util.Set) Configuration(org.apache.flink.configuration.Configuration) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) InternalAggregateProcessWindowFunction(org.apache.flink.streaming.runtime.operators.windowing.functions.InternalAggregateProcessWindowFunction) TimeWindow(org.apache.flink.streaming.api.windowing.windows.TimeWindow) LinkedList(java.util.LinkedList) Collector(org.apache.flink.util.Collector) RuntimeContext(org.apache.flink.api.common.functions.RuntimeContext) HashMap(java.util.HashMap) Map(java.util.Map) Test(org.junit.Test)

Example 97 with Configuration

use of org.apache.flink.configuration.Configuration in project flink by apache.

the class InternalWindowFunctionTest method testInternalIterableProcessWindowFunction.

@SuppressWarnings("unchecked")
@Test
public void testInternalIterableProcessWindowFunction() throws Exception {
    ProcessWindowFunctionMock mock = mock(ProcessWindowFunctionMock.class);
    InternalIterableProcessWindowFunction<Long, String, Long, TimeWindow> windowFunction = new InternalIterableProcessWindowFunction<>(mock);
    // check setOutputType
    TypeInformation<String> stringType = BasicTypeInfo.STRING_TYPE_INFO;
    ExecutionConfig execConf = new ExecutionConfig();
    execConf.setParallelism(42);
    StreamingFunctionUtils.setOutputType(windowFunction, stringType, execConf);
    verify(mock).setOutputType(stringType, execConf);
    // check open
    Configuration config = new Configuration();
    windowFunction.open(config);
    verify(mock).open(config);
    // check setRuntimeContext
    RuntimeContext rCtx = mock(RuntimeContext.class);
    windowFunction.setRuntimeContext(rCtx);
    verify(mock).setRuntimeContext(rCtx);
    // check apply
    TimeWindow w = mock(TimeWindow.class);
    Iterable<Long> i = (Iterable<Long>) mock(Iterable.class);
    Collector<String> c = (Collector<String>) mock(Collector.class);
    windowFunction.apply(42L, w, i, c);
    verify(mock).process(eq(42L), (ProcessWindowFunctionMock.Context) anyObject(), eq(i), eq(c));
    // check close
    windowFunction.close();
    verify(mock).close();
}
Also used : Configuration(org.apache.flink.configuration.Configuration) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) TimeWindow(org.apache.flink.streaming.api.windowing.windows.TimeWindow) Collector(org.apache.flink.util.Collector) InternalIterableProcessWindowFunction(org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessWindowFunction) RuntimeContext(org.apache.flink.api.common.functions.RuntimeContext) Test(org.junit.Test)

Example 98 with Configuration

use of org.apache.flink.configuration.Configuration in project flink by apache.

the class InternalWindowFunctionTest method testInternalIterableProcessAllWindowFunction.

@SuppressWarnings("unchecked")
@Test
public void testInternalIterableProcessAllWindowFunction() throws Exception {
    ProcessAllWindowFunctionMock mock = mock(ProcessAllWindowFunctionMock.class);
    InternalIterableProcessAllWindowFunction<Long, String, TimeWindow> windowFunction = new InternalIterableProcessAllWindowFunction<>(mock);
    // check setOutputType
    TypeInformation<String> stringType = BasicTypeInfo.STRING_TYPE_INFO;
    ExecutionConfig execConf = new ExecutionConfig();
    execConf.setParallelism(42);
    StreamingFunctionUtils.setOutputType(windowFunction, stringType, execConf);
    verify(mock).setOutputType(stringType, execConf);
    // check open
    Configuration config = new Configuration();
    windowFunction.open(config);
    verify(mock).open(config);
    // check setRuntimeContext
    RuntimeContext rCtx = mock(RuntimeContext.class);
    windowFunction.setRuntimeContext(rCtx);
    verify(mock).setRuntimeContext(rCtx);
    // check apply
    TimeWindow w = mock(TimeWindow.class);
    Iterable<Long> i = (Iterable<Long>) mock(Iterable.class);
    Collector<String> c = (Collector<String>) mock(Collector.class);
    windowFunction.apply(((byte) 0), w, i, c);
    verify(mock).process((ProcessAllWindowFunctionMock.Context) anyObject(), eq(i), eq(c));
    // check close
    windowFunction.close();
    verify(mock).close();
}
Also used : Configuration(org.apache.flink.configuration.Configuration) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) TimeWindow(org.apache.flink.streaming.api.windowing.windows.TimeWindow) Collector(org.apache.flink.util.Collector) InternalIterableProcessAllWindowFunction(org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessAllWindowFunction) RuntimeContext(org.apache.flink.api.common.functions.RuntimeContext) Test(org.junit.Test)

Example 99 with Configuration

use of org.apache.flink.configuration.Configuration in project flink by apache.

the class InternalWindowFunctionTest method testInternalSingleValueAllWindowFunction.

@SuppressWarnings("unchecked")
@Test
public void testInternalSingleValueAllWindowFunction() throws Exception {
    AllWindowFunctionMock mock = mock(AllWindowFunctionMock.class);
    InternalSingleValueAllWindowFunction<Long, String, TimeWindow> windowFunction = new InternalSingleValueAllWindowFunction<>(mock);
    // check setOutputType
    TypeInformation<String> stringType = BasicTypeInfo.STRING_TYPE_INFO;
    ExecutionConfig execConf = new ExecutionConfig();
    execConf.setParallelism(42);
    StreamingFunctionUtils.setOutputType(windowFunction, stringType, execConf);
    verify(mock).setOutputType(stringType, execConf);
    // check open
    Configuration config = new Configuration();
    windowFunction.open(config);
    verify(mock).open(config);
    // check setRuntimeContext
    RuntimeContext rCtx = mock(RuntimeContext.class);
    windowFunction.setRuntimeContext(rCtx);
    verify(mock).setRuntimeContext(rCtx);
    // check apply
    TimeWindow w = mock(TimeWindow.class);
    Collector<String> c = (Collector<String>) mock(Collector.class);
    windowFunction.apply(((byte) 0), w, 23L, c);
    verify(mock).apply(eq(w), (Iterable<Long>) argThat(IsIterableContainingInOrder.contains(23L)), eq(c));
    // check close
    windowFunction.close();
    verify(mock).close();
}
Also used : InternalSingleValueAllWindowFunction(org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueAllWindowFunction) Configuration(org.apache.flink.configuration.Configuration) Collector(org.apache.flink.util.Collector) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) RuntimeContext(org.apache.flink.api.common.functions.RuntimeContext) TimeWindow(org.apache.flink.streaming.api.windowing.windows.TimeWindow) Test(org.junit.Test)

Example 100 with Configuration

use of org.apache.flink.configuration.Configuration in project flink by apache.

the class InternalWindowFunctionTest method testInternalSingleValueProcessAllWindowFunction.

@SuppressWarnings("unchecked")
@Test
public void testInternalSingleValueProcessAllWindowFunction() throws Exception {
    ProcessAllWindowFunctionMock mock = mock(ProcessAllWindowFunctionMock.class);
    InternalSingleValueProcessAllWindowFunction<Long, String, TimeWindow> windowFunction = new InternalSingleValueProcessAllWindowFunction<>(mock);
    // check setOutputType
    TypeInformation<String> stringType = BasicTypeInfo.STRING_TYPE_INFO;
    ExecutionConfig execConf = new ExecutionConfig();
    execConf.setParallelism(42);
    StreamingFunctionUtils.setOutputType(windowFunction, stringType, execConf);
    verify(mock).setOutputType(stringType, execConf);
    // check open
    Configuration config = new Configuration();
    windowFunction.open(config);
    verify(mock).open(config);
    // check setRuntimeContext
    RuntimeContext rCtx = mock(RuntimeContext.class);
    windowFunction.setRuntimeContext(rCtx);
    verify(mock).setRuntimeContext(rCtx);
    // check apply
    TimeWindow w = mock(TimeWindow.class);
    Collector<String> c = (Collector<String>) mock(Collector.class);
    windowFunction.apply(((byte) 0), w, 23L, c);
    verify(mock).process((ProcessAllWindowFunctionMock.Context) anyObject(), (Iterable<Long>) argThat(IsIterableContainingInOrder.contains(23L)), eq(c));
    // check close
    windowFunction.close();
    verify(mock).close();
}
Also used : Configuration(org.apache.flink.configuration.Configuration) Collector(org.apache.flink.util.Collector) InternalSingleValueProcessAllWindowFunction(org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueProcessAllWindowFunction) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) RuntimeContext(org.apache.flink.api.common.functions.RuntimeContext) TimeWindow(org.apache.flink.streaming.api.windowing.windows.TimeWindow) Test(org.junit.Test)

Aggregations

Configuration (org.apache.flink.configuration.Configuration)630 Test (org.junit.Test)452 IOException (java.io.IOException)137 FileInputSplit (org.apache.flink.core.fs.FileInputSplit)93 File (java.io.File)92 JobID (org.apache.flink.api.common.JobID)74 ExecutionConfig (org.apache.flink.api.common.ExecutionConfig)68 JobVertex (org.apache.flink.runtime.jobgraph.JobVertex)49 ActorGateway (org.apache.flink.runtime.instance.ActorGateway)46 JobGraph (org.apache.flink.runtime.jobgraph.JobGraph)45 Path (org.apache.flink.core.fs.Path)44 ActorRef (akka.actor.ActorRef)43 ArrayList (java.util.ArrayList)43 Tuple2 (org.apache.flink.api.java.tuple.Tuple2)39 FiniteDuration (scala.concurrent.duration.FiniteDuration)38 LocalFlinkMiniCluster (org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster)36 BeforeClass (org.junit.BeforeClass)35 AkkaActorGateway (org.apache.flink.runtime.instance.AkkaActorGateway)33 MetricRegistry (org.apache.flink.runtime.metrics.MetricRegistry)33 JobVertexID (org.apache.flink.runtime.jobgraph.JobVertexID)32