Search in sources :

Example 11 with FieldList

use of org.apache.flink.api.common.operators.util.FieldList in project flink by apache.

the class GroupReduceCompilationTest method testGroupedReduceWithSelectorFunctionKeyNoncombinable.

@Test
public void testGroupedReduceWithSelectorFunctionKeyNoncombinable() {
    try {
        ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(8);
        DataSet<Tuple2<String, Double>> data = env.readCsvFile("file:///will/never/be/read").types(String.class, Double.class).name("source").setParallelism(6);
        data.groupBy(new KeySelector<Tuple2<String, Double>, String>() {

            public String getKey(Tuple2<String, Double> value) {
                return value.f0;
            }
        }).reduceGroup(new RichGroupReduceFunction<Tuple2<String, Double>, Tuple2<String, Double>>() {

            public void reduce(Iterable<Tuple2<String, Double>> values, Collector<Tuple2<String, Double>> out) {
            }
        }).name("reducer").output(new DiscardingOutputFormat<Tuple2<String, Double>>()).name("sink");
        Plan p = env.createProgramPlan();
        OptimizedPlan op = compileNoStats(p);
        OptimizerPlanNodeResolver resolver = getOptimizerPlanNodeResolver(op);
        // get the original nodes
        SourcePlanNode sourceNode = resolver.getNode("source");
        SingleInputPlanNode reduceNode = resolver.getNode("reducer");
        SinkPlanNode sinkNode = resolver.getNode("sink");
        // get the key extractors and projectors
        SingleInputPlanNode keyExtractor = (SingleInputPlanNode) reduceNode.getInput().getSource();
        SingleInputPlanNode keyProjector = (SingleInputPlanNode) sinkNode.getInput().getSource();
        // check wiring
        assertEquals(sourceNode, keyExtractor.getInput().getSource());
        assertEquals(keyProjector, sinkNode.getInput().getSource());
        // check that both reduce and combiner have the same strategy
        assertEquals(DriverStrategy.SORTED_GROUP_REDUCE, reduceNode.getDriverStrategy());
        // check the keys
        assertEquals(new FieldList(0), reduceNode.getKeys(0));
        assertEquals(new FieldList(0), reduceNode.getInput().getLocalStrategyKeys());
        // check parallelism
        assertEquals(6, sourceNode.getParallelism());
        assertEquals(6, keyExtractor.getParallelism());
        assertEquals(8, reduceNode.getParallelism());
        assertEquals(8, keyProjector.getParallelism());
        assertEquals(8, sinkNode.getParallelism());
    } catch (Exception e) {
        System.err.println(e.getMessage());
        e.printStackTrace();
        fail(e.getClass().getSimpleName() + " in test: " + e.getMessage());
    }
}
Also used : RichGroupReduceFunction(org.apache.flink.api.common.functions.RichGroupReduceFunction) ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) Plan(org.apache.flink.api.common.Plan) OptimizedPlan(org.apache.flink.optimizer.plan.OptimizedPlan) DiscardingOutputFormat(org.apache.flink.api.java.io.DiscardingOutputFormat) OptimizedPlan(org.apache.flink.optimizer.plan.OptimizedPlan) FieldList(org.apache.flink.api.common.operators.util.FieldList) SingleInputPlanNode(org.apache.flink.optimizer.plan.SingleInputPlanNode) Tuple2(org.apache.flink.api.java.tuple.Tuple2) Collector(org.apache.flink.util.Collector) SourcePlanNode(org.apache.flink.optimizer.plan.SourcePlanNode) SinkPlanNode(org.apache.flink.optimizer.plan.SinkPlanNode) Test(org.junit.Test)

Example 12 with FieldList

use of org.apache.flink.api.common.operators.util.FieldList in project flink by apache.

the class GlobalPropertiesFilteringTest method testAnyPartitioningErased.

@Test
public void testAnyPartitioningErased() {
    SingleInputSemanticProperties sprops = new SingleInputSemanticProperties();
    SemanticPropUtil.getSemanticPropsSingleFromString(sprops, new String[] { "0;1" }, null, null, tupleInfo, tupleInfo);
    GlobalProperties gprops = new GlobalProperties();
    gprops.setAnyPartitioning(new FieldList(0, 1, 4));
    GlobalProperties result = gprops.filterBySemanticProperties(sprops, 0);
    assertEquals(PartitioningProperty.RANDOM_PARTITIONED, result.getPartitioning());
    assertNull(result.getPartitioningFields());
}
Also used : SingleInputSemanticProperties(org.apache.flink.api.common.operators.SingleInputSemanticProperties) FieldList(org.apache.flink.api.common.operators.util.FieldList) Test(org.junit.Test)

Example 13 with FieldList

use of org.apache.flink.api.common.operators.util.FieldList in project flink by apache.

the class GlobalPropertiesFilteringTest method testInvalidInputIndex.

@Test(expected = IndexOutOfBoundsException.class)
public void testInvalidInputIndex() {
    SingleInputSemanticProperties sprops = new SingleInputSemanticProperties();
    SemanticPropUtil.getSemanticPropsSingleFromString(sprops, new String[] { "0;1" }, null, null, tupleInfo, tupleInfo);
    GlobalProperties gprops = new GlobalProperties();
    gprops.setHashPartitioned(new FieldList(0, 1));
    gprops.filterBySemanticProperties(sprops, 1);
}
Also used : SingleInputSemanticProperties(org.apache.flink.api.common.operators.SingleInputSemanticProperties) FieldList(org.apache.flink.api.common.operators.util.FieldList) Test(org.junit.Test)

Example 14 with FieldList

use of org.apache.flink.api.common.operators.util.FieldList in project flink by apache.

the class GlobalPropertiesFilteringTest method testAnyPartitioningPreserved1.

@Test
public void testAnyPartitioningPreserved1() {
    SingleInputSemanticProperties sprops = new SingleInputSemanticProperties();
    SemanticPropUtil.getSemanticPropsSingleFromString(sprops, new String[] { "0;1;4" }, null, null, tupleInfo, tupleInfo);
    GlobalProperties gprops = new GlobalProperties();
    gprops.setAnyPartitioning(new FieldList(0, 1, 4));
    GlobalProperties result = gprops.filterBySemanticProperties(sprops, 0);
    assertEquals(PartitioningProperty.ANY_PARTITIONING, result.getPartitioning());
    FieldList pFields = result.getPartitioningFields();
    assertEquals(3, pFields.size());
    assertTrue(pFields.contains(0));
    assertTrue(pFields.contains(1));
    assertTrue(pFields.contains(4));
}
Also used : SingleInputSemanticProperties(org.apache.flink.api.common.operators.SingleInputSemanticProperties) FieldList(org.apache.flink.api.common.operators.util.FieldList) Test(org.junit.Test)

Example 15 with FieldList

use of org.apache.flink.api.common.operators.util.FieldList in project flink by apache.

the class GlobalPropertiesFilteringTest method testCustomPartitioningPreserved1.

@Test
public void testCustomPartitioningPreserved1() {
    SingleInputSemanticProperties sprops = new SingleInputSemanticProperties();
    SemanticPropUtil.getSemanticPropsSingleFromString(sprops, new String[] { "0;1;4" }, null, null, tupleInfo, tupleInfo);
    GlobalProperties gprops = new GlobalProperties();
    Partitioner<Tuple2<Long, Integer>> myP = new MockPartitioner();
    gprops.setCustomPartitioned(new FieldList(0, 4), myP);
    GlobalProperties result = gprops.filterBySemanticProperties(sprops, 0);
    assertEquals(PartitioningProperty.CUSTOM_PARTITIONING, result.getPartitioning());
    FieldList pFields = result.getPartitioningFields();
    assertEquals(2, pFields.size());
    assertTrue(pFields.contains(0));
    assertTrue(pFields.contains(4));
    assertEquals(myP, result.getCustomPartitioner());
}
Also used : Tuple2(org.apache.flink.api.java.tuple.Tuple2) SingleInputSemanticProperties(org.apache.flink.api.common.operators.SingleInputSemanticProperties) FieldList(org.apache.flink.api.common.operators.util.FieldList) Test(org.junit.Test)

Aggregations

FieldList (org.apache.flink.api.common.operators.util.FieldList)79 Test (org.junit.Test)69 SingleInputPlanNode (org.apache.flink.optimizer.plan.SingleInputPlanNode)30 OptimizedPlan (org.apache.flink.optimizer.plan.OptimizedPlan)27 SingleInputSemanticProperties (org.apache.flink.api.common.operators.SingleInputSemanticProperties)26 Plan (org.apache.flink.api.common.Plan)25 Ordering (org.apache.flink.api.common.operators.Ordering)24 ExecutionEnvironment (org.apache.flink.api.java.ExecutionEnvironment)24 Tuple2 (org.apache.flink.api.java.tuple.Tuple2)24 SinkPlanNode (org.apache.flink.optimizer.plan.SinkPlanNode)22 DiscardingOutputFormat (org.apache.flink.api.java.io.DiscardingOutputFormat)21 SourcePlanNode (org.apache.flink.optimizer.plan.SourcePlanNode)17 FieldSet (org.apache.flink.api.common.operators.util.FieldSet)14 GlobalProperties (org.apache.flink.optimizer.dataproperties.GlobalProperties)12 DualInputPlanNode (org.apache.flink.optimizer.plan.DualInputPlanNode)12 Channel (org.apache.flink.optimizer.plan.Channel)11 RequestedGlobalProperties (org.apache.flink.optimizer.dataproperties.RequestedGlobalProperties)9 LocalProperties (org.apache.flink.optimizer.dataproperties.LocalProperties)7 PlanNode (org.apache.flink.optimizer.plan.PlanNode)7 DataSet (org.apache.flink.api.java.DataSet)6