Search in sources :

Example 1 with PointFormatter

use of org.apache.flink.test.util.PointFormatter in project flink by apache.

the class IterationWithChainingITCase method testProgram.

@Override
protected void testProgram() throws Exception {
    ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    env.setParallelism(4);
    DataSet<Tuple2<Integer, CoordVector>> initialInput = env.readFile(new PointInFormat(), dataPath).setParallelism(1).name("Input");
    IterativeDataSet<Tuple2<Integer, CoordVector>> iteration = initialInput.iterate(2).name("Loop");
    DataSet<Tuple2<Integer, CoordVector>> identity = iteration.groupBy(0).reduceGroup(new GroupReduceFunction<Tuple2<Integer, CoordVector>, Tuple2<Integer, CoordVector>>() {

        @Override
        public void reduce(Iterable<Tuple2<Integer, CoordVector>> values, Collector<Tuple2<Integer, CoordVector>> out) throws Exception {
            for (Tuple2<Integer, CoordVector> value : values) {
                out.collect(value);
            }
        }
    }).map(new MapFunction<Tuple2<Integer, CoordVector>, Tuple2<Integer, CoordVector>>() {

        @Override
        public Tuple2<Integer, CoordVector> map(Tuple2<Integer, CoordVector> value) throws Exception {
            return value;
        }
    });
    iteration.closeWith(identity).writeAsFormattedText(resultPath, new PointFormatter());
    env.execute("Iteration with chained map test");
    compareResultsByLinesInMemory(DATA_POINTS, resultPath);
}
Also used : ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) GroupReduceFunction(org.apache.flink.api.common.functions.GroupReduceFunction) CoordVector(org.apache.flink.test.util.CoordVector) PointFormatter(org.apache.flink.test.util.PointFormatter) PointInFormat(org.apache.flink.test.util.PointInFormat) Tuple2(org.apache.flink.api.java.tuple.Tuple2) Collector(org.apache.flink.util.Collector)

Example 2 with PointFormatter

use of org.apache.flink.test.util.PointFormatter in project flink by apache.

the class IterationWithUnionITCase method testProgram.

@Override
protected void testProgram() throws Exception {
    ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    DataSet<Tuple2<Integer, CoordVector>> initialInput = env.readFile(new PointInFormat(), this.dataPath).setParallelism(1);
    IterativeDataSet<Tuple2<Integer, CoordVector>> iteration = initialInput.iterate(2);
    DataSet<Tuple2<Integer, CoordVector>> result = iteration.union(iteration).map(new IdentityMapper());
    iteration.closeWith(result).writeAsFormattedText(this.resultPath, new PointFormatter());
    env.execute();
}
Also used : ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) PointInFormat(org.apache.flink.test.util.PointInFormat) Tuple2(org.apache.flink.api.java.tuple.Tuple2) PointFormatter(org.apache.flink.test.util.PointFormatter)

Aggregations

ExecutionEnvironment (org.apache.flink.api.java.ExecutionEnvironment)2 Tuple2 (org.apache.flink.api.java.tuple.Tuple2)2 PointFormatter (org.apache.flink.test.util.PointFormatter)2 PointInFormat (org.apache.flink.test.util.PointInFormat)2 GroupReduceFunction (org.apache.flink.api.common.functions.GroupReduceFunction)1 CoordVector (org.apache.flink.test.util.CoordVector)1 Collector (org.apache.flink.util.Collector)1