Search in sources :

Example 71 with KeySelector

use of org.apache.flink.api.java.functions.KeySelector in project flink by apache.

the class PartitionOperatorTest method testRangePartitionBySelectorComplexKeyWithTooManyOrders.

@Test(expected = IllegalArgumentException.class)
public void testRangePartitionBySelectorComplexKeyWithTooManyOrders() throws Exception {
    final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    final DataSet<NestedPojo> ds = getNestedPojoDataSet(env);
    ds.partitionByRange(new KeySelector<NestedPojo, CustomPojo>() {

        @Override
        public CustomPojo getKey(NestedPojo value) throws Exception {
            return value.getNested();
        }
    }).withOrders(Order.ASCENDING, Order.DESCENDING);
}
Also used : ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) KeySelector(org.apache.flink.api.java.functions.KeySelector) Test(org.junit.Test)

Example 72 with KeySelector

use of org.apache.flink.api.java.functions.KeySelector in project flink by apache.

the class GroupingTest method testGroupSortByKeySelector3.

@SuppressWarnings("serial")
@Test(expected = InvalidProgramException.class)
public void testGroupSortByKeySelector3() {
    final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    DataSet<Tuple4<Integer, Long, CustomType, Long[]>> tupleDs = env.fromCollection(tupleWithCustomData, tupleWithCustomInfo);
    // should not work
    tupleDs.groupBy(new KeySelector<Tuple4<Integer, Long, CustomType, Long[]>, Long>() {

        @Override
        public Long getKey(Tuple4<Integer, Long, CustomType, Long[]> value) throws Exception {
            return value.f1;
        }
    }).sortGroup(new KeySelector<Tuple4<Integer, Long, CustomType, Long[]>, Long[]>() {

        @Override
        public Long[] getKey(Tuple4<Integer, Long, CustomType, Long[]> value) throws Exception {
            return value.f3;
        }
    }, Order.ASCENDING);
}
Also used : Tuple4(org.apache.flink.api.java.tuple.Tuple4) ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) KeySelector(org.apache.flink.api.java.functions.KeySelector) InvalidProgramException(org.apache.flink.api.common.InvalidProgramException) Test(org.junit.Test)

Example 73 with KeySelector

use of org.apache.flink.api.java.functions.KeySelector in project flink by apache.

the class JoinOperatorTest method testJoinProjection26.

@Test
public void testJoinProjection26() {
    final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    DataSet<CustomType> ds1 = env.fromCollection(customTypeData);
    DataSet<CustomType> ds2 = env.fromCollection(customTypeData);
    // should work
    try {
        ds1.join(ds2).where(new KeySelector<CustomType, Long>() {

            @Override
            public Long getKey(CustomType value) {
                return value.myLong;
            }
        }).equalTo(new KeySelector<CustomType, Long>() {

            @Override
            public Long getKey(CustomType value) {
                return value.myLong;
            }
        }).projectFirst().projectSecond();
    } catch (Exception e) {
        System.out.println("FAILED: " + e);
        e.printStackTrace();
        Assert.fail();
    }
}
Also used : ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) KeySelector(org.apache.flink.api.java.functions.KeySelector) InvalidProgramException(org.apache.flink.api.common.InvalidProgramException) Test(org.junit.Test)

Example 74 with KeySelector

use of org.apache.flink.api.java.functions.KeySelector in project flink by apache.

the class JoinOperatorTest method testJoinKeyMixedKeySelectorTurned.

@Test
public void testJoinKeyMixedKeySelectorTurned() {
    final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    DataSet<CustomType> ds1 = env.fromCollection(customTypeData);
    DataSet<CustomType> ds2 = env.fromCollection(customTypeData);
    try {
        ds1.join(ds2).where(new KeySelector<CustomType, Integer>() {

            @Override
            public Integer getKey(CustomType value) throws Exception {
                return value.myInt;
            }
        }).equalTo("myInt");
    } catch (Exception e) {
        e.printStackTrace();
        Assert.fail();
    }
}
Also used : ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) KeySelector(org.apache.flink.api.java.functions.KeySelector) InvalidProgramException(org.apache.flink.api.common.InvalidProgramException) Test(org.junit.Test)

Example 75 with KeySelector

use of org.apache.flink.api.java.functions.KeySelector in project flink by apache.

the class JoinOperatorTest method testJoinKeyMixing1.

@Test
public void testJoinKeyMixing1() {
    final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    DataSet<CustomType> ds1 = env.fromCollection(customTypeData);
    DataSet<Tuple5<Integer, Long, String, Long, Integer>> ds2 = env.fromCollection(emptyTupleData, tupleTypeInfo);
    // should work
    try {
        ds1.join(ds2).where(new KeySelector<CustomType, Long>() {

            @Override
            public Long getKey(CustomType value) {
                return value.myLong;
            }
        }).equalTo(3);
    } catch (Exception e) {
        e.printStackTrace();
        Assert.fail();
    }
}
Also used : Tuple5(org.apache.flink.api.java.tuple.Tuple5) ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) KeySelector(org.apache.flink.api.java.functions.KeySelector) InvalidProgramException(org.apache.flink.api.common.InvalidProgramException) Test(org.junit.Test)

Aggregations

KeySelector (org.apache.flink.api.java.functions.KeySelector)120 Test (org.junit.Test)113 Tuple2 (org.apache.flink.api.java.tuple.Tuple2)45 StreamExecutionEnvironment (org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)44 ExecutionEnvironment (org.apache.flink.api.java.ExecutionEnvironment)39 Watermark (org.apache.flink.streaming.api.watermark.Watermark)30 List (java.util.List)29 StreamRecord (org.apache.flink.streaming.runtime.streamrecord.StreamRecord)28 InvalidProgramException (org.apache.flink.api.common.InvalidProgramException)22 JobID (org.apache.flink.api.common.JobID)22 JobGraph (org.apache.flink.runtime.jobgraph.JobGraph)22 IOException (java.io.IOException)21 Arrays (java.util.Arrays)21 AtomicLong (java.util.concurrent.atomic.AtomicLong)21 Configuration (org.apache.flink.configuration.Configuration)21 KeyedOneInputStreamOperatorTestHarness (org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness)21 ArrayList (java.util.ArrayList)18 Map (java.util.Map)18 ExecutionConfig (org.apache.flink.api.common.ExecutionConfig)18 ValueStateDescriptor (org.apache.flink.api.common.state.ValueStateDescriptor)16