Examples with PythonMapPartition org.apache.flink.python.api.functions.PythonMapPartition used on opensource projects

Example 1 with PythonMapPartition

use of org.apache.flink.python.api.functions.PythonMapPartition in project flink by apache.

the class PythonPlanBinder method createMapOperation.

@SuppressWarnings("unchecked")
private void createMapOperation(PythonOperationInfo info) {
    DataSet op1 = (DataSet) sets.get(info.parentID);
    sets.put(info.setID, op1.mapPartition(new PythonMapPartition(info.setID, info.types)).setParallelism(getParallelism(info)).name(info.name));
}

Example 2 with PythonMapPartition

use of org.apache.flink.python.api.functions.PythonMapPartition in project flink by apache.

the class PythonPlanBinder method createCrossOperation.

@SuppressWarnings("unchecked")
private void createCrossOperation(DatasizeHint mode, PythonOperationInfo info) {
    DataSet op1 = (DataSet) sets.get(info.parentID);
    DataSet op2 = (DataSet) sets.get(info.otherID);
    DefaultCross defaultResult;
    switch(mode) {
        case NONE:
            defaultResult = op1.cross(op2);
            break;
        case HUGE:
            defaultResult = op1.crossWithHuge(op2);
            break;
        case TINY:
            defaultResult = op1.crossWithTiny(op2);
            break;
        default:
            throw new IllegalArgumentException("Invalid Cross mode specified: " + mode);
    }
    defaultResult.setParallelism(getParallelism(info));
    if (info.usesUDF) {
        sets.put(info.setID, defaultResult.mapPartition(new PythonMapPartition(info.setID, info.types)).setParallelism(getParallelism(info)).name(info.name));
    } else {
        sets.put(info.setID, defaultResult.name("DefaultCross"));
    }
}

Example 3 with PythonMapPartition

use of org.apache.flink.python.api.functions.PythonMapPartition in project flink by apache.

the class PythonPlanBinder method createMapPartitionOperation.

@SuppressWarnings("unchecked")
private void createMapPartitionOperation(PythonOperationInfo info) {
    DataSet op1 = (DataSet) sets.get(info.parentID);
    sets.put(info.setID, op1.mapPartition(new PythonMapPartition(info.setID, info.types)).setParallelism(getParallelism(info)).name(info.name));
}

Example 4 with PythonMapPartition

use of org.apache.flink.python.api.functions.PythonMapPartition in project flink by apache.

the class PythonPlanBinder method createFilterOperation.

@SuppressWarnings("unchecked")
private void createFilterOperation(PythonOperationInfo info) {
    DataSet op1 = (DataSet) sets.get(info.parentID);
    sets.put(info.setID, op1.mapPartition(new PythonMapPartition(info.setID, info.types)).setParallelism(getParallelism(info)).name(info.name));
}

Example 5 with PythonMapPartition

use of org.apache.flink.python.api.functions.PythonMapPartition in project flink by apache.

the class PythonPlanBinder method createFlatMapOperation.

@SuppressWarnings("unchecked")
private void createFlatMapOperation(PythonOperationInfo info) {
    DataSet op1 = (DataSet) sets.get(info.parentID);
    sets.put(info.setID, op1.mapPartition(new PythonMapPartition(info.setID, info.types)).setParallelism(getParallelism(info)).name(info.name));
}