Search in sources :

Example 1 with GroupedStream

use of org.apache.storm.trident.fluent.GroupedStream in project storm by apache.

the class TridentTopology method multiReduce.

public Stream multiReduce(List<Fields> inputFields, List<GroupedStream> groupedStreams, GroupedMultiReducer function, Fields outputFields) {
    List<Fields> fullInputFields = new ArrayList<>();
    List<Stream> streams = new ArrayList<>();
    List<Fields> fullGroupFields = new ArrayList<>();
    for (int i = 0; i < groupedStreams.size(); i++) {
        GroupedStream gs = groupedStreams.get(i);
        Fields groupFields = gs.getGroupFields();
        fullGroupFields.add(groupFields);
        streams.add(gs.toStream().partitionBy(groupFields));
        fullInputFields.add(TridentUtils.fieldsUnion(groupFields, inputFields.get(i)));
    }
    return multiReduce(fullInputFields, streams, new GroupedMultiReducerExecutor(function, fullGroupFields, inputFields), outputFields);
}
Also used : Fields(org.apache.storm.tuple.Fields) GroupedStream(org.apache.storm.trident.fluent.GroupedStream) GroupedMultiReducerExecutor(org.apache.storm.trident.operation.impl.GroupedMultiReducerExecutor) GroupedStream(org.apache.storm.trident.fluent.GroupedStream) IAggregatableStream(org.apache.storm.trident.fluent.IAggregatableStream)

Aggregations

GroupedStream (org.apache.storm.trident.fluent.GroupedStream)1 IAggregatableStream (org.apache.storm.trident.fluent.IAggregatableStream)1 GroupedMultiReducerExecutor (org.apache.storm.trident.operation.impl.GroupedMultiReducerExecutor)1 Fields (org.apache.storm.tuple.Fields)1