Search in sources :

Example 1 with GroupedMultiReducerExecutor

use of storm.trident.operation.impl.GroupedMultiReducerExecutor in project storm by nathanmarz.

the class TridentTopology method multiReduce.

public Stream multiReduce(List<Fields> inputFields, List<GroupedStream> groupedStreams, GroupedMultiReducer function, Fields outputFields) {
    List<Fields> fullInputFields = new ArrayList<Fields>();
    List<Stream> streams = new ArrayList<Stream>();
    List<Fields> fullGroupFields = new ArrayList<Fields>();
    for (int i = 0; i < groupedStreams.size(); i++) {
        GroupedStream gs = groupedStreams.get(i);
        Fields groupFields = gs.getGroupFields();
        fullGroupFields.add(groupFields);
        streams.add(gs.toStream().partitionBy(groupFields));
        fullInputFields.add(TridentUtils.fieldsUnion(groupFields, inputFields.get(i)));
    }
    return multiReduce(fullInputFields, streams, new GroupedMultiReducerExecutor(function, fullGroupFields, inputFields), outputFields);
}
Also used : Fields(backtype.storm.tuple.Fields) GroupedStream(storm.trident.fluent.GroupedStream) GroupedMultiReducerExecutor(storm.trident.operation.impl.GroupedMultiReducerExecutor) ArrayList(java.util.ArrayList) GroupedStream(storm.trident.fluent.GroupedStream) IAggregatableStream(storm.trident.fluent.IAggregatableStream)

Example 2 with GroupedMultiReducerExecutor

use of storm.trident.operation.impl.GroupedMultiReducerExecutor in project jstorm by alibaba.

the class TridentTopology method multiReduce.

public Stream multiReduce(List<Fields> inputFields, List<GroupedStream> groupedStreams, GroupedMultiReducer function, Fields outputFields) {
    List<Fields> fullInputFields = new ArrayList<>();
    List<Stream> streams = new ArrayList<>();
    List<Fields> fullGroupFields = new ArrayList<>();
    for (int i = 0; i < groupedStreams.size(); i++) {
        GroupedStream gs = groupedStreams.get(i);
        Fields groupFields = gs.getGroupFields();
        fullGroupFields.add(groupFields);
        streams.add(gs.toStream().partitionBy(groupFields));
        fullInputFields.add(TridentUtils.fieldsUnion(groupFields, inputFields.get(i)));
    }
    return multiReduce(fullInputFields, streams, new GroupedMultiReducerExecutor(function, fullGroupFields, inputFields), outputFields);
}
Also used : Fields(backtype.storm.tuple.Fields) GroupedStream(storm.trident.fluent.GroupedStream) GroupedMultiReducerExecutor(storm.trident.operation.impl.GroupedMultiReducerExecutor) ArrayList(java.util.ArrayList) GroupedStream(storm.trident.fluent.GroupedStream) IAggregatableStream(storm.trident.fluent.IAggregatableStream)

Aggregations

Fields (backtype.storm.tuple.Fields)2 ArrayList (java.util.ArrayList)2 GroupedStream (storm.trident.fluent.GroupedStream)2 IAggregatableStream (storm.trident.fluent.IAggregatableStream)2 GroupedMultiReducerExecutor (storm.trident.operation.impl.GroupedMultiReducerExecutor)2