Search in sources :

Example 1 with ProcessorNode

use of org.apache.storm.trident.planner.ProcessorNode in project storm by apache.

the class Stream method stateQuery.

public Stream stateQuery(TridentState state, Fields inputFields, QueryFunction function, Fields functionFields) {
    projectionValidation(inputFields);
    String stateId = state._node.stateInfo.id;
    Node n = new ProcessorNode(_topology.getUniqueStreamId(), _name, TridentUtils.fieldsConcat(getOutputFields(), functionFields), functionFields, new StateQueryProcessor(stateId, inputFields, function));
    _topology._colocate.get(stateId).add(n);
    return _topology.addSourcedNode(this, n);
}
Also used : ProcessorNode(org.apache.storm.trident.planner.ProcessorNode) ProcessorNode(org.apache.storm.trident.planner.ProcessorNode) Node(org.apache.storm.trident.planner.Node) PartitionNode(org.apache.storm.trident.planner.PartitionNode) StateQueryProcessor(org.apache.storm.trident.planner.processor.StateQueryProcessor)

Example 2 with ProcessorNode

use of org.apache.storm.trident.planner.ProcessorNode in project storm by apache.

the class Stream method partitionPersist.

public TridentState partitionPersist(StateSpec stateSpec, Fields inputFields, StateUpdater updater, Fields functionFields) {
    projectionValidation(inputFields);
    String id = _topology.getUniqueStateId();
    ProcessorNode n = new ProcessorNode(_topology.getUniqueStreamId(), _name, functionFields, functionFields, new PartitionPersistProcessor(id, inputFields, updater));
    n.committer = true;
    n.stateInfo = new NodeStateInfo(id, stateSpec);
    return _topology.addSourcedStateNode(this, n);
}
Also used : PartitionPersistProcessor(org.apache.storm.trident.planner.processor.PartitionPersistProcessor) ProcessorNode(org.apache.storm.trident.planner.ProcessorNode) NodeStateInfo(org.apache.storm.trident.planner.NodeStateInfo)

Example 3 with ProcessorNode

use of org.apache.storm.trident.planner.ProcessorNode in project storm by apache.

the class Stream method window.

private Stream window(WindowConfig windowConfig, WindowsStoreFactory windowStoreFactory, Fields inputFields, Aggregator aggregator, Fields functionFields, boolean storeTuplesInStore) {
    projectionValidation(inputFields);
    windowConfig.validate();
    Fields fields = addTriggerField(functionFields);
    // when storeTuplesInStore is false then the given windowStoreFactory is only used to store triggers and
    // that store is passed to WindowStateUpdater to remove them after committing the batch.
    Stream stream = _topology.addSourcedNode(this, new ProcessorNode(_topology.getUniqueStreamId(), _name, fields, fields, new WindowTridentProcessor(windowConfig, _topology.getUniqueWindowId(), windowStoreFactory, inputFields, aggregator, storeTuplesInStore)));
    Stream effectiveStream = stream.project(functionFields);
    // create StateUpdater with the given windowStoreFactory to remove triggered aggregation results form store
    // when they are successfully processed.
    StateFactory stateFactory = new WindowsStateFactory();
    StateUpdater stateUpdater = new WindowsStateUpdater(windowStoreFactory);
    stream.partitionPersist(stateFactory, new Fields(WindowTridentProcessor.TRIGGER_FIELD_NAME), stateUpdater, new Fields());
    return effectiveStream;
}
Also used : Fields(org.apache.storm.tuple.Fields) ProcessorNode(org.apache.storm.trident.planner.ProcessorNode) StateFactory(org.apache.storm.trident.state.StateFactory) WindowsStateFactory(org.apache.storm.trident.windowing.WindowsStateFactory) WindowsStateUpdater(org.apache.storm.trident.windowing.WindowsStateUpdater) GroupedStream(org.apache.storm.trident.fluent.GroupedStream) IAggregatableStream(org.apache.storm.trident.fluent.IAggregatableStream) WindowTridentProcessor(org.apache.storm.trident.windowing.WindowTridentProcessor) WindowsStateFactory(org.apache.storm.trident.windowing.WindowsStateFactory) CombinerAggStateUpdater(org.apache.storm.trident.operation.impl.CombinerAggStateUpdater) ReducerAggStateUpdater(org.apache.storm.trident.operation.impl.ReducerAggStateUpdater) WindowsStateUpdater(org.apache.storm.trident.windowing.WindowsStateUpdater) StateUpdater(org.apache.storm.trident.state.StateUpdater)

Example 4 with ProcessorNode

use of org.apache.storm.trident.planner.ProcessorNode in project storm by apache.

the class TridentTopology method multiReduce.

public Stream multiReduce(List<Fields> inputFields, List<Stream> streams, MultiReducer function, Fields outputFields) {
    List<String> names = new ArrayList<>();
    for (Stream s : streams) {
        if (s._name != null) {
            names.add(s._name);
        }
    }
    Node n = new ProcessorNode(getUniqueStreamId(), Utils.join(names, "-"), outputFields, outputFields, new MultiReducerProcessor(inputFields, function));
    return addSourcedNode(streams, n);
}
Also used : ProcessorNode(org.apache.storm.trident.planner.ProcessorNode) ProcessorNode(org.apache.storm.trident.planner.ProcessorNode) Node(org.apache.storm.trident.planner.Node) PartitionNode(org.apache.storm.trident.planner.PartitionNode) SpoutNode(org.apache.storm.trident.planner.SpoutNode) GroupedStream(org.apache.storm.trident.fluent.GroupedStream) IAggregatableStream(org.apache.storm.trident.fluent.IAggregatableStream) MultiReducerProcessor(org.apache.storm.trident.planner.processor.MultiReducerProcessor)

Aggregations

ProcessorNode (org.apache.storm.trident.planner.ProcessorNode)4 GroupedStream (org.apache.storm.trident.fluent.GroupedStream)2 IAggregatableStream (org.apache.storm.trident.fluent.IAggregatableStream)2 Node (org.apache.storm.trident.planner.Node)2 PartitionNode (org.apache.storm.trident.planner.PartitionNode)2 CombinerAggStateUpdater (org.apache.storm.trident.operation.impl.CombinerAggStateUpdater)1 ReducerAggStateUpdater (org.apache.storm.trident.operation.impl.ReducerAggStateUpdater)1 NodeStateInfo (org.apache.storm.trident.planner.NodeStateInfo)1 SpoutNode (org.apache.storm.trident.planner.SpoutNode)1 MultiReducerProcessor (org.apache.storm.trident.planner.processor.MultiReducerProcessor)1 PartitionPersistProcessor (org.apache.storm.trident.planner.processor.PartitionPersistProcessor)1 StateQueryProcessor (org.apache.storm.trident.planner.processor.StateQueryProcessor)1 StateFactory (org.apache.storm.trident.state.StateFactory)1 StateUpdater (org.apache.storm.trident.state.StateUpdater)1 WindowTridentProcessor (org.apache.storm.trident.windowing.WindowTridentProcessor)1 WindowsStateFactory (org.apache.storm.trident.windowing.WindowsStateFactory)1 WindowsStateUpdater (org.apache.storm.trident.windowing.WindowsStateUpdater)1 Fields (org.apache.storm.tuple.Fields)1