Search in sources :

Example 1 with ChainedProcessorContext

use of org.apache.storm.streams.processors.ChainedProcessorContext in project storm by apache.

the class ProcessorBoltDelegate method prepare.

void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
    this.stormConf = stormConf;
    topologyContext = context;
    outputCollector = collector;
    DirectedSubgraph<Node, Edge> subgraph = new DirectedSubgraph<>(graph, new HashSet<>(nodes), null);
    TopologicalOrderIterator<Node, Edge> it = new TopologicalOrderIterator<>(subgraph);
    while (it.hasNext()) {
        Node node = it.next();
        if (!(node instanceof ProcessorNode)) {
            throw new IllegalStateException("Not a processor node " + node);
        }
        ProcessorNode processorNode = (ProcessorNode) node;
        List<ProcessorNode> children = StreamUtil.getChildren(subgraph, processorNode);
        ProcessorContext processorContext;
        if (children.isEmpty()) {
            processorContext = createEmittingContext(processorNode);
        } else {
            Multimap<String, ProcessorNode> streamToChildren = ArrayListMultimap.create();
            for (ProcessorNode child : children) {
                for (String stream : child.getParentStreams(processorNode)) {
                    streamToChildren.put(stream, child);
                }
            }
            ForwardingProcessorContext forwardingContext = new ForwardingProcessorContext(processorNode, streamToChildren);
            if (hasOutgoingChild(processorNode, new HashSet<>(children))) {
                processorContext = new ChainedProcessorContext(processorNode, forwardingContext, createEmittingContext(processorNode));
            } else {
                processorContext = forwardingContext;
            }
        }
        processorNode.initProcessorContext(processorContext);
    }
    if (timestampField != null) {
        for (EmittingProcessorContext ctx : emittingProcessorContexts) {
            ctx.setTimestampField(timestampField);
        }
    }
    for (String stream : streamToInitialProcessors.keySet()) {
        streamToInputTaskCount.put(stream, getStreamInputTaskCount(context, stream));
    }
}
Also used : ForwardingProcessorContext(org.apache.storm.streams.processors.ForwardingProcessorContext) DirectedSubgraph(org.jgrapht.graph.DirectedSubgraph) TopologicalOrderIterator(org.jgrapht.traverse.TopologicalOrderIterator) ChainedProcessorContext(org.apache.storm.streams.processors.ChainedProcessorContext) ProcessorContext(org.apache.storm.streams.processors.ProcessorContext) EmittingProcessorContext(org.apache.storm.streams.processors.EmittingProcessorContext) ForwardingProcessorContext(org.apache.storm.streams.processors.ForwardingProcessorContext) ChainedProcessorContext(org.apache.storm.streams.processors.ChainedProcessorContext) EmittingProcessorContext(org.apache.storm.streams.processors.EmittingProcessorContext)

Example 2 with ChainedProcessorContext

use of org.apache.storm.streams.processors.ChainedProcessorContext in project storm by apache.

the class ProcessorBoltDelegate method createEmittingContext.

private ProcessorContext createEmittingContext(ProcessorNode processorNode) {
    List<EmittingProcessorContext> emittingContexts = new ArrayList<>();
    for (String stream : processorNode.getOutputStreams()) {
        EmittingProcessorContext emittingContext = new EmittingProcessorContext(processorNode, outputCollector, stream);
        emittingContexts.add(emittingContext);
    }
    emittingProcessorContexts.addAll(emittingContexts);
    outgoingProcessors.add(processorNode);
    return new ChainedProcessorContext(processorNode, emittingContexts);
}
Also used : ArrayList(java.util.ArrayList) EmittingProcessorContext(org.apache.storm.streams.processors.EmittingProcessorContext) ChainedProcessorContext(org.apache.storm.streams.processors.ChainedProcessorContext)

Example 3 with ChainedProcessorContext

use of org.apache.storm.streams.processors.ChainedProcessorContext in project storm by apache.

the class ProcessorBoltDelegate method prepare.

void prepare(Map<String, Object> topoConf, TopologyContext context, OutputCollector collector) {
    this.topoConf = topoConf;
    topologyContext = context;
    outputCollector = collector;
    DirectedSubgraph<Node, Edge> subgraph = new DirectedSubgraph<>(graph, new HashSet<>(nodes), null);
    TopologicalOrderIterator<Node, Edge> it = new TopologicalOrderIterator<>(subgraph);
    while (it.hasNext()) {
        Node node = it.next();
        if (!(node instanceof ProcessorNode)) {
            throw new IllegalStateException("Not a processor node " + node);
        }
        ProcessorNode processorNode = (ProcessorNode) node;
        List<ProcessorNode> children = StreamUtil.getChildren(subgraph, processorNode);
        ProcessorContext processorContext;
        if (children.isEmpty()) {
            processorContext = createEmittingContext(processorNode);
        } else {
            Multimap<String, ProcessorNode> streamToChildren = ArrayListMultimap.create();
            for (ProcessorNode child : children) {
                for (String stream : child.getParentStreams(processorNode)) {
                    streamToChildren.put(stream, child);
                }
            }
            ForwardingProcessorContext forwardingContext = new ForwardingProcessorContext(processorNode, streamToChildren);
            if (hasOutgoingChild(processorNode, new HashSet<>(children))) {
                processorContext = new ChainedProcessorContext(processorNode, forwardingContext, createEmittingContext(processorNode));
            } else {
                processorContext = forwardingContext;
            }
        }
        processorNode.initProcessorContext(processorContext);
    }
    if (timestampField != null) {
        for (EmittingProcessorContext ctx : emittingProcessorContexts) {
            ctx.setTimestampField(timestampField);
        }
    }
    for (String stream : streamToInitialProcessors.keySet()) {
        streamToInputTaskCount.put(stream, getStreamInputTaskCount(context, stream));
    }
}
Also used : ForwardingProcessorContext(org.apache.storm.streams.processors.ForwardingProcessorContext) DirectedSubgraph(org.apache.storm.shade.org.jgrapht.graph.DirectedSubgraph) TopologicalOrderIterator(org.apache.storm.shade.org.jgrapht.traverse.TopologicalOrderIterator) ChainedProcessorContext(org.apache.storm.streams.processors.ChainedProcessorContext) ProcessorContext(org.apache.storm.streams.processors.ProcessorContext) EmittingProcessorContext(org.apache.storm.streams.processors.EmittingProcessorContext) ForwardingProcessorContext(org.apache.storm.streams.processors.ForwardingProcessorContext) ChainedProcessorContext(org.apache.storm.streams.processors.ChainedProcessorContext) EmittingProcessorContext(org.apache.storm.streams.processors.EmittingProcessorContext)

Aggregations

ChainedProcessorContext (org.apache.storm.streams.processors.ChainedProcessorContext)3 EmittingProcessorContext (org.apache.storm.streams.processors.EmittingProcessorContext)3 ForwardingProcessorContext (org.apache.storm.streams.processors.ForwardingProcessorContext)2 ProcessorContext (org.apache.storm.streams.processors.ProcessorContext)2 ArrayList (java.util.ArrayList)1 DirectedSubgraph (org.apache.storm.shade.org.jgrapht.graph.DirectedSubgraph)1 TopologicalOrderIterator (org.apache.storm.shade.org.jgrapht.traverse.TopologicalOrderIterator)1 DirectedSubgraph (org.jgrapht.graph.DirectedSubgraph)1 TopologicalOrderIterator (org.jgrapht.traverse.TopologicalOrderIterator)1