Search in sources :

Example 1 with WindowTridentProcessor

use of storm.trident.windowing.WindowTridentProcessor in project jstorm by alibaba.

the class Stream method window.

private Stream window(WindowConfig windowConfig, WindowsStoreFactory windowStoreFactory, Fields inputFields, Aggregator aggregator, Fields functionFields, boolean storeTuplesInStore) {
    projectionValidation(inputFields);
    windowConfig.validate();
    Fields fields = addTriggerField(functionFields);
    // when storeTuplesInStore is false then the given windowStoreFactory is only used to store triggers and
    // that store is passed to WindowStateUpdater to remove them after committing the batch.
    Stream stream = _topology.addSourcedNode(this, new ProcessorNode(_topology.getUniqueStreamId(), _name, fields, fields, new WindowTridentProcessor(windowConfig, _topology.getUniqueWindowId(), windowStoreFactory, inputFields, aggregator, storeTuplesInStore)));
    Stream effectiveStream = stream.project(functionFields);
    // create StateUpdater with the given windowStoreFactory to remove triggered aggregation results form store
    // when they are successfully processed.
    StateFactory stateFactory = new WindowsStateFactory();
    StateUpdater stateUpdater = new WindowsStateUpdater(windowStoreFactory);
    stream.partitionPersist(stateFactory, new Fields(WindowTridentProcessor.TRIGGER_FIELD_NAME), stateUpdater, new Fields());
    return effectiveStream;
}
Also used : Fields(backtype.storm.tuple.Fields) ProcessorNode(storm.trident.planner.ProcessorNode) WindowsStateFactory(storm.trident.windowing.WindowsStateFactory) StateFactory(storm.trident.state.StateFactory) WindowsStateUpdater(storm.trident.windowing.WindowsStateUpdater) GroupedStream(storm.trident.fluent.GroupedStream) IAggregatableStream(storm.trident.fluent.IAggregatableStream) WindowTridentProcessor(storm.trident.windowing.WindowTridentProcessor) WindowsStateFactory(storm.trident.windowing.WindowsStateFactory) ReducerAggStateUpdater(storm.trident.operation.impl.ReducerAggStateUpdater) WindowsStateUpdater(storm.trident.windowing.WindowsStateUpdater) StateUpdater(storm.trident.state.StateUpdater) CombinerAggStateUpdater(storm.trident.operation.impl.CombinerAggStateUpdater)

Aggregations

Fields (backtype.storm.tuple.Fields)1 GroupedStream (storm.trident.fluent.GroupedStream)1 IAggregatableStream (storm.trident.fluent.IAggregatableStream)1 CombinerAggStateUpdater (storm.trident.operation.impl.CombinerAggStateUpdater)1 ReducerAggStateUpdater (storm.trident.operation.impl.ReducerAggStateUpdater)1 ProcessorNode (storm.trident.planner.ProcessorNode)1 StateFactory (storm.trident.state.StateFactory)1 StateUpdater (storm.trident.state.StateUpdater)1 WindowTridentProcessor (storm.trident.windowing.WindowTridentProcessor)1 WindowsStateFactory (storm.trident.windowing.WindowsStateFactory)1 WindowsStateUpdater (storm.trident.windowing.WindowsStateUpdater)1