Examples with Fields backtype.storm.tuple.Fields used on opensource projects

Example 56 with Fields

use of backtype.storm.tuple.Fields in project jstorm by alibaba.

the class TridentTopologySource method getTopology.

public StormTopology getTopology(Config config) {
    this.spout = new FixedBatchSpout(new Fields("sentence"), 20, new Values("one two"), new Values("two three"), new Values("three four"), new Values("four five"), new Values("five six"));
    TridentTopology trident = new TridentTopology();
    trident.newStream("wordcount", spout).name("sentence").parallelismHint(1).shuffle().each(new Fields("sentence"), new Split(), new Fields("word")).parallelismHint(1).groupBy(new Fields("word")).persistentAggregate(new MemoryMapState.Factory(), new Count(), new Fields("count")).parallelismHint(1);
    return trident.build();
}

Example 57 with Fields

use of backtype.storm.tuple.Fields in project jstorm by alibaba.

the class Stream method window.

private Stream window(WindowConfig windowConfig, WindowsStoreFactory windowStoreFactory, Fields inputFields, Aggregator aggregator, Fields functionFields, boolean storeTuplesInStore) {
    projectionValidation(inputFields);
    windowConfig.validate();
    Fields fields = addTriggerField(functionFields);
    // when storeTuplesInStore is false then the given windowStoreFactory is only used to store triggers and
    // that store is passed to WindowStateUpdater to remove them after committing the batch.
    Stream stream = _topology.addSourcedNode(this, new ProcessorNode(_topology.getUniqueStreamId(), _name, fields, fields, new WindowTridentProcessor(windowConfig, _topology.getUniqueWindowId(), windowStoreFactory, inputFields, aggregator, storeTuplesInStore)));
    Stream effectiveStream = stream.project(functionFields);
    // create StateUpdater with the given windowStoreFactory to remove triggered aggregation results form store
    // when they are successfully processed.
    StateFactory stateFactory = new WindowsStateFactory();
    StateUpdater stateUpdater = new WindowsStateUpdater(windowStoreFactory);
    stream.partitionPersist(stateFactory, new Fields(WindowTridentProcessor.TRIGGER_FIELD_NAME), stateUpdater, new Fields());
    return effectiveStream;
}

Example 58 with Fields

use of backtype.storm.tuple.Fields in project jstorm by alibaba.

the class TridentTopology method multiReduce.

public Stream multiReduce(List<Fields> inputFields, List<GroupedStream> groupedStreams, GroupedMultiReducer function, Fields outputFields) {
    List<Fields> fullInputFields = new ArrayList<>();
    List<Stream> streams = new ArrayList<>();
    List<Fields> fullGroupFields = new ArrayList<>();
    for (int i = 0; i < groupedStreams.size(); i++) {
        GroupedStream gs = groupedStreams.get(i);
        Fields groupFields = gs.getGroupFields();
        fullGroupFields.add(groupFields);
        streams.add(gs.toStream().partitionBy(groupFields));
        fullInputFields.add(TridentUtils.fieldsUnion(groupFields, inputFields.get(i)));
    }
    return multiReduce(fullInputFields, streams, new GroupedMultiReducerExecutor(function, fullGroupFields, inputFields), outputFields);
}

Example 59 with Fields

use of backtype.storm.tuple.Fields in project jstorm by alibaba.

the class TridentTopology method newDRPCStream.

private Stream newDRPCStream(DRPCSpout spout) {
    // TODO: consider adding a shuffle grouping after the spout to avoid so much routing of the args/return-info all over the place
    // (at least until its possible to just pack bolt logic into the spout itself)
    Node n = new SpoutNode(getUniqueStreamId(), TridentUtils.getSingleOutputStreamFields(spout), null, spout, SpoutNode.SpoutType.DRPC);
    Stream nextStream = addNode(n);
    // later on, this will be joined back with return-info and all the results
    return nextStream.project(new Fields("args"));
}

Example 60 with Fields

use of backtype.storm.tuple.Fields in project jstorm by alibaba.

the class TridentTopology method newStaticState.

public TridentState newStaticState(StateSpec spec) {
    String stateId = getUniqueStateId();
    Node n = new Node(getUniqueStreamId(), null, new Fields());
    n.stateInfo = new NodeStateInfo(stateId, spec);
    registerNode(n);
    return new TridentState(this, n);
}