Examples with Fields backtype.storm.tuple.Fields used on opensource projects

Example 46 with Fields

use of backtype.storm.tuple.Fields in project jstorm by alibaba.

the class CoordinatedBolt method declareOutputFields.

public void declareOutputFields(OutputFieldsDeclarer declarer) {
    _delegate.declareOutputFields(declarer);
    declarer.declareStream(Constants.COORDINATED_STREAM_ID, true, new Fields("id", "count"));
}

Example 47 with Fields

use of backtype.storm.tuple.Fields in project jstorm by alibaba.

the class SkewedRollingTopWordsTest method testSkewedRollingTopWords.

@Test
public void testSkewedRollingTopWords() {
    TopologyBuilder topologyBuilder = new TopologyBuilder();
    topologyBuilder.setSpout("windowTestWordSpout", new WindowTestWordSpout(), 5);
    topologyBuilder.setBolt("windowTestRollingCountBolt", new WindowTestRollingCountBolt(9, 3), 4).partialKeyGrouping("windowTestWordSpout", new Fields("word"));
    topologyBuilder.setBolt("windowTestCountAggBolt", new WindowTestCountAggBolt(), 4).fieldsGrouping("windowTestRollingCountBolt", new Fields("obj"));
    topologyBuilder.setBolt("windowTestIntermediateRankingBolt", new WindowTestIntermediateRankingBolt(DEFAULT_COUNT), 4).fieldsGrouping("windowTestCountAggBolt", new Fields("obj"));
    topologyBuilder.setBolt("windowTestTotalRankingsBolt", new WindowTestTotalRankingsBolt(DEFAULT_COUNT)).globalGrouping("windowTestIntermediateRankingBolt");
    Map config = new HashMap();
    config.put(Config.TOPOLOGY_NAME, "SkewedRollingTopWordsTest");
    // I really don't know how to validate if the result is right since
    // the tick time is not precise. It makes the output after passing
    // a window is unpredictable.
    // Now I just let it pass all the time.
    // TODO:FIX ME: how to validate if the result is right?
    JStormUnitTestRunner.submitTopology(topologyBuilder.createTopology(), config, 90, null);
}

Example 48 with Fields

use of backtype.storm.tuple.Fields in project jstorm by alibaba.

the class ClojureSpout method declareOutputFields.

@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
    for (String stream : _fields.keySet()) {
        StreamInfo info = _fields.get(stream);
        declarer.declareStream(stream, info.is_direct(), new Fields(info.get_output_fields()));
    }
}

Example 49 with Fields

use of backtype.storm.tuple.Fields in project jstorm by alibaba.

the class Common method outbound_components.

/**
 * get current task's output <Stream_id, <componentId, MkGrouper>>
 */
public static Map<String, Map<String, MkGrouper>> outbound_components(TopologyContext topology_context, WorkerData workerData) {
    Map<String, Map<String, MkGrouper>> rr = new HashMap<>();
    // <Stream_id,<component,Grouping>>
    Map<String, Map<String, Grouping>> output_groupings = topology_context.getThisTargets();
    for (Entry<String, Map<String, Grouping>> entry : output_groupings.entrySet()) {
        String stream_id = entry.getKey();
        Map<String, Grouping> component_grouping = entry.getValue();
        Fields out_fields = topology_context.getThisOutputFields(stream_id);
        Map<String, MkGrouper> componentGrouper = new HashMap<>();
        for (Entry<String, Grouping> cg : component_grouping.entrySet()) {
            String component = cg.getKey();
            Grouping tgrouping = cg.getValue();
            List<Integer> outTasks = topology_context.getComponentTasks(component);
            // so we don't need send tuple to it
            if (outTasks.size() > 0) {
                MkGrouper grouper = new MkGrouper(topology_context, out_fields, tgrouping, component, stream_id, workerData);
                componentGrouper.put(component, grouper);
            }
            LOG.info("outbound_components, {}-{} for task-{} on {}", component, outTasks, topology_context.getThisTaskId(), stream_id);
        }
        if (componentGrouper.size() > 0) {
            rr.put(stream_id, componentGrouper);
        }
    }
    return rr;
}

Example 50 with Fields

use of backtype.storm.tuple.Fields in project jstorm by alibaba.

the class WorkerData method generateComponentToStreamToFields.

public HashMap<String, Map<String, Fields>> generateComponentToStreamToFields(StormTopology topology) {
    HashMap<String, Map<String, Fields>> componentToStreamToFields = new HashMap<>();
    Set<String> components = ThriftTopologyUtils.getComponentIds(topology);
    for (String component : components) {
        Map<String, Fields> streamToFieldsMap = new HashMap<>();
        Map<String, StreamInfo> streamInfoMap = ThriftTopologyUtils.getComponentCommon(topology, component).get_streams();
        for (Map.Entry<String, StreamInfo> entry : streamInfoMap.entrySet()) {
            String streamId = entry.getKey();
            StreamInfo streamInfo = entry.getValue();
            streamToFieldsMap.put(streamId, new Fields(streamInfo.get_output_fields()));
        }
        componentToStreamToFields.put(component, streamToFieldsMap);
    }
    return componentToStreamToFields;
}