Search in sources :

Example 1 with SimpleQueryFilterCreator

use of org.apache.storm.mongodb.common.SimpleQueryFilterCreator in project storm by apache.

the class WordCountTridentMap method buildTopology.

public static StormTopology buildTopology(String url, String collectionName) {
    Fields fields = new Fields("word", "count");
    FixedBatchSpout spout = new FixedBatchSpout(fields, 4, new Values("storm", 1), new Values("trident", 1), new Values("needs", 1), new Values("javadoc", 1));
    spout.setCycle(true);
    MongoMapper mapper = new SimpleMongoMapper().withFields("word", "count");
    MongoMapState.Options options = new MongoMapState.Options();
    options.url = url;
    options.collectionName = collectionName;
    options.mapper = mapper;
    QueryFilterCreator filterCreator = new SimpleQueryFilterCreator().withField("word");
    options.queryCreator = filterCreator;
    StateFactory factory = MongoMapState.transactional(options);
    TridentTopology topology = new TridentTopology();
    Stream stream = topology.newStream("spout1", spout);
    TridentState state = stream.groupBy(new Fields("word")).persistentAggregate(factory, new Fields("count"), new Sum(), new Fields("sum"));
    stream.stateQuery(state, new Fields("word"), new MapGet(), new Fields("sum")).each(new Fields("word", "sum"), new PrintFunction(), new Fields());
    return topology.build();
}
Also used : MongoMapState(org.apache.storm.mongodb.trident.state.MongoMapState) TridentState(org.apache.storm.trident.TridentState) Values(org.apache.storm.tuple.Values) Sum(org.apache.storm.trident.operation.builtin.Sum) MapGet(org.apache.storm.trident.operation.builtin.MapGet) SimpleQueryFilterCreator(org.apache.storm.mongodb.common.SimpleQueryFilterCreator) QueryFilterCreator(org.apache.storm.mongodb.common.QueryFilterCreator) SimpleQueryFilterCreator(org.apache.storm.mongodb.common.SimpleQueryFilterCreator) FixedBatchSpout(org.apache.storm.trident.testing.FixedBatchSpout) Fields(org.apache.storm.tuple.Fields) SimpleMongoMapper(org.apache.storm.mongodb.common.mapper.SimpleMongoMapper) MongoMapper(org.apache.storm.mongodb.common.mapper.MongoMapper) StateFactory(org.apache.storm.trident.state.StateFactory) TridentTopology(org.apache.storm.trident.TridentTopology) Stream(org.apache.storm.trident.Stream) SimpleMongoMapper(org.apache.storm.mongodb.common.mapper.SimpleMongoMapper)

Example 2 with SimpleQueryFilterCreator

use of org.apache.storm.mongodb.common.SimpleQueryFilterCreator in project storm by apache.

the class LookupWordCount method main.

public static void main(String[] args) throws Exception {
    String url = TEST_MONGODB_URL;
    String collectionName = TEST_MONGODB_COLLECTION_NAME;
    if (args.length >= 2) {
        url = args[0];
        collectionName = args[1];
    }
    WordSpout spout = new WordSpout();
    TotalWordCounter totalBolt = new TotalWordCounter();
    MongoLookupMapper mapper = new SimpleMongoLookupMapper().withFields("word", "count");
    QueryFilterCreator filterCreator = new SimpleQueryFilterCreator().withField("word");
    MongoLookupBolt lookupBolt = new MongoLookupBolt(url, collectionName, filterCreator, mapper);
    // wordspout -> lookupbolt -> totalCountBolt
    TopologyBuilder builder = new TopologyBuilder();
    builder.setSpout(WORD_SPOUT, spout, 1);
    builder.setBolt(LOOKUP_BOLT, lookupBolt, 1).shuffleGrouping(WORD_SPOUT);
    builder.setBolt(TOTAL_COUNT_BOLT, totalBolt, 1).fieldsGrouping(LOOKUP_BOLT, new Fields("word"));
    String topoName = "test";
    if (args.length == 3) {
        topoName = args[2];
    } else if (args.length > 3) {
        System.out.println("Usage: LookupWordCount <mongodb url> <mongodb collection> [topology name]");
        return;
    }
    Config config = new Config();
    StormSubmitter.submitTopology(topoName, config, builder.createTopology());
}
Also used : MongoLookupBolt(org.apache.storm.mongodb.bolt.MongoLookupBolt) Fields(org.apache.storm.tuple.Fields) TopologyBuilder(org.apache.storm.topology.TopologyBuilder) Config(org.apache.storm.Config) MongoLookupMapper(org.apache.storm.mongodb.common.mapper.MongoLookupMapper) SimpleMongoLookupMapper(org.apache.storm.mongodb.common.mapper.SimpleMongoLookupMapper) SimpleQueryFilterCreator(org.apache.storm.mongodb.common.SimpleQueryFilterCreator) SimpleQueryFilterCreator(org.apache.storm.mongodb.common.SimpleQueryFilterCreator) QueryFilterCreator(org.apache.storm.mongodb.common.QueryFilterCreator) SimpleMongoLookupMapper(org.apache.storm.mongodb.common.mapper.SimpleMongoLookupMapper)

Example 3 with SimpleQueryFilterCreator

use of org.apache.storm.mongodb.common.SimpleQueryFilterCreator in project storm by apache.

the class UpdateWordCount method main.

public static void main(String[] args) throws Exception {
    String url = TEST_MONGODB_URL;
    String collectionName = TEST_MONGODB_COLLECTION_NAME;
    if (args.length >= 2) {
        url = args[0];
        collectionName = args[1];
    }
    WordSpout spout = new WordSpout();
    WordCounter bolt = new WordCounter();
    MongoUpdateMapper mapper = new SimpleMongoUpdateMapper().withFields("word", "count");
    QueryFilterCreator filterCreator = new SimpleQueryFilterCreator().withField("word");
    MongoUpdateBolt updateBolt = new MongoUpdateBolt(url, collectionName, filterCreator, mapper);
    // if a new document should be inserted if there are no matches to the query filter
    // updateBolt.withUpsert(true);
    // whether find all documents according to the query filter
    // updateBolt.withMany(true);
    // wordSpout ==> countBolt ==> MongoUpdateBolt
    TopologyBuilder builder = new TopologyBuilder();
    builder.setSpout(WORD_SPOUT, spout, 1);
    builder.setBolt(COUNT_BOLT, bolt, 1).shuffleGrouping(WORD_SPOUT);
    builder.setBolt(UPDATE_BOLT, updateBolt, 1).fieldsGrouping(COUNT_BOLT, new Fields("word"));
    String topoName = "test";
    if (args.length == 3) {
        topoName = args[2];
    } else if (args.length > 3) {
        System.out.println("Usage: UpdateWordCount <mongodb url> <mongodb collection> [topology name]");
        return;
    }
    Config config = new Config();
    StormSubmitter.submitTopology(topoName, config, builder.createTopology());
}
Also used : SimpleMongoUpdateMapper(org.apache.storm.mongodb.common.mapper.SimpleMongoUpdateMapper) Fields(org.apache.storm.tuple.Fields) TopologyBuilder(org.apache.storm.topology.TopologyBuilder) Config(org.apache.storm.Config) SimpleQueryFilterCreator(org.apache.storm.mongodb.common.SimpleQueryFilterCreator) SimpleQueryFilterCreator(org.apache.storm.mongodb.common.SimpleQueryFilterCreator) QueryFilterCreator(org.apache.storm.mongodb.common.QueryFilterCreator) MongoUpdateBolt(org.apache.storm.mongodb.bolt.MongoUpdateBolt) MongoUpdateMapper(org.apache.storm.mongodb.common.mapper.MongoUpdateMapper) SimpleMongoUpdateMapper(org.apache.storm.mongodb.common.mapper.SimpleMongoUpdateMapper)

Aggregations

QueryFilterCreator (org.apache.storm.mongodb.common.QueryFilterCreator)3 SimpleQueryFilterCreator (org.apache.storm.mongodb.common.SimpleQueryFilterCreator)3 Fields (org.apache.storm.tuple.Fields)3 Config (org.apache.storm.Config)2 TopologyBuilder (org.apache.storm.topology.TopologyBuilder)2 MongoLookupBolt (org.apache.storm.mongodb.bolt.MongoLookupBolt)1 MongoUpdateBolt (org.apache.storm.mongodb.bolt.MongoUpdateBolt)1 MongoLookupMapper (org.apache.storm.mongodb.common.mapper.MongoLookupMapper)1 MongoMapper (org.apache.storm.mongodb.common.mapper.MongoMapper)1 MongoUpdateMapper (org.apache.storm.mongodb.common.mapper.MongoUpdateMapper)1 SimpleMongoLookupMapper (org.apache.storm.mongodb.common.mapper.SimpleMongoLookupMapper)1 SimpleMongoMapper (org.apache.storm.mongodb.common.mapper.SimpleMongoMapper)1 SimpleMongoUpdateMapper (org.apache.storm.mongodb.common.mapper.SimpleMongoUpdateMapper)1 MongoMapState (org.apache.storm.mongodb.trident.state.MongoMapState)1 Stream (org.apache.storm.trident.Stream)1 TridentState (org.apache.storm.trident.TridentState)1 TridentTopology (org.apache.storm.trident.TridentTopology)1 MapGet (org.apache.storm.trident.operation.builtin.MapGet)1 Sum (org.apache.storm.trident.operation.builtin.Sum)1 StateFactory (org.apache.storm.trident.state.StateFactory)1