Search in sources :

Example 26 with LocalCluster

use of backtype.storm.LocalCluster in project jstorm by alibaba.

the class SequenceTopologyTool method SetLocalTopology.

public void SetLocalTopology() throws Exception {
    Config conf = getConf();
    StormTopology topology = buildTopology();
    LocalCluster cluster = new LocalCluster();
    cluster.submitTopology("SplitMerge", conf, topology);
    Thread.sleep(60000);
    cluster.shutdown();
}
Also used : LocalCluster(backtype.storm.LocalCluster) Config(backtype.storm.Config) StormTopology(backtype.storm.generated.StormTopology)

Example 27 with LocalCluster

use of backtype.storm.LocalCluster in project jstorm by alibaba.

the class ReachTopology method main.

public static void main(String[] args) throws Exception {
    LinearDRPCTopologyBuilder builder = construct();
    Config conf = new Config();
    conf.setNumWorkers(6);
    if (args.length != 0) {
        try {
            Map yamlConf = LoadConf.LoadYaml(args[0]);
            if (yamlConf != null) {
                conf.putAll(yamlConf);
            }
        } catch (Exception e) {
            System.out.println("Input " + args[0] + " isn't one yaml ");
        }
        StormSubmitter.submitTopology(TOPOLOGY_NAME, conf, builder.createRemoteTopology());
    } else {
        conf.setMaxTaskParallelism(3);
        LocalDRPC drpc = new LocalDRPC();
        LocalCluster cluster = new LocalCluster();
        cluster.submitTopology(TOPOLOGY_NAME, conf, builder.createLocalTopology(drpc));
        JStormUtils.sleepMs(10000);
        String[] urlsToTry = new String[] { "foo.com/blog/1", "engineering.twitter.com/blog/5", "notaurl.com" };
        for (String url : urlsToTry) {
            System.out.println("Reach of " + url + ": " + drpc.execute(TOPOLOGY_NAME, url));
        }
        cluster.shutdown();
        drpc.shutdown();
    }
}
Also used : LocalCluster(backtype.storm.LocalCluster) Config(backtype.storm.Config) LocalDRPC(backtype.storm.LocalDRPC) LinearDRPCTopologyBuilder(backtype.storm.drpc.LinearDRPCTopologyBuilder) HashMap(java.util.HashMap) Map(java.util.Map)

Example 28 with LocalCluster

use of backtype.storm.LocalCluster in project jstorm by alibaba.

the class TransactionalWordsTest method test_transaction_word.

@Test
public void test_transaction_word() {
    try {
        MemoryTransactionalSpout spout = new MemoryTransactionalSpout(DATA, new Fields("word"), PARTITION_TAKE_PER_BATCH);
        TransactionalTopologyBuilder builder = new TransactionalTopologyBuilder("top-n-words", "spout", spout, 2);
        builder.setBolt("count", new KeyedCountUpdater(), 5).fieldsGrouping("spout", new Fields("word"));
        builder.setBolt("bucketize", new Bucketize()).shuffleGrouping("count");
        builder.setBolt("buckets", new BucketCountUpdater(), 5).fieldsGrouping("bucketize", new Fields("bucket"));
        LocalCluster cluster = new LocalCluster();
        Config config = new Config();
        config.setDebug(true);
        config.setMaxSpoutPending(3);
        cluster.submitTopology("top-n-topology", config, builder.buildTopology());
        JStormUtils.sleepMs(60 * 1000);
        assertEquals(false, outOfOrder.get());
        assertNotSame(0, receiveCounter1.get());
        assertNotSame(0, receiveCounter2.get());
    // cluster.killTopology("top-n-topology");
    // cluster.shutdown();
    } catch (Exception e) {
        e.printStackTrace();
        Assert.fail("Failed to run simple transaction");
    }
}
Also used : LocalCluster(backtype.storm.LocalCluster) MemoryTransactionalSpout(backtype.storm.testing.MemoryTransactionalSpout) Fields(backtype.storm.tuple.Fields) Config(backtype.storm.Config) TransactionalTopologyBuilder(backtype.storm.transactional.TransactionalTopologyBuilder) Test(org.junit.Test)

Example 29 with LocalCluster

use of backtype.storm.LocalCluster in project jstorm by alibaba.

the class SingleJoinTest method test_single_join.

@Test
public void test_single_join() {
    receiveCounter.set(0);
    try {
        FeederSpout genderSpout = new FeederSpout(new Fields("id", "gender"));
        FeederSpout ageSpout = new FeederSpout(new Fields("id", "age"));
        TopologyBuilder builder = new TopologyBuilder();
        builder.setSpout("gender", genderSpout);
        builder.setSpout("age", ageSpout);
        builder.setBolt("join", new SingleJoinBolt(new Fields("gender", "age"))).fieldsGrouping("gender", new Fields("id")).fieldsGrouping("age", new Fields("id"));
        Config conf = new Config();
        conf.setDebug(true);
        LocalCluster cluster = new LocalCluster();
        cluster.submitTopology("join-example", conf, builder.createTopology());
        for (int i = 0; i < 10; i++) {
            String gender;
            if (i % 2 == 0) {
                gender = "male";
            } else {
                gender = "female";
            }
            genderSpout.feed(new Values(i, gender));
        }
        for (int i = 9; i >= 0; i--) {
            ageSpout.feed(new Values(i, i + 20));
        }
        JStormUtils.sleepMs(60 * 1000);
        assertNotSame(0, receiveCounter.get());
        cluster.shutdown();
    } catch (Exception e) {
        Assert.fail("Failed to run SingleJoinExample");
    }
}
Also used : LocalCluster(backtype.storm.LocalCluster) Fields(backtype.storm.tuple.Fields) TopologyBuilder(backtype.storm.topology.TopologyBuilder) Config(backtype.storm.Config) Values(backtype.storm.tuple.Values) FeederSpout(backtype.storm.testing.FeederSpout) Test(org.junit.Test)

Example 30 with LocalCluster

use of backtype.storm.LocalCluster in project jstorm by alibaba.

the class BatchMetaTopology method SetLocalTopology.

public static void SetLocalTopology() throws Exception {
    TopologyBuilder builder = SetBuilder();
    LocalCluster cluster = new LocalCluster();
    cluster.submitTopology(topologyName, conf, builder.createTopology());
    Thread.sleep(600000);
    cluster.shutdown();
}
Also used : LocalCluster(backtype.storm.LocalCluster) TopologyBuilder(backtype.storm.topology.TopologyBuilder) BatchTopologyBuilder(com.alibaba.jstorm.batch.BatchTopologyBuilder)

Aggregations

LocalCluster (backtype.storm.LocalCluster)38 Config (backtype.storm.Config)31 TopologyBuilder (backtype.storm.topology.TopologyBuilder)23 Fields (backtype.storm.tuple.Fields)12 LocalDRPC (backtype.storm.LocalDRPC)3 TupleTableConfig (backtype.storm.contrib.hbase.utils.TupleTableConfig)3 TridentConfig (backtype.storm.contrib.hbase.utils.TridentConfig)2 LinearDRPCTopologyBuilder (backtype.storm.drpc.LinearDRPCTopologyBuilder)2 StormTopology (backtype.storm.generated.StormTopology)2 MemoryTransactionalSpout (backtype.storm.testing.MemoryTransactionalSpout)2 TransactionalTopologyBuilder (backtype.storm.transactional.TransactionalTopologyBuilder)2 ArrayList (java.util.ArrayList)2 Test (org.junit.Test)2 KafkaConfig (storm.kafka.KafkaConfig)2 KafkaSpout (storm.kafka.KafkaSpout)2 SpoutConfig (storm.kafka.SpoutConfig)2 TridentTopology (storm.trident.TridentTopology)2 FixedBatchSpout (storm.trident.testing.FixedBatchSpout)2 HBaseBolt (backtype.storm.contrib.hbase.bolts.HBaseBolt)1 HBaseCountersBatchBolt (backtype.storm.contrib.hbase.bolts.HBaseCountersBatchBolt)1