Search in sources :

Example 1 with PulsarSpoutConfiguration

use of com.yahoo.pulsar.storm.PulsarSpoutConfiguration in project pulsar by yahoo.

the class StormExample method main.

public static void main(String[] args) throws PulsarClientException {
    ClientConfiguration clientConf = new ClientConfiguration();
    // String authPluginClassName = "com.yahoo.pulsar.client.impl.auth.MyAuthentication";
    // String authParams = "key1:val1,key2:val2";
    // clientConf.setAuthentication(authPluginClassName, authParams);
    String topic1 = "persistent://my-property/use/my-ns/my-topic1";
    String topic2 = "persistent://my-property/use/my-ns/my-topic2";
    String subscriptionName1 = "my-subscriber-name1";
    String subscriptionName2 = "my-subscriber-name2";
    // create spout
    PulsarSpoutConfiguration spoutConf = new PulsarSpoutConfiguration();
    spoutConf.setServiceUrl(serviceUrl);
    spoutConf.setTopic(topic1);
    spoutConf.setSubscriptionName(subscriptionName1);
    spoutConf.setMessageToValuesMapper(messageToValuesMapper);
    PulsarSpout spout = new PulsarSpout(spoutConf, clientConf);
    // create bolt
    PulsarBoltConfiguration boltConf = new PulsarBoltConfiguration();
    boltConf.setServiceUrl(serviceUrl);
    boltConf.setTopic(topic2);
    boltConf.setTupleToMessageMapper(tupleToMessageMapper);
    PulsarBolt bolt = new PulsarBolt(boltConf, clientConf);
    TopologyBuilder builder = new TopologyBuilder();
    builder.setSpout("testSpout", spout);
    builder.setBolt("testBolt", bolt).shuffleGrouping("testSpout");
    Config conf = new Config();
    conf.setNumWorkers(2);
    conf.setDebug(true);
    conf.registerMetricsConsumer(PulsarMetricsConsumer.class);
    LocalCluster cluster = new LocalCluster();
    cluster.submitTopology("test", conf, builder.createTopology());
    Utils.sleep(10000);
    PulsarClient pulsarClient = PulsarClient.create(serviceUrl, clientConf);
    // create a consumer on topic2 to receive messages from the bolt when the processing is done
    Consumer consumer = pulsarClient.subscribe(topic2, subscriptionName2);
    // create a producer on topic1 to send messages that will be received by the spout
    Producer producer = pulsarClient.createProducer(topic1);
    for (int i = 0; i < 10; i++) {
        String msg = "msg-" + i;
        producer.send(msg.getBytes());
        LOG.info("Message {} sent", msg);
    }
    Message msg = null;
    for (int i = 0; i < 10; i++) {
        msg = consumer.receive(1, TimeUnit.SECONDS);
        LOG.info("Message {} received", new String(msg.getData()));
    }
    cluster.killTopology("test");
    cluster.shutdown();
}
Also used : LocalCluster(backtype.storm.LocalCluster) PulsarBolt(com.yahoo.pulsar.storm.PulsarBolt) Message(com.yahoo.pulsar.client.api.Message) TopologyBuilder(backtype.storm.topology.TopologyBuilder) Config(backtype.storm.Config) PulsarBoltConfiguration(com.yahoo.pulsar.storm.PulsarBoltConfiguration) PulsarSpout(com.yahoo.pulsar.storm.PulsarSpout) PulsarSpoutConfiguration(com.yahoo.pulsar.storm.PulsarSpoutConfiguration) Consumer(com.yahoo.pulsar.client.api.Consumer) IMetricsConsumer(backtype.storm.metric.api.IMetricsConsumer) Producer(com.yahoo.pulsar.client.api.Producer) PulsarClient(com.yahoo.pulsar.client.api.PulsarClient) ClientConfiguration(com.yahoo.pulsar.client.api.ClientConfiguration)

Aggregations

Config (backtype.storm.Config)1 LocalCluster (backtype.storm.LocalCluster)1 IMetricsConsumer (backtype.storm.metric.api.IMetricsConsumer)1 TopologyBuilder (backtype.storm.topology.TopologyBuilder)1 ClientConfiguration (com.yahoo.pulsar.client.api.ClientConfiguration)1 Consumer (com.yahoo.pulsar.client.api.Consumer)1 Message (com.yahoo.pulsar.client.api.Message)1 Producer (com.yahoo.pulsar.client.api.Producer)1 PulsarClient (com.yahoo.pulsar.client.api.PulsarClient)1 PulsarBolt (com.yahoo.pulsar.storm.PulsarBolt)1 PulsarBoltConfiguration (com.yahoo.pulsar.storm.PulsarBoltConfiguration)1 PulsarSpout (com.yahoo.pulsar.storm.PulsarSpout)1 PulsarSpoutConfiguration (com.yahoo.pulsar.storm.PulsarSpoutConfiguration)1