Search in sources :

Example 6 with PulsarSpout

use of org.apache.pulsar.storm.PulsarSpout in project incubator-pulsar by apache.

the class StormExample method main.

public static void main(String[] args) throws PulsarClientException {
    ClientConfiguration clientConf = new ClientConfiguration();
    // String authPluginClassName = "org.apache.pulsar.client.impl.auth.MyAuthentication";
    // String authParams = "key1:val1,key2:val2";
    // clientConf.setAuthentication(authPluginClassName, authParams);
    String topic1 = "persistent://my-property/use/my-ns/my-topic1";
    String topic2 = "persistent://my-property/use/my-ns/my-topic2";
    String subscriptionName1 = "my-subscriber-name1";
    String subscriptionName2 = "my-subscriber-name2";
    // create spout
    PulsarSpoutConfiguration spoutConf = new PulsarSpoutConfiguration();
    spoutConf.setServiceUrl(serviceUrl);
    spoutConf.setTopic(topic1);
    spoutConf.setSubscriptionName(subscriptionName1);
    spoutConf.setMessageToValuesMapper(messageToValuesMapper);
    PulsarSpout spout = new PulsarSpout(spoutConf, clientConf);
    // create bolt
    PulsarBoltConfiguration boltConf = new PulsarBoltConfiguration();
    boltConf.setServiceUrl(serviceUrl);
    boltConf.setTopic(topic2);
    boltConf.setTupleToMessageMapper(tupleToMessageMapper);
    PulsarBolt bolt = new PulsarBolt(boltConf, clientConf);
    TopologyBuilder builder = new TopologyBuilder();
    builder.setSpout("testSpout", spout);
    builder.setBolt("testBolt", bolt).shuffleGrouping("testSpout");
    Config conf = new Config();
    conf.setNumWorkers(2);
    conf.setDebug(true);
    conf.registerMetricsConsumer(PulsarMetricsConsumer.class);
    LocalCluster cluster = new LocalCluster();
    cluster.submitTopology("test", conf, builder.createTopology());
    Utils.sleep(10000);
    PulsarClient pulsarClient = PulsarClient.create(serviceUrl, clientConf);
    // create a consumer on topic2 to receive messages from the bolt when the processing is done
    Consumer consumer = pulsarClient.subscribe(topic2, subscriptionName2);
    // create a producer on topic1 to send messages that will be received by the spout
    Producer producer = pulsarClient.createProducer(topic1);
    for (int i = 0; i < 10; i++) {
        String msg = "msg-" + i;
        producer.send(msg.getBytes());
        LOG.info("Message {} sent", msg);
    }
    Message msg = null;
    for (int i = 0; i < 10; i++) {
        msg = consumer.receive(1, TimeUnit.SECONDS);
        LOG.info("Message {} received", new String(msg.getData()));
    }
    cluster.killTopology("test");
    cluster.shutdown();
}
Also used : LocalCluster(org.apache.storm.LocalCluster) PulsarBolt(org.apache.pulsar.storm.PulsarBolt) Message(org.apache.pulsar.client.api.Message) TopologyBuilder(org.apache.storm.topology.TopologyBuilder) Config(org.apache.storm.Config) PulsarBoltConfiguration(org.apache.pulsar.storm.PulsarBoltConfiguration) PulsarSpout(org.apache.pulsar.storm.PulsarSpout) PulsarSpoutConfiguration(org.apache.pulsar.storm.PulsarSpoutConfiguration) IMetricsConsumer(org.apache.storm.metric.api.IMetricsConsumer) Consumer(org.apache.pulsar.client.api.Consumer) Producer(org.apache.pulsar.client.api.Producer) PulsarClient(org.apache.pulsar.client.api.PulsarClient) ClientConfiguration(org.apache.pulsar.client.api.ClientConfiguration)

Aggregations

ClientConfiguration (org.apache.pulsar.client.api.ClientConfiguration)6 PulsarSpout (org.apache.pulsar.storm.PulsarSpout)6 SpoutOutputCollector (org.apache.storm.spout.SpoutOutputCollector)4 TopologyContext (org.apache.storm.task.TopologyContext)4 Test (org.testng.annotations.Test)4 PulsarSpoutConfiguration (org.apache.pulsar.storm.PulsarSpoutConfiguration)3 ConsumerConfiguration (org.apache.pulsar.client.api.ConsumerConfiguration)2 PersistentTopicStats (org.apache.pulsar.common.policies.data.PersistentTopicStats)2 Consumer (org.apache.pulsar.client.api.Consumer)1 Message (org.apache.pulsar.client.api.Message)1 Producer (org.apache.pulsar.client.api.Producer)1 PulsarClient (org.apache.pulsar.client.api.PulsarClient)1 PulsarBolt (org.apache.pulsar.storm.PulsarBolt)1 PulsarBoltConfiguration (org.apache.pulsar.storm.PulsarBoltConfiguration)1 Config (org.apache.storm.Config)1 LocalCluster (org.apache.storm.LocalCluster)1 IMetricsConsumer (org.apache.storm.metric.api.IMetricsConsumer)1 TopologyBuilder (org.apache.storm.topology.TopologyBuilder)1