Search in sources :

Example 6 with SchemeAsMultiScheme

use of org.apache.storm.spout.SchemeAsMultiScheme in project heron by twitter.

the class KafkaUtilsTest method generateTuplesWithValueSchemeAndKeyValueMessage.

@Test
public void generateTuplesWithValueSchemeAndKeyValueMessage() {
    config.scheme = new SchemeAsMultiScheme(new StringScheme());
    String value = "value";
    String key = "key";
    createTopicAndSendMessage(key, value);
    ByteBufferMessageSet messageAndOffsets = getLastMessage();
    for (MessageAndOffset msg : messageAndOffsets) {
        Iterable<List<Object>> lists = KafkaUtils.generateTuples(config, msg.message(), config.topic);
        assertEquals(value, lists.iterator().next().get(0));
    }
}
Also used : ArrayList(java.util.ArrayList) List(java.util.List) MessageAndOffset(kafka.message.MessageAndOffset) SchemeAsMultiScheme(org.apache.storm.spout.SchemeAsMultiScheme) ByteBufferMessageSet(kafka.javaapi.message.ByteBufferMessageSet) Test(org.junit.Test)

Example 7 with SchemeAsMultiScheme

use of org.apache.storm.spout.SchemeAsMultiScheme in project pancm_project by xuwujing.

the class MykafkaSpout method main.

/**
 * The entry point of application.
 *
 * @param args the input arguments
 */
/*
	 * 通过zookeeper进行获取kafka的数据
	 */
public static void main(String[] args) {
    String topic = "pcm_test1";
    ZkHosts zkHosts = new ZkHosts("192.169.0.23:2181");
    SpoutConfig spoutConfig = new SpoutConfig(zkHosts, topic, "", "MyTrack");
    List<String> zkServers = new ArrayList<String>();
    zkServers.add("192.169.0.23");
    spoutConfig.zkServers = zkServers;
    spoutConfig.zkPort = 2181;
    spoutConfig.socketTimeoutMs = 60 * 1000;
    spoutConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
    TopologyBuilder builder = new TopologyBuilder();
    // 设置1个Executeor(线程),默认一个
    builder.setSpout("spout", new KafkaSpout(spoutConfig), 1);
    // 设置storm 设置1个Executeor(线程) 没有设置Task,默认一个
    builder.setBolt("bolt1", new MyKafkaBolt(), 1).shuffleGrouping("spout");
    Config conf = new Config();
    conf.setDebug(false);
    if (args.length > 0) {
        System.out.println("远程模式");
        try {
            StormSubmitter.submitTopology(args[0], conf, builder.createTopology());
        } catch (AlreadyAliveException e) {
            e.printStackTrace();
        } catch (InvalidTopologyException e) {
            e.printStackTrace();
        } catch (org.apache.storm.generated.AuthorizationException e) {
            e.printStackTrace();
        }
    } else {
        System.out.println("本地模式");
        LocalCluster localCluster = new LocalCluster();
        localCluster.submitTopology("mytopology", conf, builder.createTopology());
    }
}
Also used : LocalCluster(org.apache.storm.LocalCluster) TopologyBuilder(org.apache.storm.topology.TopologyBuilder) SpoutConfig(org.apache.storm.kafka.SpoutConfig) Config(org.apache.storm.Config) SpoutConfig(org.apache.storm.kafka.SpoutConfig) InvalidTopologyException(org.apache.storm.generated.InvalidTopologyException) ZkHosts(org.apache.storm.kafka.ZkHosts) ArrayList(java.util.ArrayList) AlreadyAliveException(org.apache.storm.generated.AlreadyAliveException) SchemeAsMultiScheme(org.apache.storm.spout.SchemeAsMultiScheme) KafkaSpout(org.apache.storm.kafka.KafkaSpout) StringScheme(org.apache.storm.kafka.StringScheme)

Aggregations

SchemeAsMultiScheme (org.apache.storm.spout.SchemeAsMultiScheme)7 ArrayList (java.util.ArrayList)4 Test (org.junit.Test)4 ZkHosts (org.apache.storm.kafka.ZkHosts)3 List (java.util.List)2 ByteBufferMessageSet (kafka.javaapi.message.ByteBufferMessageSet)2 MessageAndOffset (kafka.message.MessageAndOffset)2 StringScheme (org.apache.storm.kafka.StringScheme)2 Config (org.apache.storm.Config)1 LocalCluster (org.apache.storm.LocalCluster)1 AlreadyAliveException (org.apache.storm.generated.AlreadyAliveException)1 InvalidTopologyException (org.apache.storm.generated.InvalidTopologyException)1 KafkaSpout (org.apache.storm.kafka.KafkaSpout)1 SpoutConfig (org.apache.storm.kafka.SpoutConfig)1 TridentKafkaConfig (org.apache.storm.kafka.trident.TridentKafkaConfig)1 Scheme (org.apache.storm.spout.Scheme)1 FieldInfo (org.apache.storm.sql.runtime.FieldInfo)1 IOutputSerializer (org.apache.storm.sql.runtime.IOutputSerializer)1 TopologyBuilder (org.apache.storm.topology.TopologyBuilder)1