Search in sources :

Example 1 with SchemeAsMultiScheme

use of org.apache.storm.spout.SchemeAsMultiScheme in project storm by apache.

the class TridentKafkaWordCount method newTridentKafkaConfig.

private static TridentKafkaConfig newTridentKafkaConfig(String zkUrl) {
    ZkHosts hosts = new ZkHosts(zkUrl);
    TridentKafkaConfig config = new TridentKafkaConfig(hosts, "test");
    config.scheme = new SchemeAsMultiScheme(new StringScheme());
    // Consume new data from the topic
    config.startOffsetTime = kafka.api.OffsetRequest.LatestTime();
    return config;
}
Also used : ZkHosts(org.apache.storm.kafka.ZkHosts) SchemeAsMultiScheme(org.apache.storm.spout.SchemeAsMultiScheme) StringScheme(org.apache.storm.kafka.StringScheme)

Example 2 with SchemeAsMultiScheme

use of org.apache.storm.spout.SchemeAsMultiScheme in project storm by apache.

the class KafkaDataSourcesProvider method constructTrident.

@Override
public ISqlTridentDataSource constructTrident(URI uri, String inputFormatClass, String outputFormatClass, Properties properties, List<FieldInfo> fields) {
    int port = uri.getPort() != -1 ? uri.getPort() : DEFAULT_ZK_PORT;
    ZkHosts zk = new ZkHosts(uri.getHost() + ":" + port, uri.getPath());
    Map<String, String> values = parseURIParams(uri.getQuery());
    String topic = values.get("topic");
    Preconditions.checkNotNull(topic, "No topic of the spout is specified");
    TridentKafkaConfig conf = new TridentKafkaConfig(zk, topic);
    List<String> fieldNames = new ArrayList<>();
    int primaryIndex = -1;
    for (int i = 0; i < fields.size(); ++i) {
        FieldInfo f = fields.get(i);
        fieldNames.add(f.name());
        if (f.isPrimary()) {
            primaryIndex = i;
        }
    }
    Preconditions.checkState(primaryIndex != -1, "Kafka stream table must have a primary key");
    Scheme scheme = SerdeUtils.getScheme(inputFormatClass, properties, fieldNames);
    conf.scheme = new SchemeAsMultiScheme(scheme);
    IOutputSerializer serializer = SerdeUtils.getSerializer(outputFormatClass, properties, fieldNames);
    return new KafkaTridentDataSource(conf, topic, primaryIndex, properties, serializer);
}
Also used : IOutputSerializer(org.apache.storm.sql.runtime.IOutputSerializer) SchemeAsMultiScheme(org.apache.storm.spout.SchemeAsMultiScheme) Scheme(org.apache.storm.spout.Scheme) ZkHosts(org.apache.storm.kafka.ZkHosts) ArrayList(java.util.ArrayList) TridentKafkaConfig(org.apache.storm.kafka.trident.TridentKafkaConfig) SchemeAsMultiScheme(org.apache.storm.spout.SchemeAsMultiScheme) FieldInfo(org.apache.storm.sql.runtime.FieldInfo)

Example 3 with SchemeAsMultiScheme

use of org.apache.storm.spout.SchemeAsMultiScheme in project heron by twitter.

the class KafkaUtilsTest method generateTupelsWithValueScheme.

@Test
public void generateTupelsWithValueScheme() {
    config.scheme = new SchemeAsMultiScheme(new StringScheme());
    runGetValueOnlyTuplesTest();
}
Also used : SchemeAsMultiScheme(org.apache.storm.spout.SchemeAsMultiScheme) Test(org.junit.Test)

Example 4 with SchemeAsMultiScheme

use of org.apache.storm.spout.SchemeAsMultiScheme in project storm by apache.

the class KafkaUtilsTest method generateTuplesWithValueSchemeAndKeyValueMessage.

@Test
public void generateTuplesWithValueSchemeAndKeyValueMessage() {
    config.scheme = new SchemeAsMultiScheme(new StringScheme());
    String value = "value";
    String key = "key";
    createTopicAndSendMessage(key, value);
    ByteBufferMessageSet messageAndOffsets = getLastMessage();
    for (MessageAndOffset msg : messageAndOffsets) {
        Iterable<List<Object>> lists = KafkaUtils.generateTuples(config, msg.message(), config.topic);
        assertEquals(value, lists.iterator().next().get(0));
    }
}
Also used : ArrayList(java.util.ArrayList) List(java.util.List) MessageAndOffset(kafka.message.MessageAndOffset) SchemeAsMultiScheme(org.apache.storm.spout.SchemeAsMultiScheme) ByteBufferMessageSet(kafka.javaapi.message.ByteBufferMessageSet) Test(org.junit.Test)

Example 5 with SchemeAsMultiScheme

use of org.apache.storm.spout.SchemeAsMultiScheme in project storm by apache.

the class KafkaUtilsTest method generateTupelsWithValueScheme.

@Test
public void generateTupelsWithValueScheme() {
    config.scheme = new SchemeAsMultiScheme(new StringScheme());
    runGetValueOnlyTuplesTest();
}
Also used : SchemeAsMultiScheme(org.apache.storm.spout.SchemeAsMultiScheme) Test(org.junit.Test)

Aggregations

SchemeAsMultiScheme (org.apache.storm.spout.SchemeAsMultiScheme)7 ArrayList (java.util.ArrayList)4 Test (org.junit.Test)4 ZkHosts (org.apache.storm.kafka.ZkHosts)3 List (java.util.List)2 ByteBufferMessageSet (kafka.javaapi.message.ByteBufferMessageSet)2 MessageAndOffset (kafka.message.MessageAndOffset)2 StringScheme (org.apache.storm.kafka.StringScheme)2 Config (org.apache.storm.Config)1 LocalCluster (org.apache.storm.LocalCluster)1 AlreadyAliveException (org.apache.storm.generated.AlreadyAliveException)1 InvalidTopologyException (org.apache.storm.generated.InvalidTopologyException)1 KafkaSpout (org.apache.storm.kafka.KafkaSpout)1 SpoutConfig (org.apache.storm.kafka.SpoutConfig)1 TridentKafkaConfig (org.apache.storm.kafka.trident.TridentKafkaConfig)1 Scheme (org.apache.storm.spout.Scheme)1 FieldInfo (org.apache.storm.sql.runtime.FieldInfo)1 IOutputSerializer (org.apache.storm.sql.runtime.IOutputSerializer)1 TopologyBuilder (org.apache.storm.topology.TopologyBuilder)1