use of org.apache.storm.spout.SchemeAsMultiScheme in project storm by apache.
the class TridentKafkaWordCount method newTridentKafkaConfig.
private static TridentKafkaConfig newTridentKafkaConfig(String zkUrl) {
ZkHosts hosts = new ZkHosts(zkUrl);
TridentKafkaConfig config = new TridentKafkaConfig(hosts, "test");
config.scheme = new SchemeAsMultiScheme(new StringScheme());
// Consume new data from the topic
config.startOffsetTime = kafka.api.OffsetRequest.LatestTime();
return config;
}
use of org.apache.storm.spout.SchemeAsMultiScheme in project storm by apache.
the class KafkaDataSourcesProvider method constructTrident.
@Override
public ISqlTridentDataSource constructTrident(URI uri, String inputFormatClass, String outputFormatClass, Properties properties, List<FieldInfo> fields) {
int port = uri.getPort() != -1 ? uri.getPort() : DEFAULT_ZK_PORT;
ZkHosts zk = new ZkHosts(uri.getHost() + ":" + port, uri.getPath());
Map<String, String> values = parseURIParams(uri.getQuery());
String topic = values.get("topic");
Preconditions.checkNotNull(topic, "No topic of the spout is specified");
TridentKafkaConfig conf = new TridentKafkaConfig(zk, topic);
List<String> fieldNames = new ArrayList<>();
int primaryIndex = -1;
for (int i = 0; i < fields.size(); ++i) {
FieldInfo f = fields.get(i);
fieldNames.add(f.name());
if (f.isPrimary()) {
primaryIndex = i;
}
}
Preconditions.checkState(primaryIndex != -1, "Kafka stream table must have a primary key");
Scheme scheme = SerdeUtils.getScheme(inputFormatClass, properties, fieldNames);
conf.scheme = new SchemeAsMultiScheme(scheme);
IOutputSerializer serializer = SerdeUtils.getSerializer(outputFormatClass, properties, fieldNames);
return new KafkaTridentDataSource(conf, topic, primaryIndex, properties, serializer);
}
use of org.apache.storm.spout.SchemeAsMultiScheme in project heron by twitter.
the class KafkaUtilsTest method generateTupelsWithValueScheme.
@Test
public void generateTupelsWithValueScheme() {
config.scheme = new SchemeAsMultiScheme(new StringScheme());
runGetValueOnlyTuplesTest();
}
use of org.apache.storm.spout.SchemeAsMultiScheme in project storm by apache.
the class KafkaUtilsTest method generateTuplesWithValueSchemeAndKeyValueMessage.
@Test
public void generateTuplesWithValueSchemeAndKeyValueMessage() {
config.scheme = new SchemeAsMultiScheme(new StringScheme());
String value = "value";
String key = "key";
createTopicAndSendMessage(key, value);
ByteBufferMessageSet messageAndOffsets = getLastMessage();
for (MessageAndOffset msg : messageAndOffsets) {
Iterable<List<Object>> lists = KafkaUtils.generateTuples(config, msg.message(), config.topic);
assertEquals(value, lists.iterator().next().get(0));
}
}
use of org.apache.storm.spout.SchemeAsMultiScheme in project storm by apache.
the class KafkaUtilsTest method generateTupelsWithValueScheme.
@Test
public void generateTupelsWithValueScheme() {
config.scheme = new SchemeAsMultiScheme(new StringScheme());
runGetValueOnlyTuplesTest();
}
Aggregations