Examples with KafkaSimpleStreamingSource org.apache.gobblin.source.extractor.extract.kafka.KafkaSimpleStreamingSource used on opensource projects

Example 1 with KafkaSimpleStreamingSource

use of org.apache.gobblin.source.extractor.extract.kafka.KafkaSimpleStreamingSource in project incubator-gobblin by apache.

the class KafkaSimpleStreamingTest method getWorkUnits.

private List<WorkUnit> getWorkUnits(String topic) {
    SourceState ss = new SourceState();
    ss.setProp(ConfigurationKeys.KAFKA_BROKERS, "localhost:" + _kafkaTestHelper.getKafkaServerPort());
    ss.setProp(KafkaSimpleStreamingSource.TOPIC_WHITELIST, topic);
    ss.setProp(ConfigurationKeys.JOB_NAME_KEY, topic);
    ss.setProp(KafkaSimpleStreamingSource.TOPIC_KEY_DESERIALIZER, "org.apache.kafka.common.serialization.StringDeserializer");
    ss.setProp(KafkaSimpleStreamingSource.TOPIC_VALUE_DESERIALIZER, "org.apache.kafka.common.serialization.ByteArrayDeserializer");
    KafkaSimpleStreamingSource<String, byte[]> simpleSource = new KafkaSimpleStreamingSource<String, byte[]>();
    return simpleSource.getWorkunits(ss);
}