Examples with KafkaProducer org.apache.kafka.clients.producer.KafkaProducer used on opensource projects

Example 6 with KafkaProducer

use of org.apache.kafka.clients.producer.KafkaProducer in project storm by apache.

the class TridentKafkaState method prepare.

public void prepare(Properties options) {
    Validate.notNull(mapper, "mapper can not be null");
    Validate.notNull(topicSelector, "topicSelector can not be null");
    producer = new KafkaProducer(options);
}

Example 7 with KafkaProducer

use of org.apache.kafka.clients.producer.KafkaProducer in project beam by apache.

the class ResumeFromCheckpointStreamingTest method produce.

private static void produce(Map<String, Instant> messages) {
    Properties producerProps = new Properties();
    producerProps.putAll(EMBEDDED_KAFKA_CLUSTER.getProps());
    producerProps.put("request.required.acks", 1);
    producerProps.put("bootstrap.servers", EMBEDDED_KAFKA_CLUSTER.getBrokerList());
    Serializer<String> stringSerializer = new StringSerializer();
    Serializer<Instant> instantSerializer = new InstantSerializer();
    try (@SuppressWarnings("unchecked") KafkaProducer<String, Instant> kafkaProducer = new KafkaProducer(producerProps, stringSerializer, instantSerializer)) {
        for (Map.Entry<String, Instant> en : messages.entrySet()) {
            kafkaProducer.send(new ProducerRecord<>(TOPIC, en.getKey(), en.getValue()));
        }
        kafkaProducer.close();
    }
}

Example 8 with KafkaProducer

use of org.apache.kafka.clients.producer.KafkaProducer in project samza by apache.

the class TestKafkaSystemProducerJava method testInstantiateProducer.

@Test
public void testInstantiateProducer() {
    KafkaSystemProducer ksp = new KafkaSystemProducer("SysName", new ExponentialSleepStrategy(2.0, 200, 10000), new AbstractFunction0<Producer<byte[], byte[]>>() {

        @Override
        public Producer<byte[], byte[]> apply() {
            return new KafkaProducer<>(new HashMap<String, Object>());
        }
    }, new KafkaSystemProducerMetrics("SysName", new MetricsRegistryMap()), new AbstractFunction0<Object>() {

        @Override
        public Object apply() {
            return System.currentTimeMillis();
        }
    });
    long now = System.currentTimeMillis();
    assertTrue((Long) ksp.clock().apply() >= now);
}

Example 9 with KafkaProducer

use of org.apache.kafka.clients.producer.KafkaProducer in project samza by apache.

the class TestZkStreamProcessorBase method produceMessages.

/**
   * Produces the provided number of messages to the topic.
   */
protected void produceMessages(final int start, String topic, int numMessages) {
    KafkaProducer producer = getKafkaProducer();
    for (int i = start; i < numMessages + start; i++) {
        try {
            LOG.info("producing " + i);
            producer.send(new ProducerRecord(topic, i % 2, String.valueOf(i), String.valueOf(i).getBytes())).get();
        } catch (InterruptedException | ExecutionException e) {
            e.printStackTrace();
        }
    }
}

Example 10 with KafkaProducer

use of org.apache.kafka.clients.producer.KafkaProducer in project spark-dataflow by cloudera.

the class KafkaStreamingTest method init.

@BeforeClass
public static void init() throws IOException {
    EMBEDDED_ZOOKEEPER.startup();
    EMBEDDED_KAFKA_CLUSTER.startup();
    // write to Kafka
    Properties producerProps = new Properties();
    producerProps.putAll(EMBEDDED_KAFKA_CLUSTER.getProps());
    producerProps.put("request.required.acks", 1);
    producerProps.put("bootstrap.servers", EMBEDDED_KAFKA_CLUSTER.getBrokerList());
    Serializer<String> stringSerializer = new StringSerializer();
    try (@SuppressWarnings("unchecked") KafkaProducer<String, String> kafkaProducer = new KafkaProducer(producerProps, stringSerializer, stringSerializer)) {
        for (Map.Entry<String, String> en : KAFKA_MESSAGES.entrySet()) {
            kafkaProducer.send(new ProducerRecord<>(TOPIC, en.getKey(), en.getValue()));
        }
    }
}