Examples with KafkaStreams org.apache.kafka.streams.KafkaStreams used on opensource projects

Example 36 with KafkaStreams

use of org.apache.kafka.streams.KafkaStreams in project kafka by apache.

the class SimpleBenchmark method kStreamKStreamJoin.

/**
     * Measure the performance of a KStream-KStream left join. The setup is such that each
     * KStream record joins to exactly one element in the other KStream
     */
public void kStreamKStreamJoin(String kStreamTopic1, String kStreamTopic2) throws Exception {
    if (maybeSetupPhase(kStreamTopic1, "simple-benchmark-produce-kstream-topic1", false)) {
        maybeSetupPhase(kStreamTopic2, "simple-benchmark-produce-kstream-topic2", false);
        return;
    }
    CountDownLatch latch = new CountDownLatch(1);
    // setup join
    Properties props = setStreamProperties("simple-benchmark-kstream-kstream-join");
    final KafkaStreams streams = createKafkaStreamsKStreamKStreamJoin(props, kStreamTopic1, kStreamTopic2, latch);
    // run benchmark
    runGenericBenchmark(streams, "Streams KStreamKStream LeftJoin Performance [records/latency/rec-sec/MB-sec  joined]: ", latch);
}

Example 37 with KafkaStreams

use of org.apache.kafka.streams.KafkaStreams in project kafka by apache.

the class SimpleBenchmark method createKafkaStreamsWithExceptionHandler.

private KafkaStreams createKafkaStreamsWithExceptionHandler(final KStreamBuilder builder, final Properties props) {
    final KafkaStreams streamsClient = new KafkaStreams(builder, props);
    streamsClient.setUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {

        @Override
        public void uncaughtException(Thread t, Throwable e) {
            System.out.println("FATAL: An unexpected exception is encountered on thread " + t + ": " + e);
            streamsClient.close(30, TimeUnit.SECONDS);
        }
    });
    return streamsClient;
}

Example 38 with KafkaStreams

use of org.apache.kafka.streams.KafkaStreams in project kafka by apache.

the class SimpleBenchmark method count.

/**
     * Measure the performance of a simple aggregate like count.
     * Counts the occurrence of numbers (note that normally people count words, this
     * example counts numbers)
     * @param countTopic Topic where numbers are stored
     * @throws Exception
     */
public void count(String countTopic) throws Exception {
    if (maybeSetupPhase(countTopic, "simple-benchmark-produce-count", false)) {
        return;
    }
    CountDownLatch latch = new CountDownLatch(1);
    Properties props = setStreamProperties("simple-benchmark-count");
    final KafkaStreams streams = createCountStreams(props, countTopic, latch);
    runGenericBenchmark(streams, "Streams Count Performance [records/latency/rec-sec/MB-sec counted]: ", latch);
}

Example 39 with KafkaStreams

use of org.apache.kafka.streams.KafkaStreams in project kafka by apache.

the class SimpleBenchmark method processStreamWithCachedStateStore.

public void processStreamWithCachedStateStore(String topic) throws Exception {
    if (maybeSetupPhase(topic, "simple-benchmark-process-stream-with-cached-state-store-load", true)) {
        return;
    }
    CountDownLatch latch = new CountDownLatch(1);
    final KafkaStreams streams = createKafkaStreamsWithStateStore(topic, latch, true);
    long latency = startStreamsThread(streams, latch);
    printResults("Streams Performance [records/latency/rec-sec/MB-sec source+cache+store]: ", latency);
}

Example 40 with KafkaStreams

use of org.apache.kafka.streams.KafkaStreams in project kafka by apache.

the class SimpleBenchmark method createCountStreams.

private KafkaStreams createCountStreams(Properties streamConfig, String topic, final CountDownLatch latch) {
    final KStreamBuilder builder = new KStreamBuilder();
    final KStream<Integer, byte[]> input = builder.stream(topic);
    input.groupByKey().count("tmpStoreName").foreach(new CountDownAction(latch));
    return new KafkaStreams(builder, streamConfig);
}