Search in sources :

Example 76 with Topology

use of org.apache.kafka.streams.Topology in project brave by openzipkin.

the class ITKafkaStreamsTracing method should_create_spans_from_stream_with_tracing_flatMap.

@Test
public void should_create_spans_from_stream_with_tracing_flatMap() {
    String inputTopic = testName.getMethodName() + "-input";
    String outputTopic = testName.getMethodName() + "-output";
    StreamsBuilder builder = new StreamsBuilder();
    builder.stream(inputTopic, Consumed.with(Serdes.String(), Serdes.String())).flatTransform(kafkaStreamsTracing.flatMap("flat-map-1", (key, value) -> {
        try {
            Thread.sleep(100L);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        return Arrays.asList(KeyValue.pair(key, value + "-1"), KeyValue.pair(key, value + "-2"));
    })).to(outputTopic, Produced.with(Serdes.String(), Serdes.String()));
    Topology topology = builder.build();
    KafkaStreams streams = buildKafkaStreams(topology);
    send(new ProducerRecord<>(inputTopic, TEST_KEY, TEST_VALUE));
    waitForStreamToRun(streams);
    MutableSpan spanInput = testSpanHandler.takeRemoteSpan(CONSUMER);
    assertThat(spanInput.tags()).containsEntry("kafka.topic", inputTopic);
    MutableSpan spanProcessor = testSpanHandler.takeLocalSpan();
    assertChildOf(spanProcessor, spanInput);
    assertThat(spanProcessor.tags()).containsOnlyKeys("kafka.streams.application.id", "kafka.streams.task.id");
    for (int i = 0; i < 2; i++) {
        MutableSpan spanOutput = testSpanHandler.takeRemoteSpan(PRODUCER);
        assertThat(spanOutput.tags()).containsEntry("kafka.topic", outputTopic);
        assertChildOf(spanOutput, spanProcessor);
    }
    streams.close();
    streams.cleanUp();
}
Also used : StreamsBuilder(org.apache.kafka.streams.StreamsBuilder) KafkaStreams(org.apache.kafka.streams.KafkaStreams) MutableSpan(brave.handler.MutableSpan) Topology(org.apache.kafka.streams.Topology) Test(org.junit.Test)

Example 77 with Topology

use of org.apache.kafka.streams.Topology in project brave by openzipkin.

the class ITKafkaStreamsTracing method should_create_spans_from_stream_without_tracing_with_tracing_valueTransformer.

@Test
public void should_create_spans_from_stream_without_tracing_with_tracing_valueTransformer() {
    ValueTransformerSupplier<String, String> transformerSupplier = kafkaStreamsTracing.valueTransformer("transformer-1", () -> new ValueTransformer<String, String>() {

        ProcessorContext context;

        @Override
        public void init(ProcessorContext context) {
            this.context = context;
        }

        @Override
        public String transform(String value) {
            try {
                Thread.sleep(100L);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            return value;
        }

        @Override
        public void close() {
        }
    });
    String inputTopic = testName.getMethodName() + "-input";
    String outputTopic = testName.getMethodName() + "-output";
    StreamsBuilder builder = new StreamsBuilder();
    builder.stream(inputTopic, Consumed.with(Serdes.String(), Serdes.String())).transformValues(transformerSupplier).to(outputTopic, Produced.with(Serdes.String(), Serdes.String()));
    Topology topology = builder.build();
    KafkaStreams streams = buildKafkaStreamsWithoutTracing(topology);
    send(new ProducerRecord<>(inputTopic, TEST_KEY, TEST_VALUE));
    waitForStreamToRun(streams);
    assertThat(testSpanHandler.takeLocalSpan().tags()).containsOnlyKeys("kafka.streams.application.id", "kafka.streams.task.id");
    streams.close();
    streams.cleanUp();
}
Also used : StreamsBuilder(org.apache.kafka.streams.StreamsBuilder) KafkaStreams(org.apache.kafka.streams.KafkaStreams) Topology(org.apache.kafka.streams.Topology) ProcessorContext(org.apache.kafka.streams.processor.ProcessorContext) Test(org.junit.Test)

Example 78 with Topology

use of org.apache.kafka.streams.Topology in project brave by openzipkin.

the class ITKafkaStreamsTracing method should_create_spans_from_stream_without_tracing_with_tracing_transformer.

@Test
public void should_create_spans_from_stream_without_tracing_with_tracing_transformer() {
    TransformerSupplier<String, String, KeyValue<String, String>> transformerSupplier = kafkaStreamsTracing.transformer("transformer-1", () -> new Transformer<String, String, KeyValue<String, String>>() {

        ProcessorContext context;

        @Override
        public void init(ProcessorContext context) {
            this.context = context;
        }

        @Override
        public KeyValue<String, String> transform(String key, String value) {
            try {
                Thread.sleep(100L);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            return KeyValue.pair(key, value);
        }

        @Override
        public void close() {
        }
    });
    String inputTopic = testName.getMethodName() + "-input";
    String outputTopic = testName.getMethodName() + "-output";
    StreamsBuilder builder = new StreamsBuilder();
    builder.stream(inputTopic, Consumed.with(Serdes.String(), Serdes.String())).transform(transformerSupplier).to(outputTopic, Produced.with(Serdes.String(), Serdes.String()));
    Topology topology = builder.build();
    KafkaStreams streams = buildKafkaStreamsWithoutTracing(topology);
    send(new ProducerRecord<>(inputTopic, TEST_KEY, TEST_VALUE));
    waitForStreamToRun(streams);
    assertThat(testSpanHandler.takeLocalSpan().tags()).containsOnlyKeys("kafka.streams.application.id", "kafka.streams.task.id");
    streams.close();
    streams.cleanUp();
}
Also used : StreamsBuilder(org.apache.kafka.streams.StreamsBuilder) KafkaStreams(org.apache.kafka.streams.KafkaStreams) KeyValue(org.apache.kafka.streams.KeyValue) Topology(org.apache.kafka.streams.Topology) ProcessorContext(org.apache.kafka.streams.processor.ProcessorContext) Test(org.junit.Test)

Example 79 with Topology

use of org.apache.kafka.streams.Topology in project brave by openzipkin.

the class ITKafkaStreamsTracing method should_create_spans_from_stream_without_tracing_with_tracing_valueTransformerWithKey.

@Test
public void should_create_spans_from_stream_without_tracing_with_tracing_valueTransformerWithKey() {
    ValueTransformerWithKeySupplier<String, String, String> transformerSupplier = kafkaStreamsTracing.valueTransformerWithKey("transformer-1", () -> new ValueTransformerWithKey<String, String, String>() {

        ProcessorContext context;

        @Override
        public void init(ProcessorContext context) {
            this.context = context;
        }

        @Override
        public String transform(String key, String value) {
            try {
                Thread.sleep(100L);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            return value;
        }

        @Override
        public void close() {
        }
    });
    String inputTopic = testName.getMethodName() + "-input";
    String outputTopic = testName.getMethodName() + "-output";
    StreamsBuilder builder = new StreamsBuilder();
    builder.stream(inputTopic, Consumed.with(Serdes.String(), Serdes.String())).transformValues(transformerSupplier).to(outputTopic, Produced.with(Serdes.String(), Serdes.String()));
    Topology topology = builder.build();
    KafkaStreams streams = buildKafkaStreamsWithoutTracing(topology);
    send(new ProducerRecord<>(inputTopic, TEST_KEY, TEST_VALUE));
    waitForStreamToRun(streams);
    assertThat(testSpanHandler.takeLocalSpan().tags()).containsOnlyKeys("kafka.streams.application.id", "kafka.streams.task.id");
    streams.close();
    streams.cleanUp();
}
Also used : StreamsBuilder(org.apache.kafka.streams.StreamsBuilder) KafkaStreams(org.apache.kafka.streams.KafkaStreams) Topology(org.apache.kafka.streams.Topology) ProcessorContext(org.apache.kafka.streams.processor.ProcessorContext) Test(org.junit.Test)

Example 80 with Topology

use of org.apache.kafka.streams.Topology in project brave by openzipkin.

the class ITKafkaStreamsTracing method should_create_spans_from_stream_input_and_output_topics.

@Test
public void should_create_spans_from_stream_input_and_output_topics() {
    String inputTopic = testName.getMethodName() + "-input";
    String outputTopic = testName.getMethodName() + "-output";
    StreamsBuilder builder = new StreamsBuilder();
    builder.stream(inputTopic).to(outputTopic);
    Topology topology = builder.build();
    KafkaStreams streams = buildKafkaStreams(topology);
    send(new ProducerRecord<>(inputTopic, TEST_KEY, TEST_VALUE));
    Consumer<String, String> consumer = createTracingConsumer(outputTopic);
    waitForStreamToRun(streams);
    MutableSpan spanInput = testSpanHandler.takeRemoteSpan(CONSUMER);
    assertThat(spanInput.tags()).containsEntry("kafka.topic", inputTopic);
    MutableSpan spanOutput = testSpanHandler.takeRemoteSpan(PRODUCER);
    assertThat(spanOutput.tags()).containsEntry("kafka.topic", outputTopic);
    assertChildOf(spanOutput, spanInput);
    streams.close();
    streams.cleanUp();
    consumer.close();
}
Also used : StreamsBuilder(org.apache.kafka.streams.StreamsBuilder) KafkaStreams(org.apache.kafka.streams.KafkaStreams) MutableSpan(brave.handler.MutableSpan) Topology(org.apache.kafka.streams.Topology) Test(org.junit.Test)

Aggregations

Topology (org.apache.kafka.streams.Topology)127 Test (org.junit.Test)106 StreamsBuilder (org.apache.kafka.streams.StreamsBuilder)93 KafkaStreams (org.apache.kafka.streams.KafkaStreams)53 TopologyTestDriver (org.apache.kafka.streams.TopologyTestDriver)53 Properties (java.util.Properties)47 StringSerializer (org.apache.kafka.common.serialization.StringSerializer)46 KeyValue (org.apache.kafka.streams.KeyValue)40 Serdes (org.apache.kafka.common.serialization.Serdes)39 StreamsConfig (org.apache.kafka.streams.StreamsConfig)33 List (java.util.List)29 MutableSpan (brave.handler.MutableSpan)28 Consumed (org.apache.kafka.streams.kstream.Consumed)28 Produced (org.apache.kafka.streams.kstream.Produced)26 Arrays (java.util.Arrays)25 StringDeserializer (org.apache.kafka.common.serialization.StringDeserializer)25 ArrayList (java.util.ArrayList)23 ProcessorContext (org.apache.kafka.streams.processor.ProcessorContext)23 Duration (java.time.Duration)22 KStream (org.apache.kafka.streams.kstream.KStream)22