Search in sources :

Example 11 with KafkaInputDescriptor

use of org.apache.samza.system.kafka.descriptors.KafkaInputDescriptor in project samza by apache.

the class AsyncApplicationExample method describe.

@Override
public void describe(StreamApplicationDescriptor appDescriptor) {
    KafkaSystemDescriptor trackingSystem = new KafkaSystemDescriptor("tracking");
    KafkaInputDescriptor<AdClickEvent> inputStreamDescriptor = trackingSystem.getInputDescriptor("adClickEvent", new JsonSerdeV2<>(AdClickEvent.class));
    KafkaOutputDescriptor<KV<String, EnrichedAdClickEvent>> outputStreamDescriptor = trackingSystem.getOutputDescriptor("enrichedAdClickEvent", KVSerde.of(new StringSerde(), new JsonSerdeV2<>(EnrichedAdClickEvent.class)));
    MessageStream<AdClickEvent> adClickEventStream = appDescriptor.getInputStream(inputStreamDescriptor);
    OutputStream<KV<String, EnrichedAdClickEvent>> enrichedAdClickStream = appDescriptor.getOutputStream(outputStreamDescriptor);
    adClickEventStream.flatMapAsync(AsyncApplicationExample::enrichAdClickEvent).map(enrichedAdClickEvent -> KV.of(enrichedAdClickEvent.getCountry(), enrichedAdClickEvent)).sendTo(enrichedAdClickStream);
}
Also used : CommandLine(org.apache.samza.util.CommandLine) Random(java.util.Random) CompletableFuture(java.util.concurrent.CompletableFuture) StringSerde(org.apache.samza.serializers.StringSerde) AdClickEvent(org.apache.samza.example.models.AdClickEvent) KafkaOutputDescriptor(org.apache.samza.system.kafka.descriptors.KafkaOutputDescriptor) ImmutableList(com.google.common.collect.ImmutableList) EnrichedAdClickEvent(org.apache.samza.example.models.EnrichedAdClickEvent) ApplicationRunners(org.apache.samza.runtime.ApplicationRunners) KV(org.apache.samza.operators.KV) MessageStream(org.apache.samza.operators.MessageStream) ApplicationRunner(org.apache.samza.runtime.ApplicationRunner) KafkaInputDescriptor(org.apache.samza.system.kafka.descriptors.KafkaInputDescriptor) Collection(java.util.Collection) Member(org.apache.samza.example.models.Member) KafkaSystemDescriptor(org.apache.samza.system.kafka.descriptors.KafkaSystemDescriptor) List(java.util.List) CompletionStage(java.util.concurrent.CompletionStage) StreamApplicationDescriptor(org.apache.samza.application.descriptors.StreamApplicationDescriptor) Config(org.apache.samza.config.Config) JsonSerdeV2(org.apache.samza.serializers.JsonSerdeV2) KVSerde(org.apache.samza.serializers.KVSerde) StreamApplication(org.apache.samza.application.StreamApplication) Collections(java.util.Collections) OutputStream(org.apache.samza.operators.OutputStream) AdClickEvent(org.apache.samza.example.models.AdClickEvent) EnrichedAdClickEvent(org.apache.samza.example.models.EnrichedAdClickEvent) StringSerde(org.apache.samza.serializers.StringSerde) KV(org.apache.samza.operators.KV) KafkaSystemDescriptor(org.apache.samza.system.kafka.descriptors.KafkaSystemDescriptor) JsonSerdeV2(org.apache.samza.serializers.JsonSerdeV2)

Aggregations

StreamApplication (org.apache.samza.application.StreamApplication)11 StreamApplicationDescriptor (org.apache.samza.application.descriptors.StreamApplicationDescriptor)11 Config (org.apache.samza.config.Config)11 JsonSerdeV2 (org.apache.samza.serializers.JsonSerdeV2)11 KafkaInputDescriptor (org.apache.samza.system.kafka.descriptors.KafkaInputDescriptor)11 KafkaSystemDescriptor (org.apache.samza.system.kafka.descriptors.KafkaSystemDescriptor)11 KV (org.apache.samza.operators.KV)10 ApplicationRunner (org.apache.samza.runtime.ApplicationRunner)10 ApplicationRunners (org.apache.samza.runtime.ApplicationRunners)10 KVSerde (org.apache.samza.serializers.KVSerde)10 StringSerde (org.apache.samza.serializers.StringSerde)10 KafkaOutputDescriptor (org.apache.samza.system.kafka.descriptors.KafkaOutputDescriptor)10 CommandLine (org.apache.samza.util.CommandLine)10 MessageStream (org.apache.samza.operators.MessageStream)9 Duration (java.time.Duration)8 OutputStream (org.apache.samza.operators.OutputStream)7 Windows (org.apache.samza.operators.windows.Windows)7 PageViewEvent (org.apache.samza.example.models.PageViewEvent)6 WindowPane (org.apache.samza.operators.windows.WindowPane)4 List (java.util.List)3