Search in sources :

Example 1 with Windows

use of org.apache.samza.operators.windows.Windows in project samza by apache.

the class TestExecutionPlanner method createStreamGraphWithJoinAndWindow.

private StreamGraphImpl createStreamGraphWithJoinAndWindow() {
    StreamGraphImpl streamGraph = new StreamGraphImpl(runner, config);
    BiFunction msgBuilder = mock(BiFunction.class);
    MessageStream m1 = streamGraph.getInputStream("input1", msgBuilder).map(m -> m);
    MessageStream m2 = streamGraph.getInputStream("input2", msgBuilder).partitionBy(m -> "haha").filter(m -> true);
    MessageStream m3 = streamGraph.getInputStream("input3", msgBuilder).filter(m -> true).partitionBy(m -> "hehe").map(m -> m);
    Function mockFn = mock(Function.class);
    OutputStream<Object, Object, Object> output1 = streamGraph.getOutputStream("output1", mockFn, mockFn);
    OutputStream<Object, Object, Object> output2 = streamGraph.getOutputStream("output2", mockFn, mockFn);
    m1.map(m -> m).filter(m -> true).window(Windows.<Object, Object>keyedTumblingWindow(m -> m, Duration.ofMillis(8)));
    m2.map(m -> m).filter(m -> true).window(Windows.<Object, Object>keyedTumblingWindow(m -> m, Duration.ofMillis(16)));
    m1.join(m2, mock(JoinFunction.class), Duration.ofMillis(1600)).sendTo(output1);
    m3.join(m2, mock(JoinFunction.class), Duration.ofMillis(100)).sendTo(output2);
    m3.join(m2, mock(JoinFunction.class), Duration.ofMillis(252)).sendTo(output2);
    return streamGraph;
}
Also used : BiFunction(java.util.function.BiFunction) JobConfig(org.apache.samza.config.JobConfig) HashMap(java.util.HashMap) SystemStreamPartition(org.apache.samza.system.SystemStreamPartition) SystemStreamMetadata(org.apache.samza.system.SystemStreamMetadata) Function(java.util.function.Function) ArrayList(java.util.ArrayList) Duration(java.time.Duration) Map(java.util.Map) MapConfig(org.apache.samza.config.MapConfig) MessageStream(org.apache.samza.operators.MessageStream) Before(org.junit.Before) ApplicationRunner(org.apache.samza.runtime.ApplicationRunner) Windows(org.apache.samza.operators.windows.Windows) TaskConfig(org.apache.samza.config.TaskConfig) Collection(java.util.Collection) Partition(org.apache.samza.Partition) Set(java.util.Set) Assert.assertTrue(org.junit.Assert.assertTrue) StreamSpec(org.apache.samza.system.StreamSpec) Test(org.junit.Test) Mockito.when(org.mockito.Mockito.when) JoinFunction(org.apache.samza.operators.functions.JoinFunction) StreamGraphImpl(org.apache.samza.operators.StreamGraphImpl) List(java.util.List) Assert.assertFalse(org.junit.Assert.assertFalse) SystemAdmin(org.apache.samza.system.SystemAdmin) Config(org.apache.samza.config.Config) OutputStream(org.apache.samza.operators.OutputStream) Collections(java.util.Collections) Assert.assertEquals(org.junit.Assert.assertEquals) Mockito.mock(org.mockito.Mockito.mock) BiFunction(java.util.function.BiFunction) Function(java.util.function.Function) JoinFunction(org.apache.samza.operators.functions.JoinFunction) BiFunction(java.util.function.BiFunction) MessageStream(org.apache.samza.operators.MessageStream) StreamGraphImpl(org.apache.samza.operators.StreamGraphImpl)

Example 2 with Windows

use of org.apache.samza.operators.windows.Windows in project samza by apache.

the class AppWithGlobalConfigExample method describe.

@Override
public void describe(StreamApplicationDescriptor appDescriptor) {
    KafkaSystemDescriptor trackingSystem = new KafkaSystemDescriptor("tracking");
    KafkaInputDescriptor<PageViewEvent> inputStreamDescriptor = trackingSystem.getInputDescriptor("pageViewEvent", new JsonSerdeV2<>(PageViewEvent.class));
    KafkaOutputDescriptor<KV<String, PageViewCount>> outputStreamDescriptor = trackingSystem.getOutputDescriptor("pageViewEventPerMember", KVSerde.of(new StringSerde(), new JsonSerdeV2<>(PageViewCount.class)));
    appDescriptor.getInputStream(inputStreamDescriptor).window(Windows.<PageViewEvent, String, Integer>keyedTumblingWindow(PageViewEvent::getMemberId, Duration.ofSeconds(10), () -> 0, (m, c) -> c + 1, null, null).setEarlyTrigger(Triggers.repeat(Triggers.count(5))).setAccumulationMode(AccumulationMode.DISCARDING), "window1").map(m -> KV.of(m.getKey().getKey(), buildPageViewCount(m))).sendTo(appDescriptor.getOutputStream(outputStreamDescriptor));
    appDescriptor.withMetricsReporterFactories(new HashMap<>());
}
Also used : ApplicationRunner(org.apache.samza.runtime.ApplicationRunner) Windows(org.apache.samza.operators.windows.Windows) KafkaInputDescriptor(org.apache.samza.system.kafka.descriptors.KafkaInputDescriptor) CommandLine(org.apache.samza.util.CommandLine) KafkaSystemDescriptor(org.apache.samza.system.kafka.descriptors.KafkaSystemDescriptor) HashMap(java.util.HashMap) PageViewEvent(org.apache.samza.example.models.PageViewEvent) Triggers(org.apache.samza.operators.triggers.Triggers) WindowPane(org.apache.samza.operators.windows.WindowPane) StringSerde(org.apache.samza.serializers.StringSerde) PageViewCount(org.apache.samza.example.models.PageViewCount) KafkaOutputDescriptor(org.apache.samza.system.kafka.descriptors.KafkaOutputDescriptor) StreamApplicationDescriptor(org.apache.samza.application.descriptors.StreamApplicationDescriptor) AccumulationMode(org.apache.samza.operators.windows.AccumulationMode) Duration(java.time.Duration) Config(org.apache.samza.config.Config) ApplicationRunners(org.apache.samza.runtime.ApplicationRunners) JsonSerdeV2(org.apache.samza.serializers.JsonSerdeV2) KVSerde(org.apache.samza.serializers.KVSerde) StreamApplication(org.apache.samza.application.StreamApplication) KV(org.apache.samza.operators.KV) StringSerde(org.apache.samza.serializers.StringSerde) PageViewEvent(org.apache.samza.example.models.PageViewEvent) KV(org.apache.samza.operators.KV) KafkaSystemDescriptor(org.apache.samza.system.kafka.descriptors.KafkaSystemDescriptor) JsonSerdeV2(org.apache.samza.serializers.JsonSerdeV2)

Aggregations

Duration (java.time.Duration)2 HashMap (java.util.HashMap)2 Config (org.apache.samza.config.Config)2 Windows (org.apache.samza.operators.windows.Windows)2 ApplicationRunner (org.apache.samza.runtime.ApplicationRunner)2 ArrayList (java.util.ArrayList)1 Collection (java.util.Collection)1 Collections (java.util.Collections)1 List (java.util.List)1 Map (java.util.Map)1 Set (java.util.Set)1 BiFunction (java.util.function.BiFunction)1 Function (java.util.function.Function)1 Partition (org.apache.samza.Partition)1 StreamApplication (org.apache.samza.application.StreamApplication)1 StreamApplicationDescriptor (org.apache.samza.application.descriptors.StreamApplicationDescriptor)1 JobConfig (org.apache.samza.config.JobConfig)1 MapConfig (org.apache.samza.config.MapConfig)1 TaskConfig (org.apache.samza.config.TaskConfig)1 PageViewCount (org.apache.samza.example.models.PageViewCount)1