Examples with Watermark com.hazelcast.jet.core.Watermark used on opensource projects

Example 11 with Watermark

use of com.hazelcast.jet.core.Watermark in project hazelcast by hazelcast.

the class StreamKafkaPTest method when_eventsInSinglePartition_then_watermarkAfterIdleTime.

@Test
public void when_eventsInSinglePartition_then_watermarkAfterIdleTime() throws Exception {
    // When
    StreamKafkaP processor = createProcessor(properties(), 2, r -> entry(r.key(), r.value()), 10_000);
    TestOutbox outbox = new TestOutbox(new int[] { 10 }, 10);
    processor.init(outbox, new TestProcessorContext());
    kafkaTestSupport.produce(topic1Name, 10, "foo");
    // Then
    assertEquals(entry(10, "foo"), consumeEventually(processor, outbox));
    long time1 = System.nanoTime();
    assertEquals(new Watermark(10 - LAG), consumeEventually(processor, outbox));
    long time2 = System.nanoTime();
    long elapsedMs = NANOSECONDS.toMillis(time2 - time1);
    assertBetween("elapsed time", elapsedMs, 3000, 30_000);
}

Example 12 with Watermark

use of com.hazelcast.jet.core.Watermark in project hazelcast by hazelcast.

the class AsyncTransformUsingServicePTest method test_futuresCompletedInSeparateThread.

@Test
public void test_futuresCompletedInSeparateThread() {
    TestSupport.verifyProcessor(getSupplier((ctx, item) -> {
        CompletableFuture<Traverser<String>> f = new CompletableFuture<>();
        spawn(() -> f.complete(traverseItems(item + "-1", item + "-2")));
        return f;
    })).hazelcastInstance(instance()).input(asList("a", "b", new Watermark(10))).outputChecker((expected, actual) -> actual.equals(asList("a-1", "a-2", "b-1", "b-2", wm(10))) || !ordered && actual.equals(asList("b-1", "b-2", "a-1", "a-2", wm(10)))).disableProgressAssertion().expectOutput(singletonList("<see code>"));
}

Example 13 with Watermark

use of com.hazelcast.jet.core.Watermark in project hazelcast by hazelcast.

the class AsyncTransformUsingServiceBatchedPTest method test_futuresCompletedInSeparateThread.

@Test
public void test_futuresCompletedInSeparateThread() {
    TestSupport.verifyProcessor(getSupplier((ctx, items) -> {
        CompletableFuture<Traverser<String>> f = new CompletableFuture<>();
        spawn(() -> f.complete(traverseIterable(items).flatMap(item -> traverseItems(item + "-1", item + "-2"))));
        return f;
    })).hazelcastInstance(instance()).input(asList("a", "b", new Watermark(10))).outputChecker((expected, actual) -> actual.equals(asList("a-1", "a-2", "b-1", "b-2", wm(10)))).disableProgressAssertion().expectOutput(singletonList("<see code>"));
}

Example 14 with Watermark

use of com.hazelcast.jet.core.Watermark in project hazelcast by hazelcast.

the class SessionWindowPTest method when_batchProcessing_then_flushEverything.

@Test
public void when_batchProcessing_then_flushEverything() {
    // Given
    List<Object> inbox = new ArrayList<>(eventsWithKey("a"));
    // This watermark will cause the first session to be emitted, but not the second.
    // The second session will be emitted in complete()
    inbox.add(new Watermark(25));
    verifyProcessor(supplier).input(inbox).expectOutput(asList(new KeyedWindowResult<>(1, 22, "a", 3L, false), new Watermark(25), new KeyedWindowResult<>(30, 50, "a", 3L, false)));
}

Example 15 with Watermark

use of com.hazelcast.jet.core.Watermark in project hazelcast by hazelcast.

the class SessionWindowPTest method assertCorrectness.

private void assertCorrectness(List<Object> events) {
    @SuppressWarnings("unchecked") List<Object> expectedOutput = events.stream().map(e -> ((Entry<String, Long>) e).getKey()).flatMap(SessionWindowPTest::expectedSessions).distinct().collect(toList());
    events.add(new Watermark(100));
    expectedOutput.add(new Watermark(100));
    try {
        verifyProcessor(supplier).outputChecker(SAME_ITEMS_ANY_ORDER).input(events).expectOutput(expectedOutput);
    } catch (AssertionError e) {
        System.err.println("Tested with events: " + events);
        throw e;
    }
}