Search in sources :

Example 26 with WindowBytesStoreSupplier

use of org.apache.kafka.streams.state.WindowBytesStoreSupplier in project kafka by apache.

the class KStreamKStreamJoinTest method shouldThrowExceptionOtherStoreSupplierRetentionDoNotMatchWindowsSizeAndGrace.

@Test
public void shouldThrowExceptionOtherStoreSupplierRetentionDoNotMatchWindowsSizeAndGrace() {
    // Case where retention size of otherJoinStore doesn't match JoinWindows
    final WindowBytesStoreSupplier thisStoreSupplier = buildWindowBytesStoreSupplier("in-memory-join-store", 150L, 100L, true);
    final WindowBytesStoreSupplier otherStoreSupplier = buildWindowBytesStoreSupplier("in-memory-join-store-other", 500L, 100L, true);
    buildStreamsJoinThatShouldThrow(streamJoined.withThisStoreSupplier(thisStoreSupplier).withOtherStoreSupplier(otherStoreSupplier), joinWindows, errorMessagePrefix);
}
Also used : WindowBytesStoreSupplier(org.apache.kafka.streams.state.WindowBytesStoreSupplier) Test(org.junit.Test)

Example 27 with WindowBytesStoreSupplier

use of org.apache.kafka.streams.state.WindowBytesStoreSupplier in project kafka by apache.

the class KStreamKStreamJoinTest method shouldThrowExceptionThisStoreSupplierWindowSizeDoesNotMatchJoinWindowsWindowSize.

@Test
public void shouldThrowExceptionThisStoreSupplierWindowSizeDoesNotMatchJoinWindowsWindowSize() {
    // Case where window size of thisJoinStore doesn't match JoinWindows
    final WindowBytesStoreSupplier thisStoreSupplier = buildWindowBytesStoreSupplier("in-memory-join-store", 150L, 150L, true);
    final WindowBytesStoreSupplier otherStoreSupplier = buildWindowBytesStoreSupplier("in-memory-join-store-other", 150L, 100L, true);
    buildStreamsJoinThatShouldThrow(streamJoined.withThisStoreSupplier(thisStoreSupplier).withOtherStoreSupplier(otherStoreSupplier), joinWindows, errorMessagePrefix);
}
Also used : WindowBytesStoreSupplier(org.apache.kafka.streams.state.WindowBytesStoreSupplier) Test(org.junit.Test)

Example 28 with WindowBytesStoreSupplier

use of org.apache.kafka.streams.state.WindowBytesStoreSupplier in project kafka by apache.

the class KStreamKStreamJoinTest method shouldThrowExceptionWhenOtherJoinStoreSetsRetainDuplicatesFalse.

@Test
public void shouldThrowExceptionWhenOtherJoinStoreSetsRetainDuplicatesFalse() {
    // Case where otherJoinStore retain duplicates false
    final WindowBytesStoreSupplier thisStoreSupplier = buildWindowBytesStoreSupplier("in-memory-join-store", 150L, 100L, true);
    final WindowBytesStoreSupplier otherStoreSupplier = buildWindowBytesStoreSupplier("in-memory-join-store-other", 150L, 100L, false);
    buildStreamsJoinThatShouldThrow(streamJoined.withThisStoreSupplier(thisStoreSupplier).withOtherStoreSupplier(otherStoreSupplier), joinWindows, "The StoreSupplier must set retainDuplicates=true, found retainDuplicates=false");
}
Also used : WindowBytesStoreSupplier(org.apache.kafka.streams.state.WindowBytesStoreSupplier) Test(org.junit.Test)

Example 29 with WindowBytesStoreSupplier

use of org.apache.kafka.streams.state.WindowBytesStoreSupplier in project kafka by apache.

the class KStreamKStreamJoinTest method shouldJoinWithCustomStoreSuppliers.

@Test
public void shouldJoinWithCustomStoreSuppliers() {
    final JoinWindows joinWindows = JoinWindows.ofTimeDifferenceWithNoGrace(ofMillis(100L));
    final WindowBytesStoreSupplier thisStoreSupplier = Stores.inMemoryWindowStore("in-memory-join-store", Duration.ofMillis(joinWindows.size() + joinWindows.gracePeriodMs()), Duration.ofMillis(joinWindows.size()), true);
    final WindowBytesStoreSupplier otherStoreSupplier = Stores.inMemoryWindowStore("in-memory-join-store-other", Duration.ofMillis(joinWindows.size() + joinWindows.gracePeriodMs()), Duration.ofMillis(joinWindows.size()), true);
    final StreamJoined<String, Integer, Integer> streamJoined = StreamJoined.with(Serdes.String(), Serdes.Integer(), Serdes.Integer());
    // Case with 2 custom store suppliers
    runJoin(streamJoined.withThisStoreSupplier(thisStoreSupplier).withOtherStoreSupplier(otherStoreSupplier), joinWindows);
    // Case with this stream store supplier
    runJoin(streamJoined.withThisStoreSupplier(thisStoreSupplier), joinWindows);
    // Case with other stream store supplier
    runJoin(streamJoined.withOtherStoreSupplier(otherStoreSupplier), joinWindows);
}
Also used : WindowBytesStoreSupplier(org.apache.kafka.streams.state.WindowBytesStoreSupplier) JoinWindows(org.apache.kafka.streams.kstream.JoinWindows) Test(org.junit.Test)

Example 30 with WindowBytesStoreSupplier

use of org.apache.kafka.streams.state.WindowBytesStoreSupplier in project kafka by apache.

the class TopologyTest method streamStreamLeftJoinTopologyWithCustomStoresSuppliers.

@Test
public void streamStreamLeftJoinTopologyWithCustomStoresSuppliers() {
    final StreamsBuilder builder = new StreamsBuilder();
    final KStream<Integer, String> stream1;
    final KStream<Integer, String> stream2;
    stream1 = builder.stream("input-topic1");
    stream2 = builder.stream("input-topic2");
    final JoinWindows joinWindows = JoinWindows.ofTimeDifferenceWithNoGrace(ofMillis(100));
    final WindowBytesStoreSupplier thisStoreSupplier = Stores.inMemoryWindowStore("in-memory-join-store", Duration.ofMillis(joinWindows.size() + joinWindows.gracePeriodMs()), Duration.ofMillis(joinWindows.size()), true);
    final WindowBytesStoreSupplier otherStoreSupplier = Stores.inMemoryWindowStore("in-memory-join-store-other", Duration.ofMillis(joinWindows.size() + joinWindows.gracePeriodMs()), Duration.ofMillis(joinWindows.size()), true);
    stream1.leftJoin(stream2, MockValueJoiner.TOSTRING_JOINER, joinWindows, StreamJoined.with(Serdes.Integer(), Serdes.String(), Serdes.String()).withThisStoreSupplier(thisStoreSupplier).withOtherStoreSupplier(otherStoreSupplier));
    final TopologyDescription describe = builder.build().describe();
    assertEquals("Topologies:\n" + "   Sub-topology: 0\n" + "    Source: KSTREAM-SOURCE-0000000000 (topics: [input-topic1])\n" + "      --> KSTREAM-WINDOWED-0000000002\n" + "    Source: KSTREAM-SOURCE-0000000001 (topics: [input-topic2])\n" + "      --> KSTREAM-WINDOWED-0000000003\n" + "    Processor: KSTREAM-WINDOWED-0000000002 (stores: [in-memory-join-store])\n" + "      --> KSTREAM-JOINTHIS-0000000004\n" + "      <-- KSTREAM-SOURCE-0000000000\n" + "    Processor: KSTREAM-WINDOWED-0000000003 (stores: [in-memory-join-store-other])\n" + "      --> KSTREAM-OUTEROTHER-0000000005\n" + "      <-- KSTREAM-SOURCE-0000000001\n" + "    Processor: KSTREAM-JOINTHIS-0000000004 (stores: [in-memory-join-store-other, in-memory-join-store-left-shared-join-store])\n" + "      --> KSTREAM-MERGE-0000000006\n" + "      <-- KSTREAM-WINDOWED-0000000002\n" + "    Processor: KSTREAM-OUTEROTHER-0000000005 (stores: [in-memory-join-store, in-memory-join-store-left-shared-join-store])\n" + "      --> KSTREAM-MERGE-0000000006\n" + "      <-- KSTREAM-WINDOWED-0000000003\n" + "    Processor: KSTREAM-MERGE-0000000006 (stores: [])\n" + "      --> none\n" + "      <-- KSTREAM-JOINTHIS-0000000004, KSTREAM-OUTEROTHER-0000000005\n\n", describe.toString());
}
Also used : WindowBytesStoreSupplier(org.apache.kafka.streams.state.WindowBytesStoreSupplier) JoinWindows(org.apache.kafka.streams.kstream.JoinWindows) Test(org.junit.Test)

Aggregations

WindowBytesStoreSupplier (org.apache.kafka.streams.state.WindowBytesStoreSupplier)31 Test (org.junit.Test)21 StreamsBuilder (org.apache.kafka.streams.StreamsBuilder)9 StringSerializer (org.apache.kafka.common.serialization.StringSerializer)8 TopologyTestDriver (org.apache.kafka.streams.TopologyTestDriver)8 Windowed (org.apache.kafka.streams.kstream.Windowed)8 JoinWindows (org.apache.kafka.streams.kstream.JoinWindows)7 InMemoryWindowBytesStoreSupplier (org.apache.kafka.streams.state.internals.InMemoryWindowBytesStoreSupplier)7 MockApiProcessorSupplier (org.apache.kafka.test.MockApiProcessorSupplier)6 TimestampedWindowStore (org.apache.kafka.streams.state.TimestampedWindowStore)4 ValueAndTimestamp (org.apache.kafka.streams.state.ValueAndTimestamp)4 WindowStore (org.apache.kafka.streams.state.WindowStore)4 HashMap (java.util.HashMap)3 KeyValueBytesStoreSupplier (org.apache.kafka.streams.state.KeyValueBytesStoreSupplier)3 SessionBytesStoreSupplier (org.apache.kafka.streams.state.SessionBytesStoreSupplier)3 KeyValueTimestamp (org.apache.kafka.streams.KeyValueTimestamp)2 ArrayList (java.util.ArrayList)1 HashSet (java.util.HashSet)1 List (java.util.List)1 Properties (java.util.Properties)1