Search in sources :

Example 1 with TopologyWrapper

use of org.apache.kafka.streams.TopologyWrapper in project kafka by apache.

the class ProcessorTopologyTest method createGlobalStoreTopology.

// testing old PAPI
@Deprecated
private ProcessorTopology createGlobalStoreTopology(final KeyValueBytesStoreSupplier storeSupplier) {
    final TopologyWrapper topology = new TopologyWrapper();
    final StoreBuilder<KeyValueStore<String, String>> storeBuilder = Stores.keyValueStoreBuilder(storeSupplier, Serdes.String(), Serdes.String()).withLoggingDisabled();
    topology.addGlobalStore(storeBuilder, "global", STRING_DESERIALIZER, STRING_DESERIALIZER, "topic", "processor", define(new OldAPIStatefulProcessor(storeSupplier.name())));
    return topology.getInternalBuilder("anyAppId").buildTopology();
}
Also used : KeyValueStore(org.apache.kafka.streams.state.KeyValueStore) TopologyWrapper(org.apache.kafka.streams.TopologyWrapper)

Example 2 with TopologyWrapper

use of org.apache.kafka.streams.TopologyWrapper in project kafka by apache.

the class ProcessorTopologyTest method createLocalStoreTopology.

private ProcessorTopology createLocalStoreTopology(final KeyValueBytesStoreSupplier storeSupplier) {
    final TopologyWrapper topology = new TopologyWrapper();
    final String processor = "processor";
    final StoreBuilder<KeyValueStore<String, String>> storeBuilder = Stores.keyValueStoreBuilder(storeSupplier, Serdes.String(), Serdes.String());
    topology.addSource("source", STRING_DESERIALIZER, STRING_DESERIALIZER, "topic").addProcessor(processor, () -> new StatefulProcessor(storeSupplier.name()), "source").addStateStore(storeBuilder, processor);
    return topology.getInternalBuilder("anyAppId").buildTopology();
}
Also used : KeyValueStore(org.apache.kafka.streams.state.KeyValueStore) TopologyWrapper(org.apache.kafka.streams.TopologyWrapper) CoreMatchers.containsString(org.hamcrest.CoreMatchers.containsString)

Example 3 with TopologyWrapper

use of org.apache.kafka.streams.TopologyWrapper in project kafka by apache.

the class ProcessorTopologyTest method statelessTopologyShouldNotHavePersistentStore.

@Test
public void statelessTopologyShouldNotHavePersistentStore() {
    final TopologyWrapper topology = new TopologyWrapper();
    final ProcessorTopology processorTopology = topology.getInternalBuilder("anyAppId").buildTopology();
    assertFalse(processorTopology.hasPersistentLocalStore());
    assertFalse(processorTopology.hasPersistentGlobalStore());
}
Also used : TopologyWrapper(org.apache.kafka.streams.TopologyWrapper) Test(org.junit.Test)

Example 4 with TopologyWrapper

use of org.apache.kafka.streams.TopologyWrapper in project kafka by apache.

the class RegexSourceIntegrationTest method shouldAddStateStoreToRegexDefinedSource.

@Test
public void shouldAddStateStoreToRegexDefinedSource() throws Exception {
    final StoreBuilder<KeyValueStore<Object, Object>> storeBuilder = new MockKeyValueStoreBuilder("testStateStore", false);
    final long thirtySecondTimeout = 30 * 1000;
    final TopologyWrapper topology = new TopologyWrapper();
    topology.addSource("ingest", Pattern.compile("topic-\\d+"));
    topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
    topology.addStateStore(storeBuilder, "my-processor");
    streams = new KafkaStreams(topology, streamsConfiguration);
    streams.start();
    final TestCondition stateStoreNameBoundToSourceTopic = () -> {
        final Map<String, List<String>> stateStoreToSourceTopic = topology.getInternalBuilder().stateStoreNameToFullSourceTopicNames();
        final List<String> topicNamesList = stateStoreToSourceTopic.get("testStateStore");
        return topicNamesList != null && !topicNamesList.isEmpty() && topicNamesList.get(0).equals("topic-1");
    };
    TestUtils.waitForCondition(stateStoreNameBoundToSourceTopic, thirtySecondTimeout, "Did not find topic: [topic-1] connected to state store: [testStateStore]");
}
Also used : KafkaStreams(org.apache.kafka.streams.KafkaStreams) KeyValueStore(org.apache.kafka.streams.state.KeyValueStore) TestCondition(org.apache.kafka.test.TestCondition) List(java.util.List) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) ArrayList(java.util.ArrayList) TopologyWrapper(org.apache.kafka.streams.TopologyWrapper) Map(java.util.Map) MockKeyValueStoreBuilder(org.apache.kafka.test.MockKeyValueStoreBuilder) IntegrationTest(org.apache.kafka.test.IntegrationTest) Test(org.junit.Test)

Example 5 with TopologyWrapper

use of org.apache.kafka.streams.TopologyWrapper in project kafka by apache.

the class StreamThreadStateStoreProviderTest method before.

@Before
public void before() {
    final TopologyWrapper topology = new TopologyWrapper();
    topology.addSource("the-source", topicName);
    topology.addProcessor("the-processor", new MockApiProcessorSupplier<>(), "the-source");
    topology.addStateStore(Stores.keyValueStoreBuilder(Stores.inMemoryKeyValueStore("kv-store"), Serdes.String(), Serdes.String()), "the-processor");
    topology.addStateStore(Stores.timestampedKeyValueStoreBuilder(Stores.inMemoryKeyValueStore("timestamped-kv-store"), Serdes.String(), Serdes.String()), "the-processor");
    topology.addStateStore(Stores.windowStoreBuilder(Stores.inMemoryWindowStore("window-store", Duration.ofMillis(10L), Duration.ofMillis(2L), false), Serdes.String(), Serdes.String()), "the-processor");
    topology.addStateStore(Stores.timestampedWindowStoreBuilder(Stores.inMemoryWindowStore("timestamped-window-store", Duration.ofMillis(10L), Duration.ofMillis(2L), false), Serdes.String(), Serdes.String()), "the-processor");
    topology.addStateStore(Stores.sessionStoreBuilder(Stores.inMemorySessionStore("session-store", Duration.ofMillis(10L)), Serdes.String(), Serdes.String()), "the-processor");
    final Properties properties = new Properties();
    final String applicationId = "applicationId";
    properties.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
    properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    stateDir = TestUtils.tempDirectory();
    properties.put(StreamsConfig.STATE_DIR_CONFIG, stateDir.getPath());
    final StreamsConfig streamsConfig = new StreamsConfig(properties);
    final MockClientSupplier clientSupplier = new MockClientSupplier();
    configureClients(clientSupplier, "applicationId-kv-store-changelog");
    configureClients(clientSupplier, "applicationId-window-store-changelog");
    final InternalTopologyBuilder internalTopologyBuilder = topology.getInternalBuilder(applicationId);
    final ProcessorTopology processorTopology = internalTopologyBuilder.buildTopology();
    tasks = new HashMap<>();
    stateDirectory = new StateDirectory(streamsConfig, new MockTime(), true, false);
    taskOne = createStreamsTask(streamsConfig, clientSupplier, processorTopology, new TaskId(0, 0));
    taskOne.initializeIfNeeded();
    tasks.put(new TaskId(0, 0), taskOne);
    final StreamTask taskTwo = createStreamsTask(streamsConfig, clientSupplier, processorTopology, new TaskId(0, 1));
    taskTwo.initializeIfNeeded();
    tasks.put(new TaskId(0, 1), taskTwo);
    threadMock = EasyMock.createNiceMock(StreamThread.class);
    provider = new StreamThreadStateStoreProvider(threadMock);
}
Also used : ProcessorTopology(org.apache.kafka.streams.processor.internals.ProcessorTopology) TaskId(org.apache.kafka.streams.processor.TaskId) StreamThread(org.apache.kafka.streams.processor.internals.StreamThread) Properties(java.util.Properties) InternalTopologyBuilder(org.apache.kafka.streams.processor.internals.InternalTopologyBuilder) MockClientSupplier(org.apache.kafka.test.MockClientSupplier) TopologyWrapper(org.apache.kafka.streams.TopologyWrapper) MockTime(org.apache.kafka.common.utils.MockTime) StreamTask(org.apache.kafka.streams.processor.internals.StreamTask) StreamsConfig(org.apache.kafka.streams.StreamsConfig) StateDirectory(org.apache.kafka.streams.processor.internals.StateDirectory) Before(org.junit.Before)

Aggregations

TopologyWrapper (org.apache.kafka.streams.TopologyWrapper)5 KeyValueStore (org.apache.kafka.streams.state.KeyValueStore)3 Test (org.junit.Test)2 ArrayList (java.util.ArrayList)1 List (java.util.List)1 Map (java.util.Map)1 Properties (java.util.Properties)1 CopyOnWriteArrayList (java.util.concurrent.CopyOnWriteArrayList)1 MockTime (org.apache.kafka.common.utils.MockTime)1 KafkaStreams (org.apache.kafka.streams.KafkaStreams)1 StreamsConfig (org.apache.kafka.streams.StreamsConfig)1 TaskId (org.apache.kafka.streams.processor.TaskId)1 InternalTopologyBuilder (org.apache.kafka.streams.processor.internals.InternalTopologyBuilder)1 ProcessorTopology (org.apache.kafka.streams.processor.internals.ProcessorTopology)1 StateDirectory (org.apache.kafka.streams.processor.internals.StateDirectory)1 StreamTask (org.apache.kafka.streams.processor.internals.StreamTask)1 StreamThread (org.apache.kafka.streams.processor.internals.StreamThread)1 IntegrationTest (org.apache.kafka.test.IntegrationTest)1 MockClientSupplier (org.apache.kafka.test.MockClientSupplier)1 MockKeyValueStoreBuilder (org.apache.kafka.test.MockKeyValueStoreBuilder)1