Search in sources :

Example 1 with TimestampedKeyValueStoreMaterializer

use of org.apache.kafka.streams.kstream.internals.TimestampedKeyValueStoreMaterializer in project kafka by apache.

the class GlobalStreamThreadTest method before.

@Before
public void before() {
    final MaterializedInternal<Object, Object, KeyValueStore<Bytes, byte[]>> materialized = new MaterializedInternal<>(Materialized.with(null, null), new InternalNameProvider() {

        @Override
        public String newProcessorName(final String prefix) {
            return "processorName";
        }

        @Override
        public String newStoreName(final String prefix) {
            return GLOBAL_STORE_NAME;
        }
    }, "store-");
    final ProcessorSupplier<Object, Object, Void, Void> processorSupplier = () -> new ContextualProcessor<Object, Object, Void, Void>() {

        @Override
        public void process(final Record<Object, Object> record) {
        }
    };
    builder.addGlobalStore(new TimestampedKeyValueStoreMaterializer<>(materialized).materialize().withLoggingDisabled(), "sourceName", null, null, null, GLOBAL_STORE_TOPIC_NAME, "processorName", processorSupplier);
    baseDirectoryName = TestUtils.tempDirectory().getAbsolutePath();
    final HashMap<String, Object> properties = new HashMap<>();
    properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "blah");
    properties.put(StreamsConfig.APPLICATION_ID_CONFIG, "testAppId");
    properties.put(StreamsConfig.STATE_DIR_CONFIG, baseDirectoryName);
    properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.ByteArraySerde.class.getName());
    properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.ByteArraySerde.class.getName());
    config = new StreamsConfig(properties);
    globalStreamThread = new GlobalStreamThread(builder.rewriteTopology(config).buildGlobalStateTopology(), config, mockConsumer, new StateDirectory(config, time, true, false), 0, new StreamsMetricsImpl(new Metrics(), "test-client", StreamsConfig.METRICS_LATEST, time), time, "clientId", stateRestoreListener, e -> {
    });
}
Also used : CoreMatchers.is(org.hamcrest.CoreMatchers.is) StreamsConfig(org.apache.kafka.streams.StreamsConfig) MockTime(org.apache.kafka.common.utils.MockTime) InternalNameProvider(org.apache.kafka.streams.kstream.internals.InternalNameProvider) IsInstanceOf.instanceOf(org.hamcrest.core.IsInstanceOf.instanceOf) MockConsumer(org.apache.kafka.clients.consumer.MockConsumer) CoreMatchers.equalTo(org.hamcrest.CoreMatchers.equalTo) HashMap(java.util.HashMap) StreamsException(org.apache.kafka.streams.errors.StreamsException) OffsetResetStrategy(org.apache.kafka.clients.consumer.OffsetResetStrategy) DEAD(org.apache.kafka.streams.processor.internals.GlobalStreamThread.State.DEAD) ContextualProcessor(org.apache.kafka.streams.processor.api.ContextualProcessor) ProcessorSupplier(org.apache.kafka.streams.processor.api.ProcessorSupplier) KeyValueStore(org.apache.kafka.streams.state.KeyValueStore) Serdes(org.apache.kafka.common.serialization.Serdes) Record(org.apache.kafka.streams.processor.api.Record) StreamsMetricsImpl(org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl) Assert.fail(org.junit.Assert.fail) MatcherAssert.assertThat(org.hamcrest.MatcherAssert.assertThat) Before(org.junit.Before) TopicPartition(org.apache.kafka.common.TopicPartition) RUNNING(org.apache.kafka.streams.processor.internals.GlobalStreamThread.State.RUNNING) MockStateRestoreListener(org.apache.kafka.test.MockStateRestoreListener) TestUtils(org.apache.kafka.test.TestUtils) Set(java.util.Set) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) PartitionInfo(org.apache.kafka.common.PartitionInfo) MaterializedInternal(org.apache.kafka.streams.kstream.internals.MaterializedInternal) File(java.io.File) Bytes(org.apache.kafka.common.utils.Bytes) ConsumerRecordUtil.record(org.apache.kafka.streams.processor.internals.testutil.ConsumerRecordUtil.record) List(java.util.List) Metrics(org.apache.kafka.common.metrics.Metrics) TimestampedKeyValueStoreMaterializer(org.apache.kafka.streams.kstream.internals.TimestampedKeyValueStoreMaterializer) StateStore(org.apache.kafka.streams.processor.StateStore) Assert.assertFalse(org.junit.Assert.assertFalse) Materialized(org.apache.kafka.streams.kstream.Materialized) InvalidOffsetException(org.apache.kafka.clients.consumer.InvalidOffsetException) Node(org.apache.kafka.common.Node) Collections(java.util.Collections) Assert.assertEquals(org.junit.Assert.assertEquals) HashMap(java.util.HashMap) KeyValueStore(org.apache.kafka.streams.state.KeyValueStore) InternalNameProvider(org.apache.kafka.streams.kstream.internals.InternalNameProvider) Metrics(org.apache.kafka.common.metrics.Metrics) Record(org.apache.kafka.streams.processor.api.Record) ContextualProcessor(org.apache.kafka.streams.processor.api.ContextualProcessor) MaterializedInternal(org.apache.kafka.streams.kstream.internals.MaterializedInternal) StreamsMetricsImpl(org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl) StreamsConfig(org.apache.kafka.streams.StreamsConfig) Before(org.junit.Before)

Example 2 with TimestampedKeyValueStoreMaterializer

use of org.apache.kafka.streams.kstream.internals.TimestampedKeyValueStoreMaterializer in project kafka by apache.

the class TimestampedKeyValueStoreMaterializerTest method shouldCreateBuilderThatBuildsStoreWithCachingAndLoggingDisabled.

@Test
public void shouldCreateBuilderThatBuildsStoreWithCachingAndLoggingDisabled() {
    final MaterializedInternal<String, String, KeyValueStore<Bytes, byte[]>> materialized = new MaterializedInternal<>(Materialized.<String, String, KeyValueStore<Bytes, byte[]>>as("store").withCachingDisabled().withLoggingDisabled(), nameProvider, storePrefix);
    final TimestampedKeyValueStoreMaterializer<String, String> materializer = new TimestampedKeyValueStoreMaterializer<>(materialized);
    final StoreBuilder<TimestampedKeyValueStore<String, String>> builder = materializer.materialize();
    final TimestampedKeyValueStore<String, String> store = builder.build();
    final StateStore wrapped = ((WrappedStateStore) store).wrapped();
    assertThat(wrapped, not(instanceOf(CachingKeyValueStore.class)));
    assertThat(wrapped, not(instanceOf(ChangeLoggingKeyValueBytesStore.class)));
}
Also used : Bytes(org.apache.kafka.common.utils.Bytes) TimestampedKeyValueStoreMaterializer(org.apache.kafka.streams.kstream.internals.TimestampedKeyValueStoreMaterializer) WrappedStateStore(org.apache.kafka.streams.state.internals.WrappedStateStore) TimestampedKeyValueStore(org.apache.kafka.streams.state.TimestampedKeyValueStore) MeteredTimestampedKeyValueStore(org.apache.kafka.streams.state.internals.MeteredTimestampedKeyValueStore) WrappedStateStore(org.apache.kafka.streams.state.internals.WrappedStateStore) StateStore(org.apache.kafka.streams.processor.StateStore) KeyValueStore(org.apache.kafka.streams.state.KeyValueStore) InMemoryKeyValueStore(org.apache.kafka.streams.state.internals.InMemoryKeyValueStore) CachingKeyValueStore(org.apache.kafka.streams.state.internals.CachingKeyValueStore) TimestampedKeyValueStore(org.apache.kafka.streams.state.TimestampedKeyValueStore) MeteredTimestampedKeyValueStore(org.apache.kafka.streams.state.internals.MeteredTimestampedKeyValueStore) MaterializedInternal(org.apache.kafka.streams.kstream.internals.MaterializedInternal) Test(org.junit.Test)

Example 3 with TimestampedKeyValueStoreMaterializer

use of org.apache.kafka.streams.kstream.internals.TimestampedKeyValueStoreMaterializer in project kafka by apache.

the class TimestampedKeyValueStoreMaterializerTest method shouldCreateBuilderThatBuildsMeteredStoreWithCachingAndLoggingEnabled.

@Test
public void shouldCreateBuilderThatBuildsMeteredStoreWithCachingAndLoggingEnabled() {
    final MaterializedInternal<String, String, KeyValueStore<Bytes, byte[]>> materialized = new MaterializedInternal<>(Materialized.as("store"), nameProvider, storePrefix);
    final TimestampedKeyValueStoreMaterializer<String, String> materializer = new TimestampedKeyValueStoreMaterializer<>(materialized);
    final StoreBuilder<TimestampedKeyValueStore<String, String>> builder = materializer.materialize();
    final TimestampedKeyValueStore<String, String> store = builder.build();
    final WrappedStateStore caching = (WrappedStateStore) ((WrappedStateStore) store).wrapped();
    final StateStore logging = caching.wrapped();
    assertThat(store, instanceOf(MeteredTimestampedKeyValueStore.class));
    assertThat(caching, instanceOf(CachingKeyValueStore.class));
    assertThat(logging, instanceOf(ChangeLoggingTimestampedKeyValueBytesStore.class));
}
Also used : TimestampedKeyValueStoreMaterializer(org.apache.kafka.streams.kstream.internals.TimestampedKeyValueStoreMaterializer) CachingKeyValueStore(org.apache.kafka.streams.state.internals.CachingKeyValueStore) WrappedStateStore(org.apache.kafka.streams.state.internals.WrappedStateStore) WrappedStateStore(org.apache.kafka.streams.state.internals.WrappedStateStore) StateStore(org.apache.kafka.streams.processor.StateStore) KeyValueStore(org.apache.kafka.streams.state.KeyValueStore) InMemoryKeyValueStore(org.apache.kafka.streams.state.internals.InMemoryKeyValueStore) CachingKeyValueStore(org.apache.kafka.streams.state.internals.CachingKeyValueStore) TimestampedKeyValueStore(org.apache.kafka.streams.state.TimestampedKeyValueStore) MeteredTimestampedKeyValueStore(org.apache.kafka.streams.state.internals.MeteredTimestampedKeyValueStore) ChangeLoggingTimestampedKeyValueBytesStore(org.apache.kafka.streams.state.internals.ChangeLoggingTimestampedKeyValueBytesStore) TimestampedKeyValueStore(org.apache.kafka.streams.state.TimestampedKeyValueStore) MeteredTimestampedKeyValueStore(org.apache.kafka.streams.state.internals.MeteredTimestampedKeyValueStore) MeteredTimestampedKeyValueStore(org.apache.kafka.streams.state.internals.MeteredTimestampedKeyValueStore) MaterializedInternal(org.apache.kafka.streams.kstream.internals.MaterializedInternal) Test(org.junit.Test)

Example 4 with TimestampedKeyValueStoreMaterializer

use of org.apache.kafka.streams.kstream.internals.TimestampedKeyValueStoreMaterializer in project kafka by apache.

the class TimestampedKeyValueStoreMaterializerTest method shouldCreateBuilderThatBuildsStoreWithCachingDisabled.

@Test
public void shouldCreateBuilderThatBuildsStoreWithCachingDisabled() {
    final MaterializedInternal<String, String, KeyValueStore<Bytes, byte[]>> materialized = new MaterializedInternal<>(Materialized.<String, String, KeyValueStore<Bytes, byte[]>>as("store").withCachingDisabled(), nameProvider, storePrefix);
    final TimestampedKeyValueStoreMaterializer<String, String> materializer = new TimestampedKeyValueStoreMaterializer<>(materialized);
    final StoreBuilder<TimestampedKeyValueStore<String, String>> builder = materializer.materialize();
    final TimestampedKeyValueStore<String, String> store = builder.build();
    final WrappedStateStore logging = (WrappedStateStore) ((WrappedStateStore) store).wrapped();
    assertThat(logging, instanceOf(ChangeLoggingKeyValueBytesStore.class));
}
Also used : TimestampedKeyValueStoreMaterializer(org.apache.kafka.streams.kstream.internals.TimestampedKeyValueStoreMaterializer) WrappedStateStore(org.apache.kafka.streams.state.internals.WrappedStateStore) TimestampedKeyValueStore(org.apache.kafka.streams.state.TimestampedKeyValueStore) MeteredTimestampedKeyValueStore(org.apache.kafka.streams.state.internals.MeteredTimestampedKeyValueStore) ChangeLoggingKeyValueBytesStore(org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore) KeyValueStore(org.apache.kafka.streams.state.KeyValueStore) InMemoryKeyValueStore(org.apache.kafka.streams.state.internals.InMemoryKeyValueStore) CachingKeyValueStore(org.apache.kafka.streams.state.internals.CachingKeyValueStore) TimestampedKeyValueStore(org.apache.kafka.streams.state.TimestampedKeyValueStore) MeteredTimestampedKeyValueStore(org.apache.kafka.streams.state.internals.MeteredTimestampedKeyValueStore) MaterializedInternal(org.apache.kafka.streams.kstream.internals.MaterializedInternal) Test(org.junit.Test)

Example 5 with TimestampedKeyValueStoreMaterializer

use of org.apache.kafka.streams.kstream.internals.TimestampedKeyValueStoreMaterializer in project kafka by apache.

the class TimestampedKeyValueStoreMaterializerTest method shouldCreateKeyValueStoreWithTheProvidedInnerStore.

@Test
public void shouldCreateKeyValueStoreWithTheProvidedInnerStore() {
    final KeyValueBytesStoreSupplier supplier = EasyMock.createNiceMock(KeyValueBytesStoreSupplier.class);
    final InMemoryKeyValueStore store = new InMemoryKeyValueStore("name");
    EasyMock.expect(supplier.name()).andReturn("name").anyTimes();
    EasyMock.expect(supplier.get()).andReturn(store);
    EasyMock.expect(supplier.metricsScope()).andReturn("metricScope");
    EasyMock.replay(supplier);
    final MaterializedInternal<String, Integer, KeyValueStore<Bytes, byte[]>> materialized = new MaterializedInternal<>(Materialized.as(supplier), nameProvider, storePrefix);
    final TimestampedKeyValueStoreMaterializer<String, Integer> materializer = new TimestampedKeyValueStoreMaterializer<>(materialized);
    final StoreBuilder<TimestampedKeyValueStore<String, Integer>> builder = materializer.materialize();
    final TimestampedKeyValueStore<String, Integer> built = builder.build();
    assertThat(store.name(), CoreMatchers.equalTo(built.name()));
}
Also used : TimestampedKeyValueStoreMaterializer(org.apache.kafka.streams.kstream.internals.TimestampedKeyValueStoreMaterializer) KeyValueBytesStoreSupplier(org.apache.kafka.streams.state.KeyValueBytesStoreSupplier) TimestampedKeyValueStore(org.apache.kafka.streams.state.TimestampedKeyValueStore) MeteredTimestampedKeyValueStore(org.apache.kafka.streams.state.internals.MeteredTimestampedKeyValueStore) KeyValueStore(org.apache.kafka.streams.state.KeyValueStore) InMemoryKeyValueStore(org.apache.kafka.streams.state.internals.InMemoryKeyValueStore) CachingKeyValueStore(org.apache.kafka.streams.state.internals.CachingKeyValueStore) TimestampedKeyValueStore(org.apache.kafka.streams.state.TimestampedKeyValueStore) MeteredTimestampedKeyValueStore(org.apache.kafka.streams.state.internals.MeteredTimestampedKeyValueStore) MaterializedInternal(org.apache.kafka.streams.kstream.internals.MaterializedInternal) InMemoryKeyValueStore(org.apache.kafka.streams.state.internals.InMemoryKeyValueStore) Test(org.junit.Test)

Aggregations

MaterializedInternal (org.apache.kafka.streams.kstream.internals.MaterializedInternal)6 TimestampedKeyValueStoreMaterializer (org.apache.kafka.streams.kstream.internals.TimestampedKeyValueStoreMaterializer)6 KeyValueStore (org.apache.kafka.streams.state.KeyValueStore)6 Test (org.junit.Test)6 TimestampedKeyValueStore (org.apache.kafka.streams.state.TimestampedKeyValueStore)5 CachingKeyValueStore (org.apache.kafka.streams.state.internals.CachingKeyValueStore)5 InMemoryKeyValueStore (org.apache.kafka.streams.state.internals.InMemoryKeyValueStore)5 MeteredTimestampedKeyValueStore (org.apache.kafka.streams.state.internals.MeteredTimestampedKeyValueStore)5 WrappedStateStore (org.apache.kafka.streams.state.internals.WrappedStateStore)4 StateStore (org.apache.kafka.streams.processor.StateStore)3 Bytes (org.apache.kafka.common.utils.Bytes)2 File (java.io.File)1 Collections (java.util.Collections)1 HashMap (java.util.HashMap)1 List (java.util.List)1 Set (java.util.Set)1 InvalidOffsetException (org.apache.kafka.clients.consumer.InvalidOffsetException)1 MockConsumer (org.apache.kafka.clients.consumer.MockConsumer)1 OffsetResetStrategy (org.apache.kafka.clients.consumer.OffsetResetStrategy)1 Node (org.apache.kafka.common.Node)1