Search in sources :

Example 1 with RocksDbKeyValueBytesStoreSupplier

use of org.apache.kafka.streams.state.internals.RocksDbKeyValueBytesStoreSupplier in project kafka by apache.

the class EosIntegrationTest method verifyStateIsInStoreAndOffsetsAreInCheckpoint.

private void verifyStateIsInStoreAndOffsetsAreInCheckpoint(final int partition, final Set<KeyValue<Long, Long>> expectedState) throws IOException {
    final String stateStoreDir = stateTmpDir + File.separator + "appDir" + File.separator + applicationId + File.separator + "0_" + partition + File.separator;
    // Verify that the data in the state store on disk is fully up-to-date
    final StateStoreContext context = new MockInternalProcessorContext(new Properties(), new TaskId(0, 0), new File(stateStoreDir));
    final MockKeyValueStore stateStore = new MockKeyValueStore("store", false);
    final RocksDBStore store = (RocksDBStore) new RocksDbKeyValueBytesStoreSupplier(storeName, false).get();
    store.init(context, stateStore);
    store.all().forEachRemaining(kv -> {
        final KeyValue<Long, Long> kv2 = new KeyValue<>(new BigInteger(kv.key.get()).longValue(), new BigInteger(kv.value).longValue());
        expectedState.remove(kv2);
    });
    // Verify that the checkpointed offsets match exactly with max offset of the records in the changelog
    final OffsetCheckpoint checkpoint = new OffsetCheckpoint(new File(stateStoreDir + ".checkpoint"));
    final Map<TopicPartition, Long> checkpointedOffsets = checkpoint.read();
    checkpointedOffsets.forEach(this::verifyChangelogMaxRecordOffsetMatchesCheckpointedOffset);
}
Also used : OffsetCheckpoint(org.apache.kafka.streams.state.internals.OffsetCheckpoint) TaskId(org.apache.kafka.streams.processor.TaskId) KeyValue(org.apache.kafka.streams.KeyValue) RocksDbKeyValueBytesStoreSupplier(org.apache.kafka.streams.state.internals.RocksDbKeyValueBytesStoreSupplier) Properties(java.util.Properties) MockKeyValueStore(org.apache.kafka.test.MockKeyValueStore) StateStoreContext(org.apache.kafka.streams.processor.StateStoreContext) TopicPartition(org.apache.kafka.common.TopicPartition) BigInteger(java.math.BigInteger) MockInternalProcessorContext(org.apache.kafka.test.MockInternalProcessorContext) File(java.io.File) RocksDBStore(org.apache.kafka.streams.state.internals.RocksDBStore)

Aggregations

File (java.io.File)1 BigInteger (java.math.BigInteger)1 Properties (java.util.Properties)1 TopicPartition (org.apache.kafka.common.TopicPartition)1 KeyValue (org.apache.kafka.streams.KeyValue)1 StateStoreContext (org.apache.kafka.streams.processor.StateStoreContext)1 TaskId (org.apache.kafka.streams.processor.TaskId)1 OffsetCheckpoint (org.apache.kafka.streams.state.internals.OffsetCheckpoint)1 RocksDBStore (org.apache.kafka.streams.state.internals.RocksDBStore)1 RocksDbKeyValueBytesStoreSupplier (org.apache.kafka.streams.state.internals.RocksDbKeyValueBytesStoreSupplier)1 MockInternalProcessorContext (org.apache.kafka.test.MockInternalProcessorContext)1 MockKeyValueStore (org.apache.kafka.test.MockKeyValueStore)1