Search in sources :

Example 11 with MaterializationException

use of io.confluent.ksql.execution.streams.materialization.MaterializationException in project ksql by confluentinc.

the class KsMaterializedWindowTableIQv2 method get.

public KsMaterializedQueryResult<WindowedRow> get(final int partition, final Range<Instant> windowStartBounds, final Range<Instant> windowEndBounds, final Optional<Position> position) {
    try {
        final Instant lower = calculateLowerBound(windowStartBounds, windowEndBounds);
        final Instant upper = calculateUpperBound(windowStartBounds, windowEndBounds);
        final WindowRangeQuery<GenericKey, ValueAndTimestamp<GenericRow>> query = WindowRangeQuery.withWindowStartRange(lower, upper);
        StateQueryRequest<KeyValueIterator<Windowed<GenericKey>, ValueAndTimestamp<GenericRow>>> request = inStore(stateStore.getStateStoreName()).withQuery(query);
        if (position.isPresent()) {
            request = request.withPositionBound(PositionBound.at(position.get()));
        }
        final StateQueryResult<KeyValueIterator<Windowed<GenericKey>, ValueAndTimestamp<GenericRow>>> result = stateStore.getKafkaStreams().query(request);
        final QueryResult<KeyValueIterator<Windowed<GenericKey>, ValueAndTimestamp<GenericRow>>> queryResult = result.getPartitionResults().get(partition);
        if (queryResult.isFailure()) {
            throw failedQueryException(queryResult);
        }
        final KeyValueIterator<Windowed<GenericKey>, ValueAndTimestamp<GenericRow>> iterator = queryResult.getResult();
        return KsMaterializedQueryResult.rowIteratorWithPosition(Streams.stream(IteratorUtil.onComplete(iterator, iterator::close)).map(next -> {
            final Instant windowStart = next.key.window().startTime();
            if (!windowStartBounds.contains(windowStart)) {
                return null;
            }
            final Instant windowEnd = next.key.window().endTime();
            if (!windowEndBounds.contains(windowEnd)) {
                return null;
            }
            final TimeWindow window = new TimeWindow(windowStart.toEpochMilli(), windowEnd.toEpochMilli());
            final WindowedRow row = WindowedRow.of(stateStore.schema(), new Windowed<>(next.key.key(), window), next.value.value(), next.value.timestamp());
            return row;
        }).filter(Objects::nonNull).iterator(), queryResult.getPosition());
    } catch (final NotUpToBoundException | MaterializationException e) {
        throw e;
    } catch (final Exception e) {
        throw new MaterializationException("Failed to get value from materialized table", e);
    }
}
Also used : WindowKeyQuery(org.apache.kafka.streams.query.WindowKeyQuery) StateQueryRequest.inStore(org.apache.kafka.streams.query.StateQueryRequest.inStore) MaterializationException(io.confluent.ksql.execution.streams.materialization.MaterializationException) PositionBound(org.apache.kafka.streams.query.PositionBound) WindowRangeQuery(org.apache.kafka.streams.query.WindowRangeQuery) ValueAndTimestamp(org.apache.kafka.streams.state.ValueAndTimestamp) StateQueryRequest(org.apache.kafka.streams.query.StateQueryRequest) IteratorUtil(io.confluent.ksql.util.IteratorUtil) ImmutableList(com.google.common.collect.ImmutableList) WindowedRow(io.confluent.ksql.execution.streams.materialization.WindowedRow) Windowed(org.apache.kafka.streams.kstream.Windowed) Duration(java.time.Duration) QueryResult(org.apache.kafka.streams.query.QueryResult) Position(org.apache.kafka.streams.query.Position) Range(com.google.common.collect.Range) KeyValue(org.apache.kafka.streams.KeyValue) FailureReason(org.apache.kafka.streams.query.FailureReason) MaterializedWindowedTable(io.confluent.ksql.execution.streams.materialization.MaterializedWindowedTable) Streams(com.google.common.collect.Streams) Instant(java.time.Instant) StreamsMaterializedWindowedTable(io.confluent.ksql.execution.streams.materialization.StreamsMaterializedWindowedTable) Objects(java.util.Objects) KeyValueIterator(org.apache.kafka.streams.state.KeyValueIterator) StateQueryResult(org.apache.kafka.streams.query.StateQueryResult) WindowStoreIterator(org.apache.kafka.streams.state.WindowStoreIterator) GenericRow(io.confluent.ksql.GenericRow) Optional(java.util.Optional) GenericKey(io.confluent.ksql.GenericKey) KafkaStreams(org.apache.kafka.streams.KafkaStreams) Builder(com.google.common.collect.ImmutableList.Builder) TimeWindow(org.apache.kafka.streams.kstream.internals.TimeWindow) Collections(java.util.Collections) Instant(java.time.Instant) TimeWindow(org.apache.kafka.streams.kstream.internals.TimeWindow) MaterializationException(io.confluent.ksql.execution.streams.materialization.MaterializationException) MaterializationException(io.confluent.ksql.execution.streams.materialization.MaterializationException) ValueAndTimestamp(org.apache.kafka.streams.state.ValueAndTimestamp) GenericRow(io.confluent.ksql.GenericRow) Windowed(org.apache.kafka.streams.kstream.Windowed) KeyValueIterator(org.apache.kafka.streams.state.KeyValueIterator) Objects(java.util.Objects) GenericKey(io.confluent.ksql.GenericKey) WindowedRow(io.confluent.ksql.execution.streams.materialization.WindowedRow)

Example 12 with MaterializationException

use of io.confluent.ksql.execution.streams.materialization.MaterializationException in project ksql by confluentinc.

the class KsMaterializedTable method get.

@Override
public KsMaterializedQueryResult<Row> get(final int partition, final Optional<Position> position) {
    try {
        final ReadOnlyKeyValueStore<GenericKey, ValueAndTimestamp<GenericRow>> store = stateStore.store(QueryableStoreTypes.timestampedKeyValueStore(), partition);
        final KeyValueIterator<GenericKey, ValueAndTimestamp<GenericRow>> iterator = store.all();
        return KsMaterializedQueryResult.rowIterator(Streams.stream(IteratorUtil.onComplete(iterator, iterator::close)).map(keyValue -> Row.of(stateStore.schema(), keyValue.key, keyValue.value.value(), keyValue.value.timestamp())).iterator());
    } catch (final Exception e) {
        throw new MaterializationException("Failed to scan materialized table", e);
    }
}
Also used : ValueAndTimestamp(org.apache.kafka.streams.state.ValueAndTimestamp) GenericKey(io.confluent.ksql.GenericKey) MaterializationException(io.confluent.ksql.execution.streams.materialization.MaterializationException) MaterializationException(io.confluent.ksql.execution.streams.materialization.MaterializationException)

Example 13 with MaterializationException

use of io.confluent.ksql.execution.streams.materialization.MaterializationException in project ksql by confluentinc.

the class KsMaterializedTable method get.

@Override
public KsMaterializedQueryResult<Row> get(final GenericKey key, final int partition, final Optional<Position> position) {
    try {
        final ReadOnlyKeyValueStore<GenericKey, ValueAndTimestamp<GenericRow>> store = stateStore.store(QueryableStoreTypes.timestampedKeyValueStore(), partition);
        final ValueAndTimestamp<GenericRow> row = store.get(key);
        if (row == null) {
            return KsMaterializedQueryResult.rowIterator(Collections.emptyIterator());
        } else {
            return KsMaterializedQueryResult.rowIterator(ImmutableList.of(Row.of(stateStore.schema(), key, row.value(), row.timestamp())).iterator());
        }
    } catch (final Exception e) {
        throw new MaterializationException("Failed to get value from materialized table", e);
    }
}
Also used : ValueAndTimestamp(org.apache.kafka.streams.state.ValueAndTimestamp) GenericRow(io.confluent.ksql.GenericRow) GenericKey(io.confluent.ksql.GenericKey) MaterializationException(io.confluent.ksql.execution.streams.materialization.MaterializationException) MaterializationException(io.confluent.ksql.execution.streams.materialization.MaterializationException)

Aggregations

MaterializationException (io.confluent.ksql.execution.streams.materialization.MaterializationException)13 GenericKey (io.confluent.ksql.GenericKey)10 ImmutableList (com.google.common.collect.ImmutableList)7 GenericRow (io.confluent.ksql.GenericRow)7 Objects (java.util.Objects)7 Optional (java.util.Optional)7 ValueAndTimestamp (org.apache.kafka.streams.state.ValueAndTimestamp)7 Streams (com.google.common.collect.Streams)5 WindowedRow (io.confluent.ksql.execution.streams.materialization.WindowedRow)5 IteratorUtil (io.confluent.ksql.util.IteratorUtil)4 Collections (java.util.Collections)4 Position (org.apache.kafka.streams.query.Position)4 KeyValueIterator (org.apache.kafka.streams.state.KeyValueIterator)4 VisibleForTesting (com.google.common.annotations.VisibleForTesting)3 Preconditions (com.google.common.base.Preconditions)3 Instant (java.time.Instant)3 TimeWindow (org.apache.kafka.streams.kstream.internals.TimeWindow)3 FailureReason (org.apache.kafka.streams.query.FailureReason)3 PositionBound (org.apache.kafka.streams.query.PositionBound)3 QueryResult (org.apache.kafka.streams.query.QueryResult)3