Search in sources :

Example 1 with Compression

use of org.locationtech.geowave.datastore.redis.config.RedisOptions.Compression in project geowave by locationtech.

the class RedisReader method createIteratorForReader.

private CloseableIterator<T> createIteratorForReader(final RedissonClient client, final Compression compression, final ReaderParams<T> readerParams, final GeoWaveRowIteratorTransformer<T> rowTransformer, final String namespace, final boolean visibilityEnabled, final boolean async) {
    final Collection<SinglePartitionQueryRanges> ranges = readerParams.getQueryRanges().getPartitionQueryRanges();
    final Set<String> authorizations = Sets.newHashSet(readerParams.getAdditionalAuthorizations());
    if ((ranges != null) && !ranges.isEmpty()) {
        return createIterator(client, compression, readerParams, readerParams.getRowTransformer(), namespace, ranges, authorizations, visibilityEnabled, async);
    } else {
        final Iterator<GeoWaveRedisRow>[] iterators = new Iterator[readerParams.getAdapterIds().length];
        int i = 0;
        for (final short adapterId : readerParams.getAdapterIds()) {
            final Pair<Boolean, Boolean> groupByRowAndSortByTime = RedisUtils.isGroupByRowAndIsSortByTime(readerParams, adapterId);
            final String setNamePrefix = RedisUtils.getRowSetPrefix(namespace, readerParams.getInternalAdapterStore().getTypeName(adapterId), readerParams.getIndex().getName());
            final Stream<Pair<ByteArray, Iterator<ScoredEntry<GeoWaveRedisPersistedRow>>>> streamIt = RedisUtils.getPartitions(client, setNamePrefix).stream().map(p -> {
                final Iterator<ScoredEntry<GeoWaveRedisPersistedRow>> result = RedisUtils.getRowSet(client, compression, setNamePrefix, p.getBytes(), groupByRowAndSortByTime.getRight(), visibilityEnabled).entryRange(Double.NEGATIVE_INFINITY, true, Double.POSITIVE_INFINITY, true);
                final Iterator<ScoredEntry<GeoWaveRedisPersistedRow>> it = groupByRowAndSortByTime.getLeft() ? RedisUtils.groupByRow(result, groupByRowAndSortByTime.getRight()) : result;
                return ImmutablePair.of(p, it);
            });
            iterators[i++] = Iterators.concat(streamIt.map(p -> Iterators.transform(p.getRight(), pr -> new GeoWaveRedisRow(pr.getValue(), adapterId, p.getLeft().getBytes(), RedisUtils.getFullSortKey(pr.getScore(), pr.getValue().getSortKeyPrecisionBeyondScore())))).iterator());
        }
        return wrapResults(Iterators.concat(iterators), readerParams, rowTransformer, authorizations, visibilityEnabled);
    }
}
Also used : ByteArray(org.locationtech.geowave.core.index.ByteArray) Arrays(java.util.Arrays) GeoWaveRedisRow(org.locationtech.geowave.datastore.redis.util.GeoWaveRedisRow) RecordReaderParams(org.locationtech.geowave.mapreduce.splits.RecordReaderParams) SinglePartitionQueryRanges(org.locationtech.geowave.core.index.SinglePartitionQueryRanges) GeoWaveRowIteratorTransformer(org.locationtech.geowave.core.store.entities.GeoWaveRowIteratorTransformer) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) ArrayUtils(org.apache.commons.lang3.ArrayUtils) RedisUtils(org.locationtech.geowave.datastore.redis.util.RedisUtils) Iterators(com.google.common.collect.Iterators) Compression(org.locationtech.geowave.datastore.redis.config.RedisOptions.Compression) Pair(org.apache.commons.lang3.tuple.Pair) CloseableIteratorWrapper(org.locationtech.geowave.core.store.CloseableIteratorWrapper) ReaderParams(org.locationtech.geowave.core.store.operations.ReaderParams) ByteArrayRange(org.locationtech.geowave.core.index.ByteArrayRange) ClientVisibilityFilter(org.locationtech.geowave.core.store.query.filter.ClientVisibilityFilter) RedissonClient(org.redisson.api.RedissonClient) DataStoreUtils(org.locationtech.geowave.core.store.util.DataStoreUtils) GeoWaveRow(org.locationtech.geowave.core.store.entities.GeoWaveRow) Iterator(java.util.Iterator) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) DataIndexReaderParams(org.locationtech.geowave.core.store.operations.DataIndexReaderParams) RowReader(org.locationtech.geowave.core.store.operations.RowReader) Collection(java.util.Collection) Set(java.util.Set) IOException(java.io.IOException) Streams(com.google.common.collect.Streams) ImmutablePair(org.apache.commons.lang3.tuple.ImmutablePair) ScoredEntry(org.redisson.client.protocol.ScoredEntry) Sets(com.google.common.collect.Sets) RangeReaderParams(org.locationtech.geowave.core.store.operations.RangeReaderParams) Stream(java.util.stream.Stream) CloseableIterator(org.locationtech.geowave.core.store.CloseableIterator) GeoWaveRedisPersistedRow(org.locationtech.geowave.datastore.redis.util.GeoWaveRedisPersistedRow) Closeable(java.io.Closeable) Serialization(org.locationtech.geowave.datastore.redis.config.RedisOptions.Serialization) GeoWaveRowRange(org.locationtech.geowave.mapreduce.splits.GeoWaveRowRange) Collections(java.util.Collections) SinglePartitionQueryRanges(org.locationtech.geowave.core.index.SinglePartitionQueryRanges) GeoWaveRedisRow(org.locationtech.geowave.datastore.redis.util.GeoWaveRedisRow) Iterator(java.util.Iterator) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) CloseableIterator(org.locationtech.geowave.core.store.CloseableIterator) ScoredEntry(org.redisson.client.protocol.ScoredEntry) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) Pair(org.apache.commons.lang3.tuple.Pair) ImmutablePair(org.apache.commons.lang3.tuple.ImmutablePair)

Example 2 with Compression

use of org.locationtech.geowave.datastore.redis.config.RedisOptions.Compression in project geowave by locationtech.

the class RedisReader method createIterator.

private CloseableIterator<T> createIterator(final RedissonClient client, final Compression compression, final RangeReaderParams<T> readerParams, final GeoWaveRowIteratorTransformer<T> rowTransformer, final String namespace, final Collection<SinglePartitionQueryRanges> ranges, final Set<String> authorizations, final boolean visibilityEnabled, final boolean async) {
    final Iterator<CloseableIterator> it = Arrays.stream(ArrayUtils.toObject(readerParams.getAdapterIds())).map(adapterId -> new BatchedRangeRead(client, compression, RedisUtils.getRowSetPrefix(namespace, readerParams.getInternalAdapterStore().getTypeName(adapterId), readerParams.getIndex().getName()), adapterId, ranges, rowTransformer, new ClientVisibilityFilter(authorizations), DataStoreUtils.isMergingIteratorRequired(readerParams, visibilityEnabled), async, RedisUtils.isGroupByRowAndIsSortByTime(readerParams, adapterId), RedisUtils.isSortByKeyRequired(readerParams), visibilityEnabled).results()).iterator();
    final CloseableIterator<T>[] itArray = Iterators.toArray(it, CloseableIterator.class);
    return new CloseableIteratorWrapper<>(new Closeable() {

        AtomicBoolean closed = new AtomicBoolean(false);

        @Override
        public void close() throws IOException {
            if (!closed.getAndSet(true)) {
                Arrays.stream(itArray).forEach(it -> it.close());
            }
        }
    }, Iterators.concat(itArray));
}
Also used : ByteArray(org.locationtech.geowave.core.index.ByteArray) Arrays(java.util.Arrays) GeoWaveRedisRow(org.locationtech.geowave.datastore.redis.util.GeoWaveRedisRow) RecordReaderParams(org.locationtech.geowave.mapreduce.splits.RecordReaderParams) SinglePartitionQueryRanges(org.locationtech.geowave.core.index.SinglePartitionQueryRanges) GeoWaveRowIteratorTransformer(org.locationtech.geowave.core.store.entities.GeoWaveRowIteratorTransformer) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) ArrayUtils(org.apache.commons.lang3.ArrayUtils) RedisUtils(org.locationtech.geowave.datastore.redis.util.RedisUtils) Iterators(com.google.common.collect.Iterators) Compression(org.locationtech.geowave.datastore.redis.config.RedisOptions.Compression) Pair(org.apache.commons.lang3.tuple.Pair) CloseableIteratorWrapper(org.locationtech.geowave.core.store.CloseableIteratorWrapper) ReaderParams(org.locationtech.geowave.core.store.operations.ReaderParams) ByteArrayRange(org.locationtech.geowave.core.index.ByteArrayRange) ClientVisibilityFilter(org.locationtech.geowave.core.store.query.filter.ClientVisibilityFilter) RedissonClient(org.redisson.api.RedissonClient) DataStoreUtils(org.locationtech.geowave.core.store.util.DataStoreUtils) GeoWaveRow(org.locationtech.geowave.core.store.entities.GeoWaveRow) Iterator(java.util.Iterator) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) DataIndexReaderParams(org.locationtech.geowave.core.store.operations.DataIndexReaderParams) RowReader(org.locationtech.geowave.core.store.operations.RowReader) Collection(java.util.Collection) Set(java.util.Set) IOException(java.io.IOException) Streams(com.google.common.collect.Streams) ImmutablePair(org.apache.commons.lang3.tuple.ImmutablePair) ScoredEntry(org.redisson.client.protocol.ScoredEntry) Sets(com.google.common.collect.Sets) RangeReaderParams(org.locationtech.geowave.core.store.operations.RangeReaderParams) Stream(java.util.stream.Stream) CloseableIterator(org.locationtech.geowave.core.store.CloseableIterator) GeoWaveRedisPersistedRow(org.locationtech.geowave.datastore.redis.util.GeoWaveRedisPersistedRow) Closeable(java.io.Closeable) Serialization(org.locationtech.geowave.datastore.redis.config.RedisOptions.Serialization) GeoWaveRowRange(org.locationtech.geowave.mapreduce.splits.GeoWaveRowRange) Collections(java.util.Collections) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) CloseableIterator(org.locationtech.geowave.core.store.CloseableIterator) Closeable(java.io.Closeable) IOException(java.io.IOException) CloseableIteratorWrapper(org.locationtech.geowave.core.store.CloseableIteratorWrapper) ClientVisibilityFilter(org.locationtech.geowave.core.store.query.filter.ClientVisibilityFilter)

Aggregations

Iterators (com.google.common.collect.Iterators)2 Sets (com.google.common.collect.Sets)2 Streams (com.google.common.collect.Streams)2 Closeable (java.io.Closeable)2 IOException (java.io.IOException)2 Arrays (java.util.Arrays)2 Collection (java.util.Collection)2 Collections (java.util.Collections)2 Iterator (java.util.Iterator)2 Set (java.util.Set)2 AtomicBoolean (java.util.concurrent.atomic.AtomicBoolean)2 Stream (java.util.stream.Stream)2 ArrayUtils (org.apache.commons.lang3.ArrayUtils)2 ImmutablePair (org.apache.commons.lang3.tuple.ImmutablePair)2 Pair (org.apache.commons.lang3.tuple.Pair)2 ByteArray (org.locationtech.geowave.core.index.ByteArray)2 ByteArrayRange (org.locationtech.geowave.core.index.ByteArrayRange)2 SinglePartitionQueryRanges (org.locationtech.geowave.core.index.SinglePartitionQueryRanges)2 CloseableIterator (org.locationtech.geowave.core.store.CloseableIterator)2 CloseableIteratorWrapper (org.locationtech.geowave.core.store.CloseableIteratorWrapper)2