Search in sources :

Example 1 with GeoWaveRowMergingIterator

use of org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator in project geowave by locationtech.

the class CassandraReader method wrapResults.

@SuppressWarnings("unchecked")
private CloseableIterator<T> wrapResults(final CloseableIterator<CassandraRow> results, final RangeReaderParams<T> readerParams) {
    final Set<String> authorizations = Sets.newHashSet(readerParams.getAdditionalAuthorizations());
    final Iterator<GeoWaveRow> iterator = (Iterator) Streams.stream(results).filter(new ClientVisibilityFilter(authorizations)).iterator();
    return new CloseableIteratorWrapper<>(results, rowTransformer.apply(DataStoreUtils.isMergingIteratorRequired(readerParams, visibilityEnabled) ? new GeoWaveRowMergingIterator(iterator) : iterator));
}
Also used : GeoWaveRow(org.locationtech.geowave.core.store.entities.GeoWaveRow) Iterator(java.util.Iterator) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) CloseableIterator(org.locationtech.geowave.core.store.CloseableIterator) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) CloseableIteratorWrapper(org.locationtech.geowave.core.store.CloseableIteratorWrapper) ClientVisibilityFilter(org.locationtech.geowave.core.store.query.filter.ClientVisibilityFilter)

Example 2 with GeoWaveRowMergingIterator

use of org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator in project geowave by locationtech.

the class BatchedRangeRead method executeQueryAsync.

public CloseableIterator<T> executeQueryAsync(final Statement... statements) {
    // first create a list of asynchronous query executions
    final List<CompletionStage<AsyncResultSet>> futures = Lists.newArrayListWithExpectedSize(statements.length);
    final BlockingQueue<Object> results = new LinkedBlockingQueue<>(MAX_BOUNDED_READS_ENQUEUED);
    new Thread(new Runnable() {

        @Override
        public void run() {
            // set it to 1 to make sure all queries are submitted in
            // the loop
            final AtomicInteger queryCount = new AtomicInteger(1);
            for (final Statement s : statements) {
                try {
                    readSemaphore.acquire();
                    final CompletionStage<AsyncResultSet> f = operations.getSession().executeAsync(s);
                    synchronized (futures) {
                        futures.add(f);
                    }
                    queryCount.incrementAndGet();
                    f.whenCompleteAsync((result, t) -> {
                        if (result != null) {
                            try {
                                final Iterator<GeoWaveRow> iterator = (Iterator) Streams.stream(ResultSets.newInstance(result)).map(row -> new CassandraRow(row)).filter(filter).iterator();
                                rowTransformer.apply(rowMerging ? new GeoWaveRowMergingIterator(iterator) : iterator).forEachRemaining(row -> {
                                    try {
                                        results.put(row);
                                    } catch (final InterruptedException e) {
                                        LOGGER.warn("interrupted while waiting to enqueue a cassandra result", e);
                                    }
                                });
                            } finally {
                                checkFinalize(queryCount, results, readSemaphore);
                            }
                        } else if (t != null) {
                            checkFinalize(queryCount, results, readSemaphore);
                            // can do logging or start counting errors.
                            if (!(t instanceof CancellationException)) {
                                LOGGER.error("Failure from async query", t);
                                throw new RuntimeException(t);
                            }
                        }
                    });
                } catch (final InterruptedException e) {
                    LOGGER.warn("Exception while executing query", e);
                    readSemaphore.release();
                }
            }
            // then decrement
            if (queryCount.decrementAndGet() <= 0) {
                // statements submitted
                try {
                    results.put(RowConsumer.POISON);
                } catch (final InterruptedException e) {
                    LOGGER.error("Interrupted while finishing blocking queue, this may result in deadlock!");
                }
            }
        }
    }, "Cassandra Query Executor").start();
    return new CloseableIteratorWrapper<T>(new Closeable() {

        @Override
        public void close() throws IOException {
            synchronized (futures) {
                for (final CompletionStage<AsyncResultSet> f : futures) {
                    f.toCompletableFuture().cancel(true);
                }
            }
        }
    }, new RowConsumer(results));
}
Also used : CassandraRow(org.locationtech.geowave.datastore.cassandra.CassandraRow) Arrays(java.util.Arrays) ResultSets(com.datastax.oss.driver.internal.core.cql.ResultSets) SinglePartitionQueryRanges(org.locationtech.geowave.core.index.SinglePartitionQueryRanges) GeoWaveRowIteratorTransformer(org.locationtech.geowave.core.store.entities.GeoWaveRowIteratorTransformer) LoggerFactory(org.slf4j.LoggerFactory) ArrayUtils(org.apache.commons.lang3.ArrayUtils) ByteBuffer(java.nio.ByteBuffer) ArrayList(java.util.ArrayList) AsyncResultSet(com.datastax.oss.driver.api.core.cql.AsyncResultSet) Lists(com.google.common.collect.Lists) CloseableIteratorWrapper(org.locationtech.geowave.core.store.CloseableIteratorWrapper) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) ByteArrayRange(org.locationtech.geowave.core.index.ByteArrayRange) GeoWaveRow(org.locationtech.geowave.core.store.entities.GeoWaveRow) Logger(org.slf4j.Logger) Iterator(java.util.Iterator) CancellationException(java.util.concurrent.CancellationException) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) BoundStatementBuilder(com.datastax.oss.driver.api.core.cql.BoundStatementBuilder) Semaphore(java.util.concurrent.Semaphore) Predicate(java.util.function.Predicate) Collection(java.util.Collection) IOException(java.io.IOException) BlockingQueue(java.util.concurrent.BlockingQueue) BoundStatement(com.datastax.oss.driver.api.core.cql.BoundStatement) PreparedStatement(com.datastax.oss.driver.api.core.cql.PreparedStatement) Streams(com.google.common.collect.Streams) LinkedBlockingQueue(java.util.concurrent.LinkedBlockingQueue) RowConsumer(org.locationtech.geowave.core.store.util.RowConsumer) TypeCodecs(com.datastax.oss.driver.api.core.type.codec.TypeCodecs) CassandraUtils(org.locationtech.geowave.datastore.cassandra.util.CassandraUtils) List(java.util.List) CompletionStage(java.util.concurrent.CompletionStage) CloseableIterator(org.locationtech.geowave.core.store.CloseableIterator) Closeable(java.io.Closeable) CassandraField(org.locationtech.geowave.datastore.cassandra.CassandraRow.CassandraField) Statement(com.datastax.oss.driver.api.core.cql.Statement) GeoWaveRow(org.locationtech.geowave.core.store.entities.GeoWaveRow) AsyncResultSet(com.datastax.oss.driver.api.core.cql.AsyncResultSet) Closeable(java.io.Closeable) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) LinkedBlockingQueue(java.util.concurrent.LinkedBlockingQueue) Iterator(java.util.Iterator) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) CloseableIterator(org.locationtech.geowave.core.store.CloseableIterator) CompletionStage(java.util.concurrent.CompletionStage) BoundStatement(com.datastax.oss.driver.api.core.cql.BoundStatement) PreparedStatement(com.datastax.oss.driver.api.core.cql.PreparedStatement) Statement(com.datastax.oss.driver.api.core.cql.Statement) CassandraRow(org.locationtech.geowave.datastore.cassandra.CassandraRow) IOException(java.io.IOException) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) CancellationException(java.util.concurrent.CancellationException) RowConsumer(org.locationtech.geowave.core.store.util.RowConsumer) CloseableIteratorWrapper(org.locationtech.geowave.core.store.CloseableIteratorWrapper)

Example 3 with GeoWaveRowMergingIterator

use of org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator in project geowave by locationtech.

the class KuduRangeRead method executeScanner.

public Deferred<Object> executeScanner(final AsyncKuduScanner scanner, final Semaphore semaphore, final BlockingQueue<Object> resultQueue, final AtomicInteger queryCount, final AtomicBoolean isCanceled, final boolean visibilityEnabled, final Predicate<GeoWaveRow> filter, final GeoWaveRowIteratorTransformer<T> rowTransformer, final boolean rowMerging) {
    // Errback class
    class QueryErrback implements Callback<Deferred<Object>, Exception> {

        @Override
        public Deferred<Object> call(final Exception e) {
            LOGGER.warn("While scanning rows from kudu", e);
            checkFinalize(scanner, semaphore, resultQueue, queryCount);
            return Deferred.fromError(e);
        }
    }
    final QueryErrback errBack = new QueryErrback();
    // callback class
    class QueryCallback implements Callback<Deferred<Object>, RowResultIterator> {

        @Override
        public Deferred<Object> call(final RowResultIterator rs) {
            if ((rs == null) || isCanceled.get()) {
                checkFinalize(scanner, semaphore, resultQueue, queryCount);
                return Deferred.fromResult(null);
            }
            if (rs.getNumRows() > 0) {
                Stream<GeoWaveRow> tmpStream = Streams.stream(rs.iterator()).map(KuduRow::new);
                if (visibilityEnabled) {
                    tmpStream = tmpStream.filter(filter);
                }
                final Iterator<GeoWaveRow> tmpIterator = tmpStream.iterator();
                rowTransformer.apply(rowMerging ? new GeoWaveRowMergingIterator(tmpIterator) : tmpIterator).forEachRemaining(row -> {
                    try {
                        resultQueue.put(row);
                    } catch (final InterruptedException e) {
                        LOGGER.warn("interrupted while waiting to enqueue a kudu result", e);
                    }
                });
            }
            if (scanner.hasMoreRows()) {
                return scanner.nextRows().addCallbackDeferring(this).addErrback(errBack);
            }
            checkFinalize(scanner, semaphore, resultQueue, queryCount);
            return Deferred.fromResult(null);
        }
    }
    queryCount.incrementAndGet();
    return scanner.nextRows().addCallbackDeferring(new QueryCallback()).addErrback(errBack);
}
Also used : GeoWaveRow(org.locationtech.geowave.core.store.entities.GeoWaveRow) Callback(com.stumbleupon.async.Callback) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) KuduRow(org.locationtech.geowave.datastore.kudu.KuduRow) RowResultIterator(org.apache.kudu.client.RowResultIterator)

Example 4 with GeoWaveRowMergingIterator

use of org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator in project geowave by locationtech.

the class DynamoDBReader method startRead.

private void startRead(final List<QueryRequest> requests, final String tableName, final boolean rowMerging, final boolean parallelDecode) {
    Iterator<Map<String, AttributeValue>> rawIterator;
    Predicate<DynamoDBRow> adapterIdFilter = null;
    final Function<Iterator<Map<String, AttributeValue>>, Iterator<DynamoDBRow>> rawToDynamoDBRow = new Function<Iterator<Map<String, AttributeValue>>, Iterator<DynamoDBRow>>() {

        @Override
        public Iterator<DynamoDBRow> apply(final Iterator<Map<String, AttributeValue>> input) {
            final Iterator<DynamoDBRow> rowIterator = Streams.stream(input).map(new DynamoDBRow.GuavaRowTranslationHelper()).filter(visibilityFilter).iterator();
            if (rowMerging) {
                return new GeoWaveRowMergingIterator<>(rowIterator);
            } else {
                // TODO: understand why there are duplicates coming back when there shouldn't be from
                // DynamoDB
                final DedupeFilter dedupe = new DedupeFilter();
                return Iterators.filter(rowIterator, row -> dedupe.applyDedupeFilter(row.getAdapterId(), new ByteArray(row.getDataId())));
            }
        }
    };
    if (!requests.isEmpty()) {
        if (ASYNC) {
            rawIterator = Iterators.concat(requests.parallelStream().map(this::executeAsyncQueryRequest).iterator());
        } else {
            rawIterator = Iterators.concat(requests.parallelStream().map(this::executeQueryRequest).iterator());
        }
    } else {
        if (ASYNC) {
            final ScanRequest request = new ScanRequest(tableName);
            rawIterator = new AsyncPaginatedScan(request, operations.getClient());
        } else {
            // query everything
            final ScanRequest request = new ScanRequest(tableName);
            final ScanResult scanResult = operations.getClient().scan(request);
            rawIterator = new LazyPaginatedScan(scanResult, request, operations.getClient());
            // filtering by adapter ID
            if ((readerParams.getAdapterIds() != null) && (readerParams.getAdapterIds().length > 0)) {
                adapterIdFilter = input -> ArrayUtils.contains(readerParams.getAdapterIds(), input.getAdapterId());
            }
        }
    }
    Iterator<DynamoDBRow> rowIter = rawToDynamoDBRow.apply(rawIterator);
    if (adapterIdFilter != null) {
        rowIter = Streams.stream(rowIter).filter(adapterIdFilter).iterator();
    }
    if (parallelDecode) {
        final ParallelDecoder<T> decoder = new SimpleParallelDecoder<>(rowTransformer, Iterators.transform(rowIter, r -> (GeoWaveRow) r));
        try {
            decoder.startDecode();
        } catch (final Exception e) {
            Throwables.propagate(e);
        }
        iterator = decoder;
        closeable = decoder;
    } else {
        iterator = rowTransformer.apply(Iterators.transform(rowIter, r -> (GeoWaveRow) r));
        closeable = null;
    }
}
Also used : Condition(com.amazonaws.services.dynamodbv2.model.Condition) ByteArray(org.locationtech.geowave.core.index.ByteArray) QueryRequest(com.amazonaws.services.dynamodbv2.model.QueryRequest) RecordReaderParams(org.locationtech.geowave.mapreduce.splits.RecordReaderParams) SinglePartitionQueryRanges(org.locationtech.geowave.core.index.SinglePartitionQueryRanges) GeoWaveRowIteratorTransformer(org.locationtech.geowave.core.store.entities.GeoWaveRowIteratorTransformer) ScanResult(com.amazonaws.services.dynamodbv2.model.ScanResult) InternalAdapterStore(org.locationtech.geowave.core.store.adapter.InternalAdapterStore) ArrayUtils(org.apache.commons.lang3.ArrayUtils) Function(java.util.function.Function) ByteBuffer(java.nio.ByteBuffer) Iterators(com.google.common.collect.Iterators) ArrayList(java.util.ArrayList) Lists(com.google.common.collect.Lists) ReaderParams(org.locationtech.geowave.core.store.operations.ReaderParams) AttributeValue(com.amazonaws.services.dynamodbv2.model.AttributeValue) Map(java.util.Map) ByteArrayRange(org.locationtech.geowave.core.index.ByteArrayRange) ClientVisibilityFilter(org.locationtech.geowave.core.store.query.filter.ClientVisibilityFilter) ParallelDecoder(org.locationtech.geowave.core.store.operations.ParallelDecoder) DynamoDBRow(org.locationtech.geowave.datastore.dynamodb.DynamoDBRow) DataStoreUtils(org.locationtech.geowave.core.store.util.DataStoreUtils) GeoWaveRow(org.locationtech.geowave.core.store.entities.GeoWaveRow) AsyncPaginatedQuery(org.locationtech.geowave.datastore.dynamodb.util.AsyncPaginatedQuery) DynamoDBUtils(org.locationtech.geowave.datastore.dynamodb.util.DynamoDBUtils) Iterator(java.util.Iterator) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) RowReader(org.locationtech.geowave.core.store.operations.RowReader) Predicate(java.util.function.Predicate) Collection(java.util.Collection) SimpleParallelDecoder(org.locationtech.geowave.core.store.operations.SimpleParallelDecoder) ScanRequest(com.amazonaws.services.dynamodbv2.model.ScanRequest) Throwables(com.google.common.base.Throwables) Streams(com.google.common.collect.Streams) QueryResult(com.amazonaws.services.dynamodbv2.model.QueryResult) Sets(com.google.common.collect.Sets) RangeReaderParams(org.locationtech.geowave.core.store.operations.RangeReaderParams) AsyncPaginatedScan(org.locationtech.geowave.datastore.dynamodb.util.AsyncPaginatedScan) LazyPaginatedQuery(org.locationtech.geowave.datastore.dynamodb.util.LazyPaginatedQuery) LazyPaginatedScan(org.locationtech.geowave.datastore.dynamodb.util.LazyPaginatedScan) ComparisonOperator(com.amazonaws.services.dynamodbv2.model.ComparisonOperator) List(java.util.List) DedupeFilter(org.locationtech.geowave.core.store.query.filter.DedupeFilter) GeoWaveRowRange(org.locationtech.geowave.mapreduce.splits.GeoWaveRowRange) ByteArrayUtils(org.locationtech.geowave.core.index.ByteArrayUtils) AttributeValue(com.amazonaws.services.dynamodbv2.model.AttributeValue) ScanResult(com.amazonaws.services.dynamodbv2.model.ScanResult) GeoWaveRow(org.locationtech.geowave.core.store.entities.GeoWaveRow) AsyncPaginatedScan(org.locationtech.geowave.datastore.dynamodb.util.AsyncPaginatedScan) SimpleParallelDecoder(org.locationtech.geowave.core.store.operations.SimpleParallelDecoder) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) DynamoDBRow(org.locationtech.geowave.datastore.dynamodb.DynamoDBRow) Function(java.util.function.Function) ScanRequest(com.amazonaws.services.dynamodbv2.model.ScanRequest) DedupeFilter(org.locationtech.geowave.core.store.query.filter.DedupeFilter) Iterator(java.util.Iterator) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) ByteArray(org.locationtech.geowave.core.index.ByteArray) Map(java.util.Map) LazyPaginatedScan(org.locationtech.geowave.datastore.dynamodb.util.LazyPaginatedScan)

Aggregations

GeoWaveRow (org.locationtech.geowave.core.store.entities.GeoWaveRow)4 GeoWaveRowMergingIterator (org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator)4 Iterator (java.util.Iterator)3 Lists (com.google.common.collect.Lists)2 Streams (com.google.common.collect.Streams)2 ByteBuffer (java.nio.ByteBuffer)2 ArrayList (java.util.ArrayList)2 Collection (java.util.Collection)2 List (java.util.List)2 Predicate (java.util.function.Predicate)2 ArrayUtils (org.apache.commons.lang3.ArrayUtils)2 ByteArrayRange (org.locationtech.geowave.core.index.ByteArrayRange)2 SinglePartitionQueryRanges (org.locationtech.geowave.core.index.SinglePartitionQueryRanges)2 CloseableIterator (org.locationtech.geowave.core.store.CloseableIterator)2 CloseableIteratorWrapper (org.locationtech.geowave.core.store.CloseableIteratorWrapper)2 GeoWaveRowIteratorTransformer (org.locationtech.geowave.core.store.entities.GeoWaveRowIteratorTransformer)2 ClientVisibilityFilter (org.locationtech.geowave.core.store.query.filter.ClientVisibilityFilter)2 AttributeValue (com.amazonaws.services.dynamodbv2.model.AttributeValue)1 ComparisonOperator (com.amazonaws.services.dynamodbv2.model.ComparisonOperator)1 Condition (com.amazonaws.services.dynamodbv2.model.Condition)1