Search in sources :

Example 1 with CloseableIteratorWrapper

use of org.locationtech.geowave.core.store.CloseableIteratorWrapper in project geowave by locationtech.

the class AccumuloOperations method getDataIndexResults.

public CloseableIterator<GeoWaveRow> getDataIndexResults(final short adapterId, final String... additionalAuthorizations) {
    final byte[] family = StringUtils.stringToBinary(ByteArrayUtils.shortToString(adapterId));
    // to have backwards compatibility before 1.8.0 we can assume BaseScanner is autocloseable
    final Scanner scanner;
    try {
        scanner = createScanner(DataIndexUtils.DATA_ID_INDEX.getName(), additionalAuthorizations);
        scanner.setRange(new Range());
        scanner.fetchColumnFamily(new Text(family));
        return new CloseableIteratorWrapper(new Closeable() {

            @Override
            public void close() throws IOException {
                scanner.close();
            }
        }, Streams.stream(scanner).map(entry -> DataIndexUtils.deserializeDataIndexRow(entry.getKey().getRow().getBytes(), adapterId, entry.getValue().get(), false)).iterator());
    } catch (final TableNotFoundException e) {
        LOGGER.error("unable to find data index table", e);
    }
    return new CloseableIterator.Empty<>();
}
Also used : BatchScanner(org.apache.accumulo.core.client.BatchScanner) Scanner(org.apache.accumulo.core.client.Scanner) ClientSideIteratorScanner(org.apache.accumulo.core.client.ClientSideIteratorScanner) TableNotFoundException(org.apache.accumulo.core.client.TableNotFoundException) Closeable(java.io.Closeable) Text(org.apache.hadoop.io.Text) IOException(java.io.IOException) ByteArrayRange(org.locationtech.geowave.core.index.ByteArrayRange) GeoWaveRowRange(org.locationtech.geowave.mapreduce.splits.GeoWaveRowRange) Range(org.apache.accumulo.core.data.Range) CloseableIteratorWrapper(org.locationtech.geowave.core.store.CloseableIteratorWrapper)

Example 2 with CloseableIteratorWrapper

use of org.locationtech.geowave.core.store.CloseableIteratorWrapper in project geowave by locationtech.

the class AccumuloUtils method getIterator.

private static CloseableIterator<Entry<Key, Value>> getIterator(final Connector connector, final String namespace, final Index index) throws AccumuloException, AccumuloSecurityException, IOException, TableNotFoundException {
    CloseableIterator<Entry<Key, Value>> iterator = null;
    final AccumuloOptions options = new AccumuloOptions();
    final AccumuloOperations operations = new AccumuloOperations(connector, namespace, new AccumuloOptions());
    final IndexStore indexStore = new IndexStoreImpl(operations, options);
    final PersistentAdapterStore adapterStore = new AdapterStoreImpl(operations, options);
    final AdapterIndexMappingStore mappingStore = new AdapterIndexMappingStoreImpl(operations, options);
    if (indexStore.indexExists(index.getName())) {
        final ScannerBase scanner = operations.createBatchScanner(index.getName());
        ((BatchScanner) scanner).setRanges(AccumuloUtils.byteArrayRangesToAccumuloRanges(null));
        final IteratorSetting iteratorSettings = new IteratorSetting(10, "GEOWAVE_WHOLE_ROW_ITERATOR", WholeRowIterator.class);
        scanner.addScanIterator(iteratorSettings);
        final Iterator<Entry<Key, Value>> it = new IteratorWrapper(adapterStore, mappingStore, index, scanner.iterator(), new QueryFilter[] { new DedupeFilter() });
        iterator = new CloseableIteratorWrapper<>(new ScannerClosableWrapper(scanner), it);
    }
    return iterator;
}
Also used : AccumuloOperations(org.locationtech.geowave.datastore.accumulo.operations.AccumuloOperations) ScannerBase(org.apache.accumulo.core.client.ScannerBase) BatchScanner(org.apache.accumulo.core.client.BatchScanner) AdapterIndexMappingStore(org.locationtech.geowave.core.store.adapter.AdapterIndexMappingStore) IndexStoreImpl(org.locationtech.geowave.core.store.metadata.IndexStoreImpl) Entry(java.util.Map.Entry) PersistentAdapterStore(org.locationtech.geowave.core.store.adapter.PersistentAdapterStore) AdapterIndexMappingStoreImpl(org.locationtech.geowave.core.store.metadata.AdapterIndexMappingStoreImpl) IteratorSetting(org.apache.accumulo.core.client.IteratorSetting) CloseableIteratorWrapper(org.locationtech.geowave.core.store.CloseableIteratorWrapper) AccumuloOptions(org.locationtech.geowave.datastore.accumulo.config.AccumuloOptions) DedupeFilter(org.locationtech.geowave.core.store.query.filter.DedupeFilter) AdapterStoreImpl(org.locationtech.geowave.core.store.metadata.AdapterStoreImpl) IndexStore(org.locationtech.geowave.core.store.index.IndexStore)

Example 3 with CloseableIteratorWrapper

use of org.locationtech.geowave.core.store.CloseableIteratorWrapper in project geowave by locationtech.

the class CassandraReader method wrapResults.

@SuppressWarnings("unchecked")
private CloseableIterator<T> wrapResults(final CloseableIterator<CassandraRow> results, final RangeReaderParams<T> readerParams) {
    final Set<String> authorizations = Sets.newHashSet(readerParams.getAdditionalAuthorizations());
    final Iterator<GeoWaveRow> iterator = (Iterator) Streams.stream(results).filter(new ClientVisibilityFilter(authorizations)).iterator();
    return new CloseableIteratorWrapper<>(results, rowTransformer.apply(DataStoreUtils.isMergingIteratorRequired(readerParams, visibilityEnabled) ? new GeoWaveRowMergingIterator(iterator) : iterator));
}
Also used : GeoWaveRow(org.locationtech.geowave.core.store.entities.GeoWaveRow) Iterator(java.util.Iterator) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) CloseableIterator(org.locationtech.geowave.core.store.CloseableIterator) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) CloseableIteratorWrapper(org.locationtech.geowave.core.store.CloseableIteratorWrapper) ClientVisibilityFilter(org.locationtech.geowave.core.store.query.filter.ClientVisibilityFilter)

Example 4 with CloseableIteratorWrapper

use of org.locationtech.geowave.core.store.CloseableIteratorWrapper in project geowave by locationtech.

the class BatchedRangeRead method executeQueryAsync.

public CloseableIterator<T> executeQueryAsync(final Statement... statements) {
    // first create a list of asynchronous query executions
    final List<CompletionStage<AsyncResultSet>> futures = Lists.newArrayListWithExpectedSize(statements.length);
    final BlockingQueue<Object> results = new LinkedBlockingQueue<>(MAX_BOUNDED_READS_ENQUEUED);
    new Thread(new Runnable() {

        @Override
        public void run() {
            // set it to 1 to make sure all queries are submitted in
            // the loop
            final AtomicInteger queryCount = new AtomicInteger(1);
            for (final Statement s : statements) {
                try {
                    readSemaphore.acquire();
                    final CompletionStage<AsyncResultSet> f = operations.getSession().executeAsync(s);
                    synchronized (futures) {
                        futures.add(f);
                    }
                    queryCount.incrementAndGet();
                    f.whenCompleteAsync((result, t) -> {
                        if (result != null) {
                            try {
                                final Iterator<GeoWaveRow> iterator = (Iterator) Streams.stream(ResultSets.newInstance(result)).map(row -> new CassandraRow(row)).filter(filter).iterator();
                                rowTransformer.apply(rowMerging ? new GeoWaveRowMergingIterator(iterator) : iterator).forEachRemaining(row -> {
                                    try {
                                        results.put(row);
                                    } catch (final InterruptedException e) {
                                        LOGGER.warn("interrupted while waiting to enqueue a cassandra result", e);
                                    }
                                });
                            } finally {
                                checkFinalize(queryCount, results, readSemaphore);
                            }
                        } else if (t != null) {
                            checkFinalize(queryCount, results, readSemaphore);
                            // can do logging or start counting errors.
                            if (!(t instanceof CancellationException)) {
                                LOGGER.error("Failure from async query", t);
                                throw new RuntimeException(t);
                            }
                        }
                    });
                } catch (final InterruptedException e) {
                    LOGGER.warn("Exception while executing query", e);
                    readSemaphore.release();
                }
            }
            // then decrement
            if (queryCount.decrementAndGet() <= 0) {
                // statements submitted
                try {
                    results.put(RowConsumer.POISON);
                } catch (final InterruptedException e) {
                    LOGGER.error("Interrupted while finishing blocking queue, this may result in deadlock!");
                }
            }
        }
    }, "Cassandra Query Executor").start();
    return new CloseableIteratorWrapper<T>(new Closeable() {

        @Override
        public void close() throws IOException {
            synchronized (futures) {
                for (final CompletionStage<AsyncResultSet> f : futures) {
                    f.toCompletableFuture().cancel(true);
                }
            }
        }
    }, new RowConsumer(results));
}
Also used : CassandraRow(org.locationtech.geowave.datastore.cassandra.CassandraRow) Arrays(java.util.Arrays) ResultSets(com.datastax.oss.driver.internal.core.cql.ResultSets) SinglePartitionQueryRanges(org.locationtech.geowave.core.index.SinglePartitionQueryRanges) GeoWaveRowIteratorTransformer(org.locationtech.geowave.core.store.entities.GeoWaveRowIteratorTransformer) LoggerFactory(org.slf4j.LoggerFactory) ArrayUtils(org.apache.commons.lang3.ArrayUtils) ByteBuffer(java.nio.ByteBuffer) ArrayList(java.util.ArrayList) AsyncResultSet(com.datastax.oss.driver.api.core.cql.AsyncResultSet) Lists(com.google.common.collect.Lists) CloseableIteratorWrapper(org.locationtech.geowave.core.store.CloseableIteratorWrapper) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) ByteArrayRange(org.locationtech.geowave.core.index.ByteArrayRange) GeoWaveRow(org.locationtech.geowave.core.store.entities.GeoWaveRow) Logger(org.slf4j.Logger) Iterator(java.util.Iterator) CancellationException(java.util.concurrent.CancellationException) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) BoundStatementBuilder(com.datastax.oss.driver.api.core.cql.BoundStatementBuilder) Semaphore(java.util.concurrent.Semaphore) Predicate(java.util.function.Predicate) Collection(java.util.Collection) IOException(java.io.IOException) BlockingQueue(java.util.concurrent.BlockingQueue) BoundStatement(com.datastax.oss.driver.api.core.cql.BoundStatement) PreparedStatement(com.datastax.oss.driver.api.core.cql.PreparedStatement) Streams(com.google.common.collect.Streams) LinkedBlockingQueue(java.util.concurrent.LinkedBlockingQueue) RowConsumer(org.locationtech.geowave.core.store.util.RowConsumer) TypeCodecs(com.datastax.oss.driver.api.core.type.codec.TypeCodecs) CassandraUtils(org.locationtech.geowave.datastore.cassandra.util.CassandraUtils) List(java.util.List) CompletionStage(java.util.concurrent.CompletionStage) CloseableIterator(org.locationtech.geowave.core.store.CloseableIterator) Closeable(java.io.Closeable) CassandraField(org.locationtech.geowave.datastore.cassandra.CassandraRow.CassandraField) Statement(com.datastax.oss.driver.api.core.cql.Statement) GeoWaveRow(org.locationtech.geowave.core.store.entities.GeoWaveRow) AsyncResultSet(com.datastax.oss.driver.api.core.cql.AsyncResultSet) Closeable(java.io.Closeable) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) LinkedBlockingQueue(java.util.concurrent.LinkedBlockingQueue) Iterator(java.util.Iterator) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) CloseableIterator(org.locationtech.geowave.core.store.CloseableIterator) CompletionStage(java.util.concurrent.CompletionStage) BoundStatement(com.datastax.oss.driver.api.core.cql.BoundStatement) PreparedStatement(com.datastax.oss.driver.api.core.cql.PreparedStatement) Statement(com.datastax.oss.driver.api.core.cql.Statement) CassandraRow(org.locationtech.geowave.datastore.cassandra.CassandraRow) IOException(java.io.IOException) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) CancellationException(java.util.concurrent.CancellationException) RowConsumer(org.locationtech.geowave.core.store.util.RowConsumer) CloseableIteratorWrapper(org.locationtech.geowave.core.store.CloseableIteratorWrapper)

Example 5 with CloseableIteratorWrapper

use of org.locationtech.geowave.core.store.CloseableIteratorWrapper in project geowave by locationtech.

the class FileSystemReader method createIterator.

private CloseableIterator<T> createIterator(final FileSystemClient client, final RangeReaderParams<T> readerParams, final GeoWaveRowIteratorTransformer<T> rowTransformer, final Collection<SinglePartitionQueryRanges> ranges, final Set<String> authorizations, final boolean async) {
    final Iterator<CloseableIterator> it = Arrays.stream(ArrayUtils.toObject(readerParams.getAdapterIds())).map(adapterId -> new FileSystemQueryExecution(client, adapterId, readerParams.getInternalAdapterStore().getTypeName(adapterId), readerParams.getIndex().getName(), client.getFormat(), rowTransformer, ranges, new ClientVisibilityFilter(authorizations), DataStoreUtils.isMergingIteratorRequired(readerParams, client.isVisibilityEnabled()), async, FileSystemUtils.isGroupByRowAndIsSortByTime(readerParams, adapterId), FileSystemUtils.isSortByKeyRequired(readerParams)).results()).iterator();
    final CloseableIterator<T>[] itArray = Iterators.toArray(it, CloseableIterator.class);
    return new CloseableIteratorWrapper<>(new Closeable() {

        AtomicBoolean closed = new AtomicBoolean(false);

        @Override
        public void close() throws IOException {
            if (!closed.getAndSet(true)) {
                Arrays.stream(itArray).forEach(it -> it.close());
            }
        }
    }, Iterators.concat(itArray));
}
Also used : Arrays(java.util.Arrays) RecordReaderParams(org.locationtech.geowave.mapreduce.splits.RecordReaderParams) SinglePartitionQueryRanges(org.locationtech.geowave.core.index.SinglePartitionQueryRanges) GeoWaveRowIteratorTransformer(org.locationtech.geowave.core.store.entities.GeoWaveRowIteratorTransformer) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) ArrayUtils(org.apache.commons.lang3.ArrayUtils) IndexFormatter(org.locationtech.geowave.datastore.filesystem.FileSystemDataFormatter.IndexFormatter) Iterators(com.google.common.collect.Iterators) ArrayList(java.util.ArrayList) Pair(org.apache.commons.lang3.tuple.Pair) CloseableIteratorWrapper(org.locationtech.geowave.core.store.CloseableIteratorWrapper) ReaderParams(org.locationtech.geowave.core.store.operations.ReaderParams) ByteArrayRange(org.locationtech.geowave.core.index.ByteArrayRange) ClientVisibilityFilter(org.locationtech.geowave.core.store.query.filter.ClientVisibilityFilter) FileSystemDataIndexTable(org.locationtech.geowave.datastore.filesystem.util.FileSystemDataIndexTable) DataFormatterCache(org.locationtech.geowave.datastore.filesystem.util.DataFormatterCache) DataStoreUtils(org.locationtech.geowave.core.store.util.DataStoreUtils) GeoWaveRow(org.locationtech.geowave.core.store.entities.GeoWaveRow) Iterator(java.util.Iterator) GeoWaveRowMergingIterator(org.locationtech.geowave.core.store.entities.GeoWaveRowMergingIterator) DataIndexReaderParams(org.locationtech.geowave.core.store.operations.DataIndexReaderParams) RowReader(org.locationtech.geowave.core.store.operations.RowReader) Collection(java.util.Collection) Set(java.util.Set) FileSystemUtils(org.locationtech.geowave.datastore.filesystem.util.FileSystemUtils) IOException(java.io.IOException) Streams(com.google.common.collect.Streams) Collectors(java.util.stream.Collectors) Sets(com.google.common.collect.Sets) RangeReaderParams(org.locationtech.geowave.core.store.operations.RangeReaderParams) List(java.util.List) Stream(java.util.stream.Stream) CloseableIterator(org.locationtech.geowave.core.store.CloseableIterator) Closeable(java.io.Closeable) GeoWaveRowRange(org.locationtech.geowave.mapreduce.splits.GeoWaveRowRange) Collections(java.util.Collections) FileSystemClient(org.locationtech.geowave.datastore.filesystem.util.FileSystemClient) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) CloseableIterator(org.locationtech.geowave.core.store.CloseableIterator) Closeable(java.io.Closeable) IOException(java.io.IOException) CloseableIteratorWrapper(org.locationtech.geowave.core.store.CloseableIteratorWrapper) ClientVisibilityFilter(org.locationtech.geowave.core.store.query.filter.ClientVisibilityFilter)

Aggregations

CloseableIteratorWrapper (org.locationtech.geowave.core.store.CloseableIteratorWrapper)20 CloseableIterator (org.locationtech.geowave.core.store.CloseableIterator)14 IOException (java.io.IOException)12 Closeable (java.io.Closeable)11 Iterator (java.util.Iterator)10 GeoWaveRow (org.locationtech.geowave.core.store.entities.GeoWaveRow)10 Iterators (com.google.common.collect.Iterators)9 Arrays (java.util.Arrays)9 ArrayList (java.util.ArrayList)8 List (java.util.List)8 ByteArrayRange (org.locationtech.geowave.core.index.ByteArrayRange)8 Logger (org.slf4j.Logger)8 LoggerFactory (org.slf4j.LoggerFactory)8 Streams (com.google.common.collect.Streams)7 Collection (java.util.Collection)7 Collections (java.util.Collections)7 ByteArray (org.locationtech.geowave.core.index.ByteArray)7 MetadataReader (org.locationtech.geowave.core.store.operations.MetadataReader)6 MetadataType (org.locationtech.geowave.core.store.operations.MetadataType)6 Collectors (java.util.stream.Collectors)5