Search in sources :

Example 1 with BatchGetPartitionRequest

use of com.amazonaws.services.glue.model.BatchGetPartitionRequest in project presto by prestodb.

the class GlueHiveMetastore method batchGetPartition.

private List<Partition> batchGetPartition(String databaseName, String tableName, List<String> partitionNames) {
    try {
        List<Future<BatchGetPartitionResult>> batchGetPartitionFutures = new ArrayList<>();
        for (List<String> partitionNamesBatch : Lists.partition(partitionNames, BATCH_GET_PARTITION_MAX_PAGE_SIZE)) {
            List<PartitionValueList> partitionValuesBatch = mappedCopy(partitionNamesBatch, partitionName -> new PartitionValueList().withValues(toPartitionValues(partitionName)));
            batchGetPartitionFutures.add(glueClient.batchGetPartitionAsync(new BatchGetPartitionRequest().withCatalogId(catalogId).withDatabaseName(databaseName).withTableName(tableName).withPartitionsToGet(partitionValuesBatch), stats.getBatchGetPartitions().metricsAsyncHandler()));
        }
        GluePartitionConverter converter = new GluePartitionConverter(databaseName, tableName);
        ImmutableList.Builder<Partition> resultsBuilder = ImmutableList.builderWithExpectedSize(partitionNames.size());
        for (Future<BatchGetPartitionResult> future : batchGetPartitionFutures) {
            future.get().getPartitions().stream().map(converter).forEach(resultsBuilder::add);
        }
        return resultsBuilder.build();
    } catch (AmazonServiceException | InterruptedException | ExecutionException e) {
        if (e instanceof InterruptedException) {
            Thread.currentThread().interrupt();
        }
        throw new PrestoException(HIVE_METASTORE_ERROR, e);
    }
}
Also used : Partition(com.facebook.presto.hive.metastore.Partition) BatchGetPartitionRequest(com.amazonaws.services.glue.model.BatchGetPartitionRequest) ImmutableList(com.google.common.collect.ImmutableList) ArrayList(java.util.ArrayList) PrestoException(com.facebook.presto.spi.PrestoException) BatchGetPartitionResult(com.amazonaws.services.glue.model.BatchGetPartitionResult) GluePartitionConverter(com.facebook.presto.hive.metastore.glue.converter.GlueToPrestoConverter.GluePartitionConverter) PartitionValueList(com.amazonaws.services.glue.model.PartitionValueList) AmazonServiceException(com.amazonaws.AmazonServiceException) Future(java.util.concurrent.Future) ExecutionException(java.util.concurrent.ExecutionException)

Aggregations

AmazonServiceException (com.amazonaws.AmazonServiceException)1 BatchGetPartitionRequest (com.amazonaws.services.glue.model.BatchGetPartitionRequest)1 BatchGetPartitionResult (com.amazonaws.services.glue.model.BatchGetPartitionResult)1 PartitionValueList (com.amazonaws.services.glue.model.PartitionValueList)1 Partition (com.facebook.presto.hive.metastore.Partition)1 GluePartitionConverter (com.facebook.presto.hive.metastore.glue.converter.GlueToPrestoConverter.GluePartitionConverter)1 PrestoException (com.facebook.presto.spi.PrestoException)1 ImmutableList (com.google.common.collect.ImmutableList)1 ArrayList (java.util.ArrayList)1 ExecutionException (java.util.concurrent.ExecutionException)1 Future (java.util.concurrent.Future)1