Search in sources :

Example 86 with TrinoException

use of io.trino.spi.TrinoException in project trino by trinodb.

the class AlluxioHiveMetastore method getPartitionNamesByFilter.

@Override
public Optional<List<String>> getPartitionNamesByFilter(String databaseName, String tableName, List<String> columnNames, TupleDomain<String> partitionKeysFilter) {
    try {
        List<PartitionInfo> partitionInfos = ProtoUtils.toPartitionInfoList(client.readTable(databaseName, tableName, Constraint.getDefaultInstance()));
        List<String> partitionNames = partitionInfos.stream().map(PartitionInfo::getPartitionName).collect(Collectors.toList());
        return Optional.of(partitionNames);
    } catch (AlluxioStatusException e) {
        throw new TrinoException(HIVE_METASTORE_ERROR, e);
    }
}
Also used : AlluxioStatusException(alluxio.exception.status.AlluxioStatusException) TrinoException(io.trino.spi.TrinoException) PartitionInfo(alluxio.grpc.table.layout.hive.PartitionInfo)

Example 87 with TrinoException

use of io.trino.spi.TrinoException in project trino by trinodb.

the class GlueHiveMetastore method batchGetPartition.

private List<Partition> batchGetPartition(Table table, List<String> partitionNames) {
    try {
        List<PartitionValueList> pendingPartitions = partitionNames.stream().map(partitionName -> new PartitionValueList().withValues(toPartitionValues(partitionName))).collect(toCollection(ArrayList::new));
        ImmutableList.Builder<Partition> resultsBuilder = ImmutableList.builderWithExpectedSize(partitionNames.size());
        // Reuse immutable field instances opportunistically between partitions
        GluePartitionConverter converter = new GluePartitionConverter(table);
        while (!pendingPartitions.isEmpty()) {
            List<Future<BatchGetPartitionResult>> batchGetPartitionFutures = new ArrayList<>();
            for (List<PartitionValueList> partitions : Lists.partition(pendingPartitions, BATCH_GET_PARTITION_MAX_PAGE_SIZE)) {
                long startTimestamp = System.currentTimeMillis();
                batchGetPartitionFutures.add(glueClient.batchGetPartitionAsync(new BatchGetPartitionRequest().withCatalogId(catalogId).withDatabaseName(table.getDatabaseName()).withTableName(table.getTableName()).withPartitionsToGet(partitions), new StatsRecordingAsyncHandler(stats.getGetPartitions(), startTimestamp)));
            }
            pendingPartitions.clear();
            for (Future<BatchGetPartitionResult> future : batchGetPartitionFutures) {
                BatchGetPartitionResult batchGetPartitionResult = future.get();
                List<com.amazonaws.services.glue.model.Partition> partitions = batchGetPartitionResult.getPartitions();
                List<PartitionValueList> unprocessedKeys = batchGetPartitionResult.getUnprocessedKeys();
                // In the unlikely scenario where batchGetPartition call cannot make progress on retrieving partitions, avoid infinite loop
                if (partitions.isEmpty()) {
                    verify(!unprocessedKeys.isEmpty(), "Empty unprocessedKeys for non-empty BatchGetPartitionRequest and empty partitions result");
                    throw new TrinoException(HIVE_METASTORE_ERROR, "Cannot make progress retrieving partitions. Unable to retrieve partitions: " + unprocessedKeys);
                }
                partitions.stream().map(converter).forEach(resultsBuilder::add);
                pendingPartitions.addAll(unprocessedKeys);
            }
        }
        return resultsBuilder.build();
    } catch (AmazonServiceException | InterruptedException | ExecutionException e) {
        if (e instanceof InterruptedException) {
            Thread.currentThread().interrupt();
        }
        throw new TrinoException(HIVE_METASTORE_ERROR, e);
    }
}
Also used : ThriftMetastoreUtil.updateStatisticsParameters(io.trino.plugin.hive.metastore.thrift.ThriftMetastoreUtil.updateStatisticsParameters) AWSStaticCredentialsProvider(com.amazonaws.auth.AWSStaticCredentialsProvider) UnaryOperator.identity(java.util.function.UnaryOperator.identity) DefaultAWSCredentialsProviderChain(com.amazonaws.auth.DefaultAWSCredentialsProviderChain) USER(io.trino.spi.security.PrincipalType.USER) RequestMetricCollector(com.amazonaws.metrics.RequestMetricCollector) DeleteTableRequest(com.amazonaws.services.glue.model.DeleteTableRequest) ColumnStatisticType(io.trino.spi.statistics.ColumnStatisticType) NOT_SUPPORTED(io.trino.spi.StandardErrorCode.NOT_SUPPORTED) Future(java.util.concurrent.Future) GetDatabasesResult(com.amazonaws.services.glue.model.GetDatabasesResult) TableNotFoundException(io.trino.spi.connector.TableNotFoundException) Column(io.trino.plugin.hive.metastore.Column) Map(java.util.Map) PartitionWithStatistics(io.trino.plugin.hive.metastore.PartitionWithStatistics) BatchCreatePartitionRequest(com.amazonaws.services.glue.model.BatchCreatePartitionRequest) GetTablesResult(com.amazonaws.services.glue.model.GetTablesResult) RequestHandler2(com.amazonaws.handlers.RequestHandler2) AcidTransaction(io.trino.plugin.hive.acid.AcidTransaction) HdfsEnvironment(io.trino.plugin.hive.HdfsEnvironment) Table(io.trino.plugin.hive.metastore.Table) ConnectorIdentity(io.trino.spi.security.ConnectorIdentity) AmazonServiceException(com.amazonaws.AmazonServiceException) GlueToTrinoConverter.mappedCopy(io.trino.plugin.hive.metastore.glue.converter.GlueToTrinoConverter.mappedCopy) ImmutableList.toImmutableList(com.google.common.collect.ImmutableList.toImmutableList) DeletePartitionRequest(com.amazonaws.services.glue.model.DeletePartitionRequest) Set(java.util.Set) DatabaseInput(com.amazonaws.services.glue.model.DatabaseInput) TableInput(com.amazonaws.services.glue.model.TableInput) UpdateTableRequest(com.amazonaws.services.glue.model.UpdateTableRequest) MANAGED_TABLE(org.apache.hadoop.hive.metastore.TableType.MANAGED_TABLE) SchemaTableName(io.trino.spi.connector.SchemaTableName) ImmutableMap.toImmutableMap(com.google.common.collect.ImmutableMap.toImmutableMap) BatchUpdatePartitionRequestEntry(com.amazonaws.services.glue.model.BatchUpdatePartitionRequestEntry) PartitionInput(com.amazonaws.services.glue.model.PartitionInput) GlueInputConverter.convertPartition(io.trino.plugin.hive.metastore.glue.converter.GlueInputConverter.convertPartition) EntityNotFoundException(com.amazonaws.services.glue.model.EntityNotFoundException) AWSGlueAsync(com.amazonaws.services.glue.AWSGlueAsync) Partition(io.trino.plugin.hive.metastore.Partition) GetPartitionsRequest(com.amazonaws.services.glue.model.GetPartitionsRequest) Segment(com.amazonaws.services.glue.model.Segment) PartitionStatistics(io.trino.plugin.hive.PartitionStatistics) HivePrincipal(io.trino.plugin.hive.metastore.HivePrincipal) HiveUtil(io.trino.plugin.hive.util.HiveUtil) Iterables(com.google.common.collect.Iterables) GetPartitionResult(com.amazonaws.services.glue.model.GetPartitionResult) Strings.isNullOrEmpty(com.google.common.base.Strings.isNullOrEmpty) PartitionNotFoundException(io.trino.plugin.hive.PartitionNotFoundException) ColumnNotFoundException(io.trino.spi.connector.ColumnNotFoundException) ArrayList(java.util.ArrayList) HiveType(io.trino.plugin.hive.HiveType) OptionalLong(java.util.OptionalLong) Comparators.lexicographical(com.google.common.collect.Comparators.lexicographical) Lists(com.google.common.collect.Lists) HiveMetastore(io.trino.plugin.hive.metastore.HiveMetastore) AlreadyExistsException(com.amazonaws.services.glue.model.AlreadyExistsException) AWSCredentialsProvider(com.amazonaws.auth.AWSCredentialsProvider) CreateTableRequest(com.amazonaws.services.glue.model.CreateTableRequest) SchemaAlreadyExistsException(io.trino.plugin.hive.SchemaAlreadyExistsException) ImmutableSet.toImmutableSet(com.google.common.collect.ImmutableSet.toImmutableSet) CreateDatabaseRequest(com.amazonaws.services.glue.model.CreateDatabaseRequest) HiveWriteUtils(io.trino.plugin.hive.util.HiveWriteUtils) Nullable(javax.annotation.Nullable) Executor(java.util.concurrent.Executor) MoreFutures(io.airlift.concurrent.MoreFutures) PartitionValueList(com.amazonaws.services.glue.model.PartitionValueList) GetTableResult(com.amazonaws.services.glue.model.GetTableResult) RoleGrant(io.trino.spi.security.RoleGrant) ExecutionException(java.util.concurrent.ExecutionException) ClientConfiguration(com.amazonaws.ClientConfiguration) AwsSdkUtil.getPaginatedResults(io.trino.plugin.hive.metastore.glue.AwsSdkUtil.getPaginatedResults) AsyncHandler(com.amazonaws.handlers.AsyncHandler) GetPartitionsResult(com.amazonaws.services.glue.model.GetPartitionsResult) HivePrivilege(io.trino.plugin.hive.metastore.HivePrivilegeInfo.HivePrivilege) ThriftMetastoreUtil.getHiveBasicStatistics(io.trino.plugin.hive.metastore.thrift.ThriftMetastoreUtil.getHiveBasicStatistics) MetastoreUtil.makePartitionName(io.trino.plugin.hive.metastore.MetastoreUtil.makePartitionName) HiveUtil.toPartitionValues(io.trino.plugin.hive.util.HiveUtil.toPartitionValues) Database(io.trino.plugin.hive.metastore.Database) GetDatabaseRequest(com.amazonaws.services.glue.model.GetDatabaseRequest) SchemaNotFoundException(io.trino.spi.connector.SchemaNotFoundException) CompletionService(java.util.concurrent.CompletionService) Preconditions.checkArgument(com.google.common.base.Preconditions.checkArgument) GetDatabasesRequest(com.amazonaws.services.glue.model.GetDatabasesRequest) Collectors.toMap(java.util.stream.Collectors.toMap) ALREADY_EXISTS(io.trino.spi.StandardErrorCode.ALREADY_EXISTS) Path(org.apache.hadoop.fs.Path) BatchUpdatePartitionResult(com.amazonaws.services.glue.model.BatchUpdatePartitionResult) ImmutableSet(com.google.common.collect.ImmutableSet) AWSGlueAsyncClientBuilder(com.amazonaws.services.glue.AWSGlueAsyncClientBuilder) ImmutableMap(com.google.common.collect.ImmutableMap) Predicate(java.util.function.Predicate) GluePartitionConverter(io.trino.plugin.hive.metastore.glue.converter.GlueToTrinoConverter.GluePartitionConverter) TableAlreadyExistsException(io.trino.plugin.hive.TableAlreadyExistsException) TrinoException(io.trino.spi.TrinoException) STSAssumeRoleSessionCredentialsProvider(com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider) String.format(java.lang.String.format) HdfsContext(io.trino.plugin.hive.HdfsEnvironment.HdfsContext) List(java.util.List) GetTableRequest(com.amazonaws.services.glue.model.GetTableRequest) PartitionError(com.amazonaws.services.glue.model.PartitionError) Entry(java.util.Map.Entry) Optional(java.util.Optional) HivePrivilegeInfo(io.trino.plugin.hive.metastore.HivePrivilegeInfo) UpdateDatabaseRequest(com.amazonaws.services.glue.model.UpdateDatabaseRequest) ExecutorCompletionService(java.util.concurrent.ExecutorCompletionService) Logger(io.airlift.log.Logger) Type(io.trino.spi.type.Type) Function(java.util.function.Function) Inject(javax.inject.Inject) EndpointConfiguration(com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration) GetPartitionRequest(com.amazonaws.services.glue.model.GetPartitionRequest) Collectors.toCollection(java.util.stream.Collectors.toCollection) HiveColumnStatistics(io.trino.plugin.hive.metastore.HiveColumnStatistics) ImmutableList(com.google.common.collect.ImmutableList) Verify.verify(com.google.common.base.Verify.verify) HIVE_METASTORE_ERROR(io.trino.plugin.hive.HiveErrorCode.HIVE_METASTORE_ERROR) Objects.requireNonNull(java.util.Objects.requireNonNull) GlueInputConverter(io.trino.plugin.hive.metastore.glue.converter.GlueInputConverter) DeleteDatabaseRequest(com.amazonaws.services.glue.model.DeleteDatabaseRequest) Comparator.comparing(java.util.Comparator.comparing) VIRTUAL_VIEW(org.apache.hadoop.hive.metastore.TableType.VIRTUAL_VIEW) AwsCurrentRegionHolder.getCurrentRegionFromEC2Metadata(io.trino.plugin.hive.aws.AwsCurrentRegionHolder.getCurrentRegionFromEC2Metadata) BatchGetPartitionRequest(com.amazonaws.services.glue.model.BatchGetPartitionRequest) AmazonWebServiceRequest(com.amazonaws.AmazonWebServiceRequest) BatchCreatePartitionResult(com.amazonaws.services.glue.model.BatchCreatePartitionResult) BasicAWSCredentials(com.amazonaws.auth.BasicAWSCredentials) BatchGetPartitionResult(com.amazonaws.services.glue.model.BatchGetPartitionResult) ErrorDetail(com.amazonaws.services.glue.model.ErrorDetail) TupleDomain(io.trino.spi.predicate.TupleDomain) BatchUpdatePartitionRequest(com.amazonaws.services.glue.model.BatchUpdatePartitionRequest) GlueToTrinoConverter(io.trino.plugin.hive.metastore.glue.converter.GlueToTrinoConverter) GetDatabaseResult(com.amazonaws.services.glue.model.GetDatabaseResult) GetTablesRequest(com.amazonaws.services.glue.model.GetTablesRequest) MetastoreUtil.verifyCanDropColumn(io.trino.plugin.hive.metastore.MetastoreUtil.verifyCanDropColumn) UpdatePartitionRequest(com.amazonaws.services.glue.model.UpdatePartitionRequest) PrincipalPrivileges(io.trino.plugin.hive.metastore.PrincipalPrivileges) Comparator(java.util.Comparator) GlueInputConverter.convertPartition(io.trino.plugin.hive.metastore.glue.converter.GlueInputConverter.convertPartition) Partition(io.trino.plugin.hive.metastore.Partition) BatchGetPartitionRequest(com.amazonaws.services.glue.model.BatchGetPartitionRequest) ImmutableList.toImmutableList(com.google.common.collect.ImmutableList.toImmutableList) ImmutableList(com.google.common.collect.ImmutableList) ArrayList(java.util.ArrayList) BatchGetPartitionResult(com.amazonaws.services.glue.model.BatchGetPartitionResult) GluePartitionConverter(io.trino.plugin.hive.metastore.glue.converter.GlueToTrinoConverter.GluePartitionConverter) PartitionValueList(com.amazonaws.services.glue.model.PartitionValueList) AmazonServiceException(com.amazonaws.AmazonServiceException) Future(java.util.concurrent.Future) TrinoException(io.trino.spi.TrinoException) ExecutionException(java.util.concurrent.ExecutionException)

Example 88 with TrinoException

use of io.trino.spi.TrinoException in project trino by trinodb.

the class GlueHiveMetastore method renameDatabase.

@Override
public void renameDatabase(String databaseName, String newDatabaseName) {
    try {
        Database database = getDatabase(databaseName).orElseThrow(() -> new SchemaNotFoundException(databaseName));
        DatabaseInput renamedDatabase = GlueInputConverter.convertDatabase(database).withName(newDatabaseName);
        stats.getUpdateDatabase().call(() -> glueClient.updateDatabase(new UpdateDatabaseRequest().withCatalogId(catalogId).withName(databaseName).withDatabaseInput(renamedDatabase)));
    } catch (AmazonServiceException e) {
        throw new TrinoException(HIVE_METASTORE_ERROR, e);
    }
}
Also used : UpdateDatabaseRequest(com.amazonaws.services.glue.model.UpdateDatabaseRequest) Database(io.trino.plugin.hive.metastore.Database) AmazonServiceException(com.amazonaws.AmazonServiceException) TrinoException(io.trino.spi.TrinoException) DatabaseInput(com.amazonaws.services.glue.model.DatabaseInput) SchemaNotFoundException(io.trino.spi.connector.SchemaNotFoundException)

Example 89 with TrinoException

use of io.trino.spi.TrinoException in project trino by trinodb.

the class GlueHiveMetastore method propagatePartitionErrorToTrinoException.

private static void propagatePartitionErrorToTrinoException(String databaseName, String tableName, List<PartitionError> partitionErrors) {
    if (partitionErrors != null && !partitionErrors.isEmpty()) {
        ErrorDetail errorDetail = partitionErrors.get(0).getErrorDetail();
        String glueExceptionCode = errorDetail.getErrorCode();
        switch(glueExceptionCode) {
            case "AlreadyExistsException":
                throw new TrinoException(ALREADY_EXISTS, errorDetail.getErrorMessage());
            case "EntityNotFoundException":
                throw new TableNotFoundException(new SchemaTableName(databaseName, tableName), errorDetail.getErrorMessage());
            default:
                throw new TrinoException(HIVE_METASTORE_ERROR, errorDetail.getErrorCode() + ": " + errorDetail.getErrorMessage());
        }
    }
}
Also used : ErrorDetail(com.amazonaws.services.glue.model.ErrorDetail) TableNotFoundException(io.trino.spi.connector.TableNotFoundException) TrinoException(io.trino.spi.TrinoException) SchemaTableName(io.trino.spi.connector.SchemaTableName)

Example 90 with TrinoException

use of io.trino.spi.TrinoException in project trino by trinodb.

the class GlueHiveMetastore method replaceTable.

@Override
public void replaceTable(String databaseName, String tableName, Table newTable, PrincipalPrivileges principalPrivileges) {
    try {
        TableInput newTableInput = GlueInputConverter.convertTable(newTable);
        stats.getUpdateTable().call(() -> glueClient.updateTable(new UpdateTableRequest().withCatalogId(catalogId).withDatabaseName(databaseName).withTableInput(newTableInput)));
    } catch (EntityNotFoundException e) {
        throw new TableNotFoundException(new SchemaTableName(databaseName, tableName));
    } catch (AmazonServiceException e) {
        throw new TrinoException(HIVE_METASTORE_ERROR, e);
    }
}
Also used : TableInput(com.amazonaws.services.glue.model.TableInput) TableNotFoundException(io.trino.spi.connector.TableNotFoundException) UpdateTableRequest(com.amazonaws.services.glue.model.UpdateTableRequest) AmazonServiceException(com.amazonaws.AmazonServiceException) TrinoException(io.trino.spi.TrinoException) EntityNotFoundException(com.amazonaws.services.glue.model.EntityNotFoundException) SchemaTableName(io.trino.spi.connector.SchemaTableName)

Aggregations

TrinoException (io.trino.spi.TrinoException)623 IOException (java.io.IOException)151 ImmutableList (com.google.common.collect.ImmutableList)105 List (java.util.List)100 Type (io.trino.spi.type.Type)93 ImmutableList.toImmutableList (com.google.common.collect.ImmutableList.toImmutableList)90 SchemaTableName (io.trino.spi.connector.SchemaTableName)83 Path (org.apache.hadoop.fs.Path)83 Optional (java.util.Optional)79 ArrayList (java.util.ArrayList)77 Map (java.util.Map)76 ImmutableMap (com.google.common.collect.ImmutableMap)70 Objects.requireNonNull (java.util.Objects.requireNonNull)69 ConnectorSession (io.trino.spi.connector.ConnectorSession)63 TableNotFoundException (io.trino.spi.connector.TableNotFoundException)62 ImmutableSet (com.google.common.collect.ImmutableSet)56 VarcharType (io.trino.spi.type.VarcharType)54 Set (java.util.Set)54 Slice (io.airlift.slice.Slice)53 Table (io.trino.plugin.hive.metastore.Table)52