Search in sources :

Example 1 with GetTablesResult

use of com.amazonaws.services.glue.model.GetTablesResult in project presto by prestodb.

the class GlueHiveMetastore method getAllTables.

@Override
public Optional<List<String>> getAllTables(MetastoreContext metastoreContext, String databaseName) {
    try {
        List<String> tableNames = new ArrayList<>();
        GetTablesRequest request = new GetTablesRequest().withCatalogId(catalogId).withDatabaseName(databaseName);
        do {
            GetTablesResult result = stats.getGetTables().record(() -> glueClient.getTables(request));
            request.setNextToken(result.getNextToken());
            result.getTableList().forEach(table -> tableNames.add(table.getName()));
        } while (request.getNextToken() != null);
        return Optional.of(tableNames);
    } catch (EntityNotFoundException e) {
        // database does not exist
        return Optional.empty();
    } catch (AmazonServiceException e) {
        throw new PrestoException(HIVE_METASTORE_ERROR, e);
    }
}
Also used : GetTablesResult(com.amazonaws.services.glue.model.GetTablesResult) ArrayList(java.util.ArrayList) AmazonServiceException(com.amazonaws.AmazonServiceException) PrestoException(com.facebook.presto.spi.PrestoException) EntityNotFoundException(com.amazonaws.services.glue.model.EntityNotFoundException) GetTablesRequest(com.amazonaws.services.glue.model.GetTablesRequest)

Example 2 with GetTablesResult

use of com.amazonaws.services.glue.model.GetTablesResult in project presto by prestodb.

the class GlueHiveMetastore method getAllViews.

@Override
public Optional<List<String>> getAllViews(MetastoreContext metastoreContext, String databaseName) {
    try {
        List<String> views = new ArrayList<>();
        GetTablesRequest request = new GetTablesRequest().withCatalogId(catalogId).withDatabaseName(databaseName);
        do {
            GetTablesResult result = stats.getGetTables().record(() -> glueClient.getTables(request));
            request.setNextToken(result.getNextToken());
            result.getTableList().stream().filter(table -> VIRTUAL_VIEW.name().equals(table.getTableType())).forEach(table -> views.add(table.getName()));
        } while (request.getNextToken() != null);
        return Optional.of(views);
    } catch (EntityNotFoundException e) {
        // database does not exist
        return Optional.empty();
    } catch (AmazonServiceException e) {
        throw new PrestoException(HIVE_METASTORE_ERROR, e);
    }
}
Also used : AWSStaticCredentialsProvider(com.amazonaws.auth.AWSStaticCredentialsProvider) SchemaAlreadyExistsException(com.facebook.presto.hive.SchemaAlreadyExistsException) MetastoreUtil.makePartName(com.facebook.presto.hive.metastore.MetastoreUtil.makePartName) UnaryOperator.identity(java.util.function.UnaryOperator.identity) PartitionWithStatistics(com.facebook.presto.hive.metastore.PartitionWithStatistics) PrestoPrincipal(com.facebook.presto.spi.security.PrestoPrincipal) MetastoreContext(com.facebook.presto.hive.metastore.MetastoreContext) RequestMetricCollector(com.amazonaws.metrics.RequestMetricCollector) DeleteTableRequest(com.amazonaws.services.glue.model.DeleteTableRequest) Future(java.util.concurrent.Future) MetastoreUtil.getHiveBasicStatistics(com.facebook.presto.hive.metastore.MetastoreUtil.getHiveBasicStatistics) GetDatabasesResult(com.amazonaws.services.glue.model.GetDatabasesResult) ExtendedHiveMetastore(com.facebook.presto.hive.metastore.ExtendedHiveMetastore) Map(java.util.Map) TableAlreadyExistsException(com.facebook.presto.hive.TableAlreadyExistsException) GlueInputConverter(com.facebook.presto.hive.metastore.glue.converter.GlueInputConverter) BatchCreatePartitionRequest(com.amazonaws.services.glue.model.BatchCreatePartitionRequest) GetTablesResult(com.amazonaws.services.glue.model.GetTablesResult) AmazonServiceException(com.amazonaws.AmazonServiceException) PartitionNameWithVersion(com.facebook.presto.hive.metastore.PartitionNameWithVersion) PrincipalPrivileges(com.facebook.presto.hive.metastore.PrincipalPrivileges) DeletePartitionRequest(com.amazonaws.services.glue.model.DeletePartitionRequest) Set(java.util.Set) DatabaseInput(com.amazonaws.services.glue.model.DatabaseInput) TableInput(com.amazonaws.services.glue.model.TableInput) UpdateTableRequest(com.amazonaws.services.glue.model.UpdateTableRequest) HivePrivilegeInfo(com.facebook.presto.hive.metastore.HivePrivilegeInfo) PartitionInput(com.amazonaws.services.glue.model.PartitionInput) EntityNotFoundException(com.amazonaws.services.glue.model.EntityNotFoundException) AWSGlueAsync(com.amazonaws.services.glue.AWSGlueAsync) MetastoreUtil.verifyCanDropColumn(com.facebook.presto.hive.metastore.MetastoreUtil.verifyCanDropColumn) GetPartitionsRequest(com.amazonaws.services.glue.model.GetPartitionsRequest) Segment(com.amazonaws.services.glue.model.Segment) GetPartitionResult(com.amazonaws.services.glue.model.GetPartitionResult) Table(com.facebook.presto.hive.metastore.Table) Database(com.facebook.presto.hive.metastore.Database) Flatten(org.weakref.jmx.Flatten) Strings.isNullOrEmpty(com.google.common.base.Strings.isNullOrEmpty) MetastoreUtil.convertPredicateToParts(com.facebook.presto.hive.metastore.MetastoreUtil.convertPredicateToParts) GlueInputConverter.convertColumn(com.facebook.presto.hive.metastore.glue.converter.GlueInputConverter.convertColumn) Regions(com.amazonaws.regions.Regions) ArrayList(java.util.ArrayList) MetastoreUtil(com.facebook.presto.hive.metastore.MetastoreUtil) Comparators.lexicographical(com.google.common.collect.Comparators.lexicographical) Lists(com.google.common.collect.Lists) AlreadyExistsException(com.amazonaws.services.glue.model.AlreadyExistsException) Managed(org.weakref.jmx.Managed) GlueExpressionUtil.buildGlueExpression(com.facebook.presto.hive.metastore.glue.GlueExpressionUtil.buildGlueExpression) AWSCredentialsProvider(com.amazonaws.auth.AWSCredentialsProvider) CreateTableRequest(com.amazonaws.services.glue.model.CreateTableRequest) CreateDatabaseRequest(com.amazonaws.services.glue.model.CreateDatabaseRequest) Nullable(javax.annotation.Nullable) Executor(java.util.concurrent.Executor) PartitionValueList(com.amazonaws.services.glue.model.PartitionValueList) GetTableResult(com.amazonaws.services.glue.model.GetTableResult) Domain(com.facebook.presto.common.predicate.Domain) GlueToPrestoConverter(com.facebook.presto.hive.metastore.glue.converter.GlueToPrestoConverter) ExecutionException(java.util.concurrent.ExecutionException) GlueToPrestoConverter.mappedCopy(com.facebook.presto.hive.metastore.glue.converter.GlueToPrestoConverter.mappedCopy) ClientConfiguration(com.amazonaws.ClientConfiguration) ColumnStatisticType(com.facebook.presto.spi.statistics.ColumnStatisticType) PartitionStatistics(com.facebook.presto.hive.metastore.PartitionStatistics) GetPartitionsResult(com.amazonaws.services.glue.model.GetPartitionsResult) HdfsEnvironment(com.facebook.presto.hive.HdfsEnvironment) PartitionNotFoundException(com.facebook.presto.hive.PartitionNotFoundException) GetDatabaseRequest(com.amazonaws.services.glue.model.GetDatabaseRequest) CompletionService(java.util.concurrent.CompletionService) Duration(io.airlift.units.Duration) GluePartitionConverter(com.facebook.presto.hive.metastore.glue.converter.GlueToPrestoConverter.GluePartitionConverter) GlueInputConverter.toTableInput(com.facebook.presto.hive.metastore.glue.converter.GlueInputConverter.toTableInput) Preconditions.checkArgument(com.google.common.base.Preconditions.checkArgument) SchemaTableName(com.facebook.presto.spi.SchemaTableName) HIVE_METASTORE_ERROR(com.facebook.presto.hive.HiveErrorCode.HIVE_METASTORE_ERROR) GetDatabasesRequest(com.amazonaws.services.glue.model.GetDatabasesRequest) ColumnNotFoundException(com.facebook.presto.spi.ColumnNotFoundException) Collectors.toMap(java.util.stream.Collectors.toMap) SchemaNotFoundException(com.facebook.presto.spi.SchemaNotFoundException) MANAGED_TABLE(com.facebook.presto.hive.metastore.PrestoTableType.MANAGED_TABLE) Path(org.apache.hadoop.fs.Path) HdfsContext(com.facebook.presto.hive.HdfsContext) ImmutableSet(com.google.common.collect.ImmutableSet) AWSGlueAsyncClientBuilder(com.amazonaws.services.glue.AWSGlueAsyncClientBuilder) ImmutableMap(com.google.common.collect.ImmutableMap) STSAssumeRoleSessionCredentialsProvider(com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider) List(java.util.List) RoleGrant(com.facebook.presto.spi.security.RoleGrant) NOT_SUPPORTED(com.facebook.presto.spi.StandardErrorCode.NOT_SUPPORTED) GetTableRequest(com.amazonaws.services.glue.model.GetTableRequest) PartitionError(com.amazonaws.services.glue.model.PartitionError) Entry(java.util.Map.Entry) MetastoreUtil.toPartitionValues(com.facebook.presto.hive.metastore.MetastoreUtil.toPartitionValues) Optional(java.util.Optional) UpdateDatabaseRequest(com.amazonaws.services.glue.model.UpdateDatabaseRequest) ExecutorCompletionService(java.util.concurrent.ExecutorCompletionService) Logger(com.facebook.airlift.log.Logger) Column(com.facebook.presto.hive.metastore.Column) HiveType(com.facebook.presto.hive.HiveType) HIVE_PARTITION_DROPPED_DURING_QUERY(com.facebook.presto.hive.HiveErrorCode.HIVE_PARTITION_DROPPED_DURING_QUERY) PrestoException(com.facebook.presto.spi.PrestoException) Function(java.util.function.Function) Partition(com.facebook.presto.hive.metastore.Partition) Inject(javax.inject.Inject) EndpointConfiguration(com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration) GetPartitionRequest(com.amazonaws.services.glue.model.GetPartitionRequest) ImmutableList(com.google.common.collect.ImmutableList) ALREADY_EXISTS(com.facebook.presto.spi.StandardErrorCode.ALREADY_EXISTS) Objects.requireNonNull(java.util.Objects.requireNonNull) DeleteDatabaseRequest(com.amazonaws.services.glue.model.DeleteDatabaseRequest) Comparator.comparing(java.util.Comparator.comparing) MetastoreUtil.updateStatisticsParameters(com.facebook.presto.hive.metastore.MetastoreUtil.updateStatisticsParameters) Type(com.facebook.presto.common.type.Type) BatchGetPartitionRequest(com.amazonaws.services.glue.model.BatchGetPartitionRequest) USER(com.facebook.presto.spi.security.PrincipalType.USER) BatchCreatePartitionResult(com.amazonaws.services.glue.model.BatchCreatePartitionResult) BasicAWSCredentials(com.amazonaws.auth.BasicAWSCredentials) BatchGetPartitionResult(com.amazonaws.services.glue.model.BatchGetPartitionResult) ErrorDetail(com.amazonaws.services.glue.model.ErrorDetail) MetastoreUtil.createDirectory(com.facebook.presto.hive.metastore.MetastoreUtil.createDirectory) VIRTUAL_VIEW(com.facebook.presto.hive.metastore.PrestoTableType.VIRTUAL_VIEW) ConnectorIdentity(com.facebook.presto.spi.security.ConnectorIdentity) TableNotFoundException(com.facebook.presto.spi.TableNotFoundException) GetDatabaseResult(com.amazonaws.services.glue.model.GetDatabaseResult) GetTablesRequest(com.amazonaws.services.glue.model.GetTablesRequest) UpdatePartitionRequest(com.amazonaws.services.glue.model.UpdatePartitionRequest) Region(com.amazonaws.regions.Region) Comparator(java.util.Comparator) GetTablesResult(com.amazonaws.services.glue.model.GetTablesResult) ArrayList(java.util.ArrayList) AmazonServiceException(com.amazonaws.AmazonServiceException) PrestoException(com.facebook.presto.spi.PrestoException) EntityNotFoundException(com.amazonaws.services.glue.model.EntityNotFoundException) GetTablesRequest(com.amazonaws.services.glue.model.GetTablesRequest)

Example 3 with GetTablesResult

use of com.amazonaws.services.glue.model.GetTablesResult in project alluxio by Alluxio.

the class GlueDatabase method getTableNames.

@Override
public List<String> getTableNames() throws IOException {
    try {
        String nextToken = null;
        List<String> tableNames = new ArrayList<>();
        do {
            GetTablesRequest tablesRequest = new GetTablesRequest().withCatalogId(mGlueConfiguration.get(Property.CATALOG_ID)).withDatabaseName(mGlueDbName).withNextToken(nextToken);
            GetTablesResult tablesResult = mGlueClient.getTables(tablesRequest);
            tablesResult.getTableList().forEach(table -> tableNames.add(table.getName()));
            nextToken = tablesResult.getNextToken();
        } while (nextToken != null);
        return tableNames;
    } catch (EntityNotFoundException e) {
        throw new IOException("Failed to get glue tables: " + e.getMessage() + " in Database: " + mGlueDbName + "; with Catalog ID: " + mGlueConfiguration.get(Property.CATALOG_ID) + ".", e);
    }
}
Also used : GetTablesResult(com.amazonaws.services.glue.model.GetTablesResult) ArrayList(java.util.ArrayList) EntityNotFoundException(com.amazonaws.services.glue.model.EntityNotFoundException) IOException(java.io.IOException) GetTablesRequest(com.amazonaws.services.glue.model.GetTablesRequest)

Aggregations

EntityNotFoundException (com.amazonaws.services.glue.model.EntityNotFoundException)3 GetTablesRequest (com.amazonaws.services.glue.model.GetTablesRequest)3 GetTablesResult (com.amazonaws.services.glue.model.GetTablesResult)3 ArrayList (java.util.ArrayList)3 AmazonServiceException (com.amazonaws.AmazonServiceException)2 PrestoException (com.facebook.presto.spi.PrestoException)2 ClientConfiguration (com.amazonaws.ClientConfiguration)1 AWSCredentialsProvider (com.amazonaws.auth.AWSCredentialsProvider)1 AWSStaticCredentialsProvider (com.amazonaws.auth.AWSStaticCredentialsProvider)1 BasicAWSCredentials (com.amazonaws.auth.BasicAWSCredentials)1 STSAssumeRoleSessionCredentialsProvider (com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider)1 EndpointConfiguration (com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration)1 RequestMetricCollector (com.amazonaws.metrics.RequestMetricCollector)1 Region (com.amazonaws.regions.Region)1 Regions (com.amazonaws.regions.Regions)1 AWSGlueAsync (com.amazonaws.services.glue.AWSGlueAsync)1 AWSGlueAsyncClientBuilder (com.amazonaws.services.glue.AWSGlueAsyncClientBuilder)1 AlreadyExistsException (com.amazonaws.services.glue.model.AlreadyExistsException)1 BatchCreatePartitionRequest (com.amazonaws.services.glue.model.BatchCreatePartitionRequest)1 BatchCreatePartitionResult (com.amazonaws.services.glue.model.BatchCreatePartitionResult)1