Search in sources :

Example 6 with EntityNotFoundException

use of com.amazonaws.services.glue.model.EntityNotFoundException in project alluxio by Alluxio.

the class GlueDatabase method getDatabaseInfo.

@Override
public DatabaseInfo getDatabaseInfo() throws IOException {
    try {
        GetDatabaseRequest dbRequest = new GetDatabaseRequest().withCatalogId(mGlueConfiguration.get(Property.CATALOG_ID)).withName(mGlueDbName);
        GetDatabaseResult dbResult = mGlueClient.getDatabase(dbRequest);
        Database glueDatabase = dbResult.getDatabase();
        // Glue database location, description and parameters could be null
        String glueDbLocation = glueDatabase.getLocationUri() == null ? "" : glueDatabase.getLocationUri();
        String glueDbDescription = glueDatabase.getDescription() == null ? "" : glueDatabase.getDescription();
        Map<String, String> glueParameters = new HashMap<>();
        if (glueDatabase.getParameters() != null) {
            glueParameters.putAll(glueDatabase.getParameters());
        }
        return new DatabaseInfo(glueDbLocation, mOwnerName, mOwnerType, glueDbDescription, glueParameters);
    } catch (EntityNotFoundException e) {
        throw new IOException("Cannot find glue database: " + mGlueDbName + "Catalog ID: " + mGlueConfiguration.get(Property.CATALOG_ID) + ". " + e.getMessage(), e);
    }
}
Also used : GetDatabaseRequest(com.amazonaws.services.glue.model.GetDatabaseRequest) HashMap(java.util.HashMap) DatabaseInfo(alluxio.master.table.DatabaseInfo) UnderDatabase(alluxio.table.common.udb.UnderDatabase) Database(com.amazonaws.services.glue.model.Database) GetDatabaseResult(com.amazonaws.services.glue.model.GetDatabaseResult) EntityNotFoundException(com.amazonaws.services.glue.model.EntityNotFoundException) IOException(java.io.IOException)

Example 7 with EntityNotFoundException

use of com.amazonaws.services.glue.model.EntityNotFoundException in project presto by prestodb.

the class GlueHiveMetastore method alterPartition.

@Override
public void alterPartition(MetastoreContext metastoreContext, String databaseName, String tableName, PartitionWithStatistics partition) {
    try {
        PartitionInput newPartition = GlueInputConverter.convertPartition(partition);
        stats.getUpdatePartition().record(() -> glueClient.updatePartition(new UpdatePartitionRequest().withCatalogId(catalogId).withDatabaseName(databaseName).withTableName(tableName).withPartitionInput(newPartition).withPartitionValueList(partition.getPartition().getValues())));
    } catch (EntityNotFoundException e) {
        throw new PartitionNotFoundException(new SchemaTableName(databaseName, tableName), partition.getPartition().getValues());
    } catch (AmazonServiceException e) {
        throw new PrestoException(HIVE_METASTORE_ERROR, e);
    }
}
Also used : PartitionNotFoundException(com.facebook.presto.hive.PartitionNotFoundException) AmazonServiceException(com.amazonaws.AmazonServiceException) PrestoException(com.facebook.presto.spi.PrestoException) UpdatePartitionRequest(com.amazonaws.services.glue.model.UpdatePartitionRequest) EntityNotFoundException(com.amazonaws.services.glue.model.EntityNotFoundException) SchemaTableName(com.facebook.presto.spi.SchemaTableName) PartitionInput(com.amazonaws.services.glue.model.PartitionInput)

Example 8 with EntityNotFoundException

use of com.amazonaws.services.glue.model.EntityNotFoundException in project presto by prestodb.

the class GlueHiveMetastore method updateTableStatistics.

@Override
public void updateTableStatistics(MetastoreContext metastoreContext, String databaseName, String tableName, Function<PartitionStatistics, PartitionStatistics> update) {
    PartitionStatistics currentStatistics = getTableStatistics(metastoreContext, databaseName, tableName);
    PartitionStatistics updatedStatistics = update.apply(currentStatistics);
    if (!updatedStatistics.getColumnStatistics().isEmpty()) {
        throw new PrestoException(NOT_SUPPORTED, "Glue metastore does not support column level statistics");
    }
    Table table = getTableOrElseThrow(metastoreContext, databaseName, tableName);
    try {
        TableInput tableInput = GlueInputConverter.convertTable(table);
        tableInput.setParameters(updateStatisticsParameters(table.getParameters(), updatedStatistics.getBasicStatistics()));
        UpdateTableRequest request = new UpdateTableRequest().withCatalogId(catalogId).withDatabaseName(databaseName).withTableInput(tableInput);
        stats.getUpdateTable().record(() -> glueClient.updateTable(request));
    } catch (EntityNotFoundException e) {
        throw new TableNotFoundException(new SchemaTableName(databaseName, tableName));
    } catch (AmazonServiceException e) {
        throw new PrestoException(HIVE_METASTORE_ERROR, e);
    }
}
Also used : TableInput(com.amazonaws.services.glue.model.TableInput) GlueInputConverter.toTableInput(com.facebook.presto.hive.metastore.glue.converter.GlueInputConverter.toTableInput) TableNotFoundException(com.facebook.presto.spi.TableNotFoundException) Table(com.facebook.presto.hive.metastore.Table) PartitionStatistics(com.facebook.presto.hive.metastore.PartitionStatistics) UpdateTableRequest(com.amazonaws.services.glue.model.UpdateTableRequest) AmazonServiceException(com.amazonaws.AmazonServiceException) PrestoException(com.facebook.presto.spi.PrestoException) EntityNotFoundException(com.amazonaws.services.glue.model.EntityNotFoundException) SchemaTableName(com.facebook.presto.spi.SchemaTableName)

Example 9 with EntityNotFoundException

use of com.amazonaws.services.glue.model.EntityNotFoundException in project alluxio by Alluxio.

the class GlueDatabase method getTableNames.

@Override
public List<String> getTableNames() throws IOException {
    try {
        String nextToken = null;
        List<String> tableNames = new ArrayList<>();
        do {
            GetTablesRequest tablesRequest = new GetTablesRequest().withCatalogId(mGlueConfiguration.get(Property.CATALOG_ID)).withDatabaseName(mGlueDbName).withNextToken(nextToken);
            GetTablesResult tablesResult = mGlueClient.getTables(tablesRequest);
            tablesResult.getTableList().forEach(table -> tableNames.add(table.getName()));
            nextToken = tablesResult.getNextToken();
        } while (nextToken != null);
        return tableNames;
    } catch (EntityNotFoundException e) {
        throw new IOException("Failed to get glue tables: " + e.getMessage() + " in Database: " + mGlueDbName + "; with Catalog ID: " + mGlueConfiguration.get(Property.CATALOG_ID) + ".", e);
    }
}
Also used : GetTablesResult(com.amazonaws.services.glue.model.GetTablesResult) ArrayList(java.util.ArrayList) EntityNotFoundException(com.amazonaws.services.glue.model.EntityNotFoundException) IOException(java.io.IOException) GetTablesRequest(com.amazonaws.services.glue.model.GetTablesRequest)

Example 10 with EntityNotFoundException

use of com.amazonaws.services.glue.model.EntityNotFoundException in project alluxio by Alluxio.

the class GlueDatabase method getTable.

@Override
public UdbTable getTable(String tableName, UdbBypassSpec bypassSpec) throws IOException {
    Table table;
    List<Partition> partitions;
    try {
        GetTableRequest tableRequest = new GetTableRequest().withCatalogId(mGlueConfiguration.get(Property.CATALOG_ID)).withDatabaseName(mGlueDbName).withName(tableName);
        table = getClient().getTable(tableRequest).getTable();
        partitions = batchGetPartitions(getClient(), tableName);
        PathTranslator pathTranslator = mountAlluxioPaths(table, partitions, bypassSpec);
        List<Column> partitionColumns;
        if (table.getPartitionKeys() == null) {
            partitionColumns = Collections.emptyList();
        } else {
            partitionColumns = table.getPartitionKeys();
        }
        // Get table parameters
        Map<String, String> tableParameters = table.getParameters() == null ? Collections.emptyMap() : table.getParameters();
        // Get column statistics info for table
        List<String> columnNames = table.getStorageDescriptor().getColumns().stream().map(Column::getName).collect(Collectors.toList());
        GetColumnStatisticsForTableRequest getColumnStatisticsForTableRequest = new GetColumnStatisticsForTableRequest().withCatalogId(mGlueConfiguration.get(Property.CATALOG_ID)).withDatabaseName(mGlueDbName).withTableName(tableName).withColumnNames(columnNames);
        List<ColumnStatisticsInfo> columnStatisticsTableData = new ArrayList<>();
        if (mGlueConfiguration.getBoolean(Property.TABLE_COLUMN_STATISTICS_ENABLE)) {
            columnStatisticsTableData = getTableColumnStatistics(mGlueDbName, tableName, getColumnStatisticsForTableRequest);
        }
        // Get column statistics info for partitions
        // potential expensive call
        Map<String, List<ColumnStatisticsInfo>> statsMap = new HashMap<>();
        if (mGlueConfiguration.getBoolean(Property.PARTITION_COLUMN_STATISTICS_ENABLE)) {
            for (Partition partition : partitions) {
                List<String> partitionValue = partition.getValues();
                if (partitionValue != null) {
                    GetColumnStatisticsForPartitionRequest getColumnStatisticsForPartitionRequest = new GetColumnStatisticsForPartitionRequest().withCatalogId(mGlueConfiguration.get(Property.CATALOG_ID)).withDatabaseName(mGlueDbName).withTableName(tableName).withColumnNames(columnNames).withPartitionValues(partitionValue);
                    String partName = GlueUtils.makePartitionName(partitionColumns, partition.getValues());
                    statsMap.put(partName, getPartitionColumnStatistics(mGlueDbName, tableName, getColumnStatisticsForPartitionRequest));
                }
            }
        }
        PartitionInfo partitionInfo = PartitionInfo.newBuilder().setDbName(mGlueDbName).setTableName(tableName).addAllDataCols(GlueUtils.toProto(table.getStorageDescriptor().getColumns())).setStorage(GlueUtils.toProto(table.getStorageDescriptor(), pathTranslator)).putAllParameters(tableParameters).build();
        Layout layout = Layout.newBuilder().setLayoutType(HiveLayout.TYPE).setLayoutData(partitionInfo.toByteString()).build();
        List<UdbPartition> udbPartitions = new ArrayList<>();
        if (partitionColumns.isEmpty()) {
            PartitionInfo.Builder partitionInfoBuilder = PartitionInfo.newBuilder().setDbName(mGlueDbName).setTableName(tableName).addAllDataCols(GlueUtils.toProto(table.getStorageDescriptor().getColumns())).setStorage(GlueUtils.toProto(table.getStorageDescriptor(), pathTranslator)).setPartitionName(tableName).putAllParameters(tableParameters);
            udbPartitions.add(new GluePartition(new HiveLayout(partitionInfoBuilder.build(), Collections.emptyList())));
        } else {
            for (Partition partition : partitions) {
                String partName = GlueUtils.makePartitionName(partitionColumns, partition.getValues());
                PartitionInfo.Builder partitionInfoBuilder = PartitionInfo.newBuilder().setDbName(mGlueDbName).setTableName(tableName).addAllDataCols(GlueUtils.toProto(partition.getStorageDescriptor().getColumns())).setStorage(GlueUtils.toProto(partition.getStorageDescriptor(), pathTranslator)).setPartitionName(partName).putAllParameters(partition.getParameters() == null ? Collections.emptyMap() : partition.getParameters());
                if (partition.getValues() != null) {
                    partitionInfoBuilder.addAllValues(partition.getValues());
                }
                udbPartitions.add(new GluePartition(new HiveLayout(partitionInfoBuilder.build(), statsMap.getOrDefault(partName, Collections.emptyList()))));
            }
        }
        return new GlueTable(this, pathTranslator, tableName, GlueUtils.toProtoSchema(table.getStorageDescriptor().getColumns()), columnStatisticsTableData, // Get FieldSchema from partition keys
        GlueUtils.toProto(table.getPartitionKeys()), udbPartitions, layout, table);
    } catch (EntityNotFoundException e) {
        throw new NotFoundException("Table " + tableName + " does not exist in Database: " + mGlueDbName + "; Catalog ID: " + mGlueConfiguration.get(Property.CATALOG_ID) + ".", e);
    } catch (ValidationException e) {
        throw new IOException("Failed to get table: " + tableName + " in Database: " + mGlueDbName + "; Catalog ID: " + mGlueConfiguration.get(Property.CATALOG_ID) + " with validation error: " + e.getMessage(), e);
    } catch (GlueEncryptionException e) {
        throw new IOException("Failed to get table: " + tableName + " in Database: " + mGlueDbName + "; Catalog ID: " + mGlueConfiguration.get(Property.CATALOG_ID) + " error: " + e.getMessage(), e);
    }
}
Also used : HiveLayout(alluxio.table.common.layout.HiveLayout) ValidationException(com.amazonaws.services.glue.model.ValidationException) HashMap(java.util.HashMap) ArrayList(java.util.ArrayList) EntityNotFoundException(com.amazonaws.services.glue.model.EntityNotFoundException) NotFoundException(alluxio.exception.status.NotFoundException) UdbPartition(alluxio.table.common.UdbPartition) Column(com.amazonaws.services.glue.model.Column) GlueEncryptionException(com.amazonaws.services.glue.model.GlueEncryptionException) List(java.util.List) ArrayList(java.util.ArrayList) PartitionInfo(alluxio.grpc.table.layout.hive.PartitionInfo) UdbPartition(alluxio.table.common.UdbPartition) Partition(com.amazonaws.services.glue.model.Partition) UdbTable(alluxio.table.common.udb.UdbTable) Table(com.amazonaws.services.glue.model.Table) GetColumnStatisticsForTableRequest(com.amazonaws.services.glue.model.GetColumnStatisticsForTableRequest) EntityNotFoundException(com.amazonaws.services.glue.model.EntityNotFoundException) IOException(java.io.IOException) GetTableRequest(com.amazonaws.services.glue.model.GetTableRequest) GetColumnStatisticsForPartitionRequest(com.amazonaws.services.glue.model.GetColumnStatisticsForPartitionRequest) PathTranslator(alluxio.table.common.udb.PathTranslator) Layout(alluxio.grpc.table.Layout) HiveLayout(alluxio.table.common.layout.HiveLayout) ColumnStatisticsInfo(alluxio.grpc.table.ColumnStatisticsInfo)

Aggregations

EntityNotFoundException (com.amazonaws.services.glue.model.EntityNotFoundException)10 AmazonServiceException (com.amazonaws.AmazonServiceException)7 PrestoException (com.facebook.presto.spi.PrestoException)7 SchemaTableName (com.facebook.presto.spi.SchemaTableName)6 TableInput (com.amazonaws.services.glue.model.TableInput)4 GlueInputConverter.toTableInput (com.facebook.presto.hive.metastore.glue.converter.GlueInputConverter.toTableInput)4 ArrayList (java.util.ArrayList)4 GetTablesRequest (com.amazonaws.services.glue.model.GetTablesRequest)3 GetTablesResult (com.amazonaws.services.glue.model.GetTablesResult)3 PartitionInput (com.amazonaws.services.glue.model.PartitionInput)3 UpdatePartitionRequest (com.amazonaws.services.glue.model.UpdatePartitionRequest)3 UpdateTableRequest (com.amazonaws.services.glue.model.UpdateTableRequest)3 PartitionNotFoundException (com.facebook.presto.hive.PartitionNotFoundException)3 PartitionStatistics (com.facebook.presto.hive.metastore.PartitionStatistics)3 TableNotFoundException (com.facebook.presto.spi.TableNotFoundException)3 AlreadyExistsException (com.amazonaws.services.glue.model.AlreadyExistsException)2 CreateTableRequest (com.amazonaws.services.glue.model.CreateTableRequest)2 GetDatabaseRequest (com.amazonaws.services.glue.model.GetDatabaseRequest)2 GetDatabaseResult (com.amazonaws.services.glue.model.GetDatabaseResult)2 SchemaAlreadyExistsException (com.facebook.presto.hive.SchemaAlreadyExistsException)2