Search in sources :

Example 1 with SerDeInfo

use of com.amazonaws.services.glue.model.SerDeInfo in project presto by prestodb.

the class GlueInputConverter method convertStorage.

private static StorageDescriptor convertStorage(Storage storage, List<Column> columns) {
    if (storage.isSkewed()) {
        throw new IllegalArgumentException("Writing to skewed table/partition is not supported");
    }
    SerDeInfo serdeInfo = new SerDeInfo().withSerializationLibrary(storage.getStorageFormat().getSerDeNullable()).withParameters(storage.getSerdeParameters());
    StorageDescriptor sd = new StorageDescriptor();
    sd.setLocation(storage.getLocation());
    sd.setColumns(columns.stream().map(GlueInputConverter::convertColumn).collect(toList()));
    sd.setSerdeInfo(serdeInfo);
    sd.setInputFormat(storage.getStorageFormat().getInputFormatNullable());
    sd.setOutputFormat(storage.getStorageFormat().getOutputFormatNullable());
    sd.setParameters(ImmutableMap.of());
    Optional<HiveBucketProperty> bucketProperty = storage.getBucketProperty();
    if (bucketProperty.isPresent()) {
        sd.setNumberOfBuckets(bucketProperty.get().getBucketCount());
        sd.setBucketColumns(bucketProperty.get().getBucketedBy());
        if (!bucketProperty.get().getSortedBy().isEmpty()) {
            sd.setSortColumns(bucketProperty.get().getSortedBy().stream().map(column -> new Order().withColumn(column.getColumnName()).withSortOrder(column.getOrder().getHiveOrder())).collect(toImmutableList()));
        }
    }
    return sd;
}
Also used : Order(com.amazonaws.services.glue.model.Order) HiveBucketProperty(com.facebook.presto.hive.HiveBucketProperty) SerDeInfo(com.amazonaws.services.glue.model.SerDeInfo) StorageDescriptor(com.amazonaws.services.glue.model.StorageDescriptor)

Aggregations

Order (com.amazonaws.services.glue.model.Order)1 SerDeInfo (com.amazonaws.services.glue.model.SerDeInfo)1 StorageDescriptor (com.amazonaws.services.glue.model.StorageDescriptor)1 HiveBucketProperty (com.facebook.presto.hive.HiveBucketProperty)1