Search in sources :

Example 6 with HiveClientConfig

use of com.facebook.presto.hive.HiveClientConfig in project presto by prestodb.

the class TestHiveClientGlueMetastore method createMetastore.

/**
 * GlueHiveMetastore currently uses AWS Default Credential Provider Chain,
 * See https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default
 * on ways to set your AWS credentials which will be needed to run this test.
 */
@Override
protected ExtendedHiveMetastore createMetastore(File tempDir) {
    HiveClientConfig hiveClientConfig = new HiveClientConfig();
    MetastoreClientConfig metastoreClientConfig = new MetastoreClientConfig();
    HdfsConfiguration hdfsConfiguration = new HiveHdfsConfiguration(new HdfsConfigurationInitializer(hiveClientConfig, metastoreClientConfig), ImmutableSet.of());
    HdfsEnvironment hdfsEnvironment = new HdfsEnvironment(hdfsConfiguration, metastoreClientConfig, new NoHdfsAuthentication());
    GlueHiveMetastoreConfig glueConfig = new GlueHiveMetastoreConfig();
    glueConfig.setDefaultWarehouseDir(tempDir.toURI().toString());
    return new GlueHiveMetastore(hdfsEnvironment, glueConfig, executor);
}
Also used : HdfsConfigurationInitializer(com.facebook.presto.hive.HdfsConfigurationInitializer) HiveHdfsConfiguration(com.facebook.presto.hive.HiveHdfsConfiguration) HdfsConfiguration(com.facebook.presto.hive.HdfsConfiguration) HiveHdfsConfiguration(com.facebook.presto.hive.HiveHdfsConfiguration) NoHdfsAuthentication(com.facebook.presto.hive.authentication.NoHdfsAuthentication) MetastoreClientConfig(com.facebook.presto.hive.MetastoreClientConfig) HiveClientConfig(com.facebook.presto.hive.HiveClientConfig) HdfsEnvironment(com.facebook.presto.hive.HdfsEnvironment)

Example 7 with HiveClientConfig

use of com.facebook.presto.hive.HiveClientConfig in project presto by prestodb.

the class ParquetTester method testSingleRead.

public static void testSingleRead(Iterable<?>[] readValues, List<String> columnNames, List<Type> columnTypes, ParquetMetadataSource parquetMetadataSource, File dataFile) {
    HiveClientConfig config = new HiveClientConfig().setHiveStorageFormat(HiveStorageFormat.PARQUET).setUseParquetColumnNames(false).setParquetMaxReadBlockSize(new DataSize(1_000, DataSize.Unit.BYTE));
    ConnectorSession session = new TestingConnectorSession(new HiveSessionProperties(config, new OrcFileWriterConfig(), new ParquetFileWriterConfig(), new CacheConfig()).getSessionProperties());
    HiveBatchPageSourceFactory pageSourceFactory = new ParquetPageSourceFactory(FUNCTION_AND_TYPE_MANAGER, FUNCTION_RESOLUTION, HDFS_ENVIRONMENT, new FileFormatDataSourceStats(), parquetMetadataSource);
    ConnectorPageSource connectorPageSource = createPageSource(pageSourceFactory, session, dataFile, columnNames, columnTypes, HiveStorageFormat.PARQUET);
    Iterator<?>[] expectedValues = stream(readValues).map(Iterable::iterator).toArray(size -> new Iterator<?>[size]);
    if (connectorPageSource instanceof RecordPageSource) {
        assertRecordCursor(columnTypes, expectedValues, ((RecordPageSource) connectorPageSource).getCursor());
    } else {
        assertPageSource(columnTypes, expectedValues, connectorPageSource);
    }
    assertFalse(stream(expectedValues).allMatch(Iterator::hasNext));
}
Also used : HiveBatchPageSourceFactory(com.facebook.presto.hive.HiveBatchPageSourceFactory) TestingConnectorSession(com.facebook.presto.testing.TestingConnectorSession) OrcFileWriterConfig(com.facebook.presto.hive.OrcFileWriterConfig) FileFormatDataSourceStats(com.facebook.presto.hive.FileFormatDataSourceStats) ConnectorPageSource(com.facebook.presto.spi.ConnectorPageSource) HiveSessionProperties(com.facebook.presto.hive.HiveSessionProperties) RecordPageSource(com.facebook.presto.spi.RecordPageSource) DataSize(io.airlift.units.DataSize) AbstractIterator(com.google.common.collect.AbstractIterator) Iterator(java.util.Iterator) ConnectorSession(com.facebook.presto.spi.ConnectorSession) TestingConnectorSession(com.facebook.presto.testing.TestingConnectorSession) CacheConfig(com.facebook.presto.cache.CacheConfig) ParquetFileWriterConfig(com.facebook.presto.hive.ParquetFileWriterConfig) HiveClientConfig(com.facebook.presto.hive.HiveClientConfig)

Example 8 with HiveClientConfig

use of com.facebook.presto.hive.HiveClientConfig in project presto by prestodb.

the class ParquetTester method createHiveClientConfig.

private static HiveClientConfig createHiveClientConfig(boolean useParquetColumnNames, boolean batchReadsEnabled) {
    HiveClientConfig config = new HiveClientConfig();
    config.setHiveStorageFormat(HiveStorageFormat.PARQUET).setUseParquetColumnNames(useParquetColumnNames).setParquetBatchReadOptimizationEnabled(batchReadsEnabled);
    return config;
}
Also used : HiveClientConfig(com.facebook.presto.hive.HiveClientConfig)

Example 9 with HiveClientConfig

use of com.facebook.presto.hive.HiveClientConfig in project presto by prestodb.

the class TestMetastoreHiveStatisticsProvider method testGetTableStatisticsEmpty.

@Test
public void testGetTableStatisticsEmpty() {
    String partitionName = "p1=string1/p2=1234";
    MetastoreHiveStatisticsProvider statisticsProvider = new MetastoreHiveStatisticsProvider((session, table, hivePartitions) -> ImmutableMap.of(partitionName, PartitionStatistics.empty()));
    TestingConnectorSession session = new TestingConnectorSession(new HiveSessionProperties(new HiveClientConfig(), new OrcFileWriterConfig(), new ParquetFileWriterConfig(), new CacheConfig()).getSessionProperties());
    assertEquals(statisticsProvider.getTableStatistics(session, TABLE, ImmutableMap.of(), ImmutableMap.of(), ImmutableList.of(partition(partitionName))), TableStatistics.empty());
}
Also used : TestingConnectorSession(com.facebook.presto.testing.TestingConnectorSession) OrcFileWriterConfig(com.facebook.presto.hive.OrcFileWriterConfig) CacheConfig(com.facebook.presto.cache.CacheConfig) HiveSessionProperties(com.facebook.presto.hive.HiveSessionProperties) ParquetFileWriterConfig(com.facebook.presto.hive.ParquetFileWriterConfig) HiveClientConfig(com.facebook.presto.hive.HiveClientConfig) Test(org.testng.annotations.Test)

Example 10 with HiveClientConfig

use of com.facebook.presto.hive.HiveClientConfig in project presto by prestodb.

the class TestMetastoreHiveStatisticsProvider method testGetTableStatisticsSampling.

@Test
public void testGetTableStatisticsSampling() {
    MetastoreHiveStatisticsProvider statisticsProvider = new MetastoreHiveStatisticsProvider((session, table, hivePartitions) -> {
        assertEquals(table, TABLE);
        assertEquals(hivePartitions.size(), 1);
        return ImmutableMap.of();
    });
    TestingConnectorSession session = new TestingConnectorSession(new HiveSessionProperties(new HiveClientConfig().setPartitionStatisticsSampleSize(1), new OrcFileWriterConfig(), new ParquetFileWriterConfig(), new CacheConfig()).getSessionProperties());
    statisticsProvider.getTableStatistics(session, TABLE, ImmutableMap.of(), ImmutableMap.of(), ImmutableList.of(partition("p1=string1/p2=1234"), partition("p1=string1/p2=1235")));
}
Also used : TestingConnectorSession(com.facebook.presto.testing.TestingConnectorSession) OrcFileWriterConfig(com.facebook.presto.hive.OrcFileWriterConfig) CacheConfig(com.facebook.presto.cache.CacheConfig) HiveSessionProperties(com.facebook.presto.hive.HiveSessionProperties) ParquetFileWriterConfig(com.facebook.presto.hive.ParquetFileWriterConfig) HiveClientConfig(com.facebook.presto.hive.HiveClientConfig) Test(org.testng.annotations.Test)

Aggregations

HiveClientConfig (com.facebook.presto.hive.HiveClientConfig)10 CacheConfig (com.facebook.presto.cache.CacheConfig)7 HiveSessionProperties (com.facebook.presto.hive.HiveSessionProperties)7 OrcFileWriterConfig (com.facebook.presto.hive.OrcFileWriterConfig)7 ParquetFileWriterConfig (com.facebook.presto.hive.ParquetFileWriterConfig)7 TestingConnectorSession (com.facebook.presto.testing.TestingConnectorSession)7 Test (org.testng.annotations.Test)5 HiveBasicStatistics (com.facebook.presto.hive.HiveBasicStatistics)3 PartitionStatistics (com.facebook.presto.hive.metastore.PartitionStatistics)3 MetastoreHiveStatisticsProvider.validatePartitionStatistics (com.facebook.presto.hive.statistics.MetastoreHiveStatisticsProvider.validatePartitionStatistics)3 HdfsConfiguration (com.facebook.presto.hive.HdfsConfiguration)2 HdfsConfigurationInitializer (com.facebook.presto.hive.HdfsConfigurationInitializer)2 HdfsEnvironment (com.facebook.presto.hive.HdfsEnvironment)2 HiveColumnHandle (com.facebook.presto.hive.HiveColumnHandle)2 HiveHdfsConfiguration (com.facebook.presto.hive.HiveHdfsConfiguration)2 MetastoreClientConfig (com.facebook.presto.hive.MetastoreClientConfig)2 NoHdfsAuthentication (com.facebook.presto.hive.authentication.NoHdfsAuthentication)2 ConnectorPageSource (com.facebook.presto.spi.ConnectorPageSource)2 ConnectorSession (com.facebook.presto.spi.ConnectorSession)2 DoubleRange (com.facebook.presto.spi.statistics.DoubleRange)2