Search in sources :

Example 1 with BridgingHiveMetastore

use of io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore in project trino by trinodb.

the class AbstractTestHive method setup.

protected final void setup(String host, int port, String databaseName, String timeZone) {
    HiveConfig hiveConfig = getHiveConfig().setParquetTimeZone(timeZone).setRcfileTimeZone(timeZone);
    Optional<HostAndPort> proxy = Optional.ofNullable(System.getProperty("hive.metastore.thrift.client.socks-proxy")).map(HostAndPort::fromString);
    MetastoreLocator metastoreLocator = new TestingMetastoreLocator(proxy, HostAndPort.fromParts(host, port));
    hdfsEnvironment = new HdfsEnvironment(createTestHdfsConfiguration(), new HdfsConfig(), new NoHdfsAuthentication());
    HiveMetastore metastore = cachingHiveMetastore(new BridgingHiveMetastore(new ThriftHiveMetastore(metastoreLocator, hiveConfig, new MetastoreConfig(), new ThriftMetastoreConfig(), hdfsEnvironment, false), new HiveIdentity(SESSION.getIdentity())), executor, new Duration(1, MINUTES), Optional.of(new Duration(15, SECONDS)), 10000);
    setup(databaseName, hiveConfig, metastore, hdfsEnvironment);
}
Also used : ThriftMetastoreConfig(io.trino.plugin.hive.metastore.thrift.ThriftMetastoreConfig) MetastoreConfig(io.trino.plugin.hive.metastore.MetastoreConfig) TestingMetastoreLocator(io.trino.plugin.hive.metastore.thrift.TestingMetastoreLocator) HiveMetastore(io.trino.plugin.hive.metastore.HiveMetastore) BridgingHiveMetastore(io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore) CachingHiveMetastore.cachingHiveMetastore(io.trino.plugin.hive.metastore.cache.CachingHiveMetastore.cachingHiveMetastore) ThriftHiveMetastore(io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastore) SemiTransactionalHiveMetastore(io.trino.plugin.hive.metastore.SemiTransactionalHiveMetastore) ThriftHiveMetastore(io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastore) MetastoreLocator(io.trino.plugin.hive.metastore.thrift.MetastoreLocator) TestingMetastoreLocator(io.trino.plugin.hive.metastore.thrift.TestingMetastoreLocator) Duration(io.airlift.units.Duration) ThriftMetastoreConfig(io.trino.plugin.hive.metastore.thrift.ThriftMetastoreConfig) NoHdfsAuthentication(io.trino.plugin.hive.authentication.NoHdfsAuthentication) HiveIdentity(io.trino.plugin.hive.authentication.HiveIdentity) HostAndPort(com.google.common.net.HostAndPort) BridgingHiveMetastore(io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore)

Example 2 with BridgingHiveMetastore

use of io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore in project trino by trinodb.

the class BaseTestHiveOnDataLake method createQueryRunner.

@Override
protected QueryRunner createQueryRunner() throws Exception {
    this.bucketName = "test-hive-insert-overwrite-" + randomTableSuffix();
    this.dockerizedS3DataLake = closeAfterClass(new HiveMinioDataLake(bucketName, ImmutableMap.of(), hiveHadoopImage));
    this.dockerizedS3DataLake.start();
    this.metastoreClient = new BridgingHiveMetastore(new ThriftHiveMetastore(new TestingMetastoreLocator(Optional.empty(), this.dockerizedS3DataLake.getHiveHadoop().getHiveMetastoreEndpoint()), new HiveConfig(), new MetastoreConfig(), new ThriftMetastoreConfig(), new HdfsEnvironment(new HiveHdfsConfiguration(new HdfsConfigurationInitializer(new HdfsConfig(), ImmutableSet.of()), ImmutableSet.of()), new HdfsConfig(), new NoHdfsAuthentication()), false), HiveIdentity.none());
    return S3HiveQueryRunner.create(dockerizedS3DataLake, ImmutableMap.<String, String>builder().put("hive.insert-existing-partitions-behavior", "OVERWRITE").put("hive.non-managed-table-writes-enabled", "true").put("hive.metastore-cache-ttl", "1d").put("hive.metastore-refresh-interval", "1d").buildOrThrow());
}
Also used : MetastoreConfig(io.trino.plugin.hive.metastore.MetastoreConfig) ThriftMetastoreConfig(io.trino.plugin.hive.metastore.thrift.ThriftMetastoreConfig) TestingMetastoreLocator(io.trino.plugin.hive.metastore.thrift.TestingMetastoreLocator) ThriftHiveMetastore(io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastore) HiveMinioDataLake(io.trino.plugin.hive.containers.HiveMinioDataLake) ThriftMetastoreConfig(io.trino.plugin.hive.metastore.thrift.ThriftMetastoreConfig) NoHdfsAuthentication(io.trino.plugin.hive.authentication.NoHdfsAuthentication) BridgingHiveMetastore(io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore)

Example 3 with BridgingHiveMetastore

use of io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore in project trino by trinodb.

the class TestCachingHiveMetastore method setUp.

@BeforeMethod
public void setUp() {
    mockClient = new MockThriftMetastoreClient();
    ThriftHiveMetastore thriftHiveMetastore = createThriftHiveMetastore();
    executor = listeningDecorator(newCachedThreadPool(daemonThreadsNamed(getClass().getSimpleName() + "-%s")));
    metastore = cachingHiveMetastore(new BridgingHiveMetastore(thriftHiveMetastore, IDENTITY), executor, new Duration(5, TimeUnit.MINUTES), Optional.of(new Duration(1, TimeUnit.MINUTES)), 1000);
    stats = thriftHiveMetastore.getStats();
}
Also used : MockThriftMetastoreClient(io.trino.plugin.hive.metastore.thrift.MockThriftMetastoreClient) ThriftHiveMetastore(io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastore) Duration(io.airlift.units.Duration) BridgingHiveMetastore(io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore) BeforeMethod(org.testng.annotations.BeforeMethod)

Example 4 with BridgingHiveMetastore

use of io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore in project trino by trinodb.

the class TestCachingHiveMetastore method testCachingHiveMetastoreCreationViaMemoize.

@Test
public void testCachingHiveMetastoreCreationViaMemoize() {
    ThriftHiveMetastore thriftHiveMetastore = createThriftHiveMetastore();
    metastore = memoizeMetastore(new BridgingHiveMetastore(thriftHiveMetastore, IDENTITY), 1000);
    assertEquals(mockClient.getAccessCount(), 0);
    assertEquals(metastore.getAllDatabases(), ImmutableList.of(TEST_DATABASE));
    assertEquals(mockClient.getAccessCount(), 1);
    assertEquals(metastore.getAllDatabases(), ImmutableList.of(TEST_DATABASE));
    assertEquals(mockClient.getAccessCount(), 1);
    assertEquals(metastore.getDatabaseNamesStats().getRequestCount(), 0);
}
Also used : ThriftHiveMetastore(io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastore) BridgingHiveMetastore(io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore) Test(org.testng.annotations.Test)

Example 5 with BridgingHiveMetastore

use of io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore in project trino by trinodb.

the class TestHiveInMemoryMetastore method createMetastore.

@Override
protected HiveMetastore createMetastore(File tempDir, HiveIdentity identity) {
    File baseDir = new File(tempDir, "metastore");
    ThriftMetastoreConfig metastoreConfig = new ThriftMetastoreConfig();
    InMemoryThriftMetastore hiveMetastore = new InMemoryThriftMetastore(baseDir, metastoreConfig);
    return new BridgingHiveMetastore(hiveMetastore, identity);
}
Also used : InMemoryThriftMetastore(io.trino.plugin.hive.metastore.thrift.InMemoryThriftMetastore) ThriftMetastoreConfig(io.trino.plugin.hive.metastore.thrift.ThriftMetastoreConfig) File(java.io.File) BridgingHiveMetastore(io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore)

Aggregations

BridgingHiveMetastore (io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore)6 ThriftHiveMetastore (io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastore)5 ThriftMetastoreConfig (io.trino.plugin.hive.metastore.thrift.ThriftMetastoreConfig)4 NoHdfsAuthentication (io.trino.plugin.hive.authentication.NoHdfsAuthentication)3 MetastoreConfig (io.trino.plugin.hive.metastore.MetastoreConfig)3 TestingMetastoreLocator (io.trino.plugin.hive.metastore.thrift.TestingMetastoreLocator)3 HostAndPort (com.google.common.net.HostAndPort)2 Duration (io.airlift.units.Duration)2 HiveIdentity (io.trino.plugin.hive.authentication.HiveIdentity)2 MetastoreLocator (io.trino.plugin.hive.metastore.thrift.MetastoreLocator)2 BoundedExecutor (io.airlift.concurrent.BoundedExecutor)1 CounterStat (io.airlift.stats.CounterStat)1 GroupByHashPageIndexerFactory (io.trino.operator.GroupByHashPageIndexerFactory)1 CatalogName (io.trino.plugin.base.CatalogName)1 HiveMinioDataLake (io.trino.plugin.hive.containers.HiveMinioDataLake)1 HiveMetastore (io.trino.plugin.hive.metastore.HiveMetastore)1 SemiTransactionalHiveMetastore (io.trino.plugin.hive.metastore.SemiTransactionalHiveMetastore)1 CachingHiveMetastore.cachingHiveMetastore (io.trino.plugin.hive.metastore.cache.CachingHiveMetastore.cachingHiveMetastore)1 InMemoryThriftMetastore (io.trino.plugin.hive.metastore.thrift.InMemoryThriftMetastore)1 MockThriftMetastoreClient (io.trino.plugin.hive.metastore.thrift.MockThriftMetastoreClient)1