Search in sources :

Example 36 with TestingConnectorContext

use of io.trino.testing.TestingConnectorContext in project trino by trinodb.

the class TestDeltaLakePlugin method testCreateConnector.

@Test
public void testCreateConnector() {
    Plugin plugin = new TestingDeltaLakePlugin();
    ConnectorFactory factory = getOnlyElement(plugin.getConnectorFactories());
    factory.create("test", ImmutableMap.of("hive.metastore.uri", "thrift://foo:1234"), new TestingConnectorContext());
}
Also used : ConnectorFactory(io.trino.spi.connector.ConnectorFactory) TestingConnectorContext(io.trino.testing.TestingConnectorContext) Plugin(io.trino.spi.Plugin) Test(org.testng.annotations.Test)

Example 37 with TestingConnectorContext

use of io.trino.testing.TestingConnectorContext in project trino by trinodb.

the class TestDeltaLakePlugin method testAlluxioMetastore.

/**
 * Verify the Alluxio metastore is not supported for Delta. Delta connector extends Hive connector and Hive connector supports Alluxio metastore.
 * We explicitly disallow Alluxio metastore use with Delta.
 */
@Test
public void testAlluxioMetastore() {
    Plugin plugin = new TestingDeltaLakePlugin();
    ConnectorFactory factory = getOnlyElement(plugin.getConnectorFactories());
    assertThatThrownBy(() -> factory.create("test", ImmutableMap.of("hive.metastore", "alluxio"), new TestingConnectorContext())).hasMessageMatching("(?s)Unable to create injector, see the following errors:.*" + "Explicit bindings are required and HiveMetastoreFactory .* is not explicitly bound.*");
}
Also used : ConnectorFactory(io.trino.spi.connector.ConnectorFactory) TestingConnectorContext(io.trino.testing.TestingConnectorContext) Plugin(io.trino.spi.Plugin) Test(org.testng.annotations.Test)

Example 38 with TestingConnectorContext

use of io.trino.testing.TestingConnectorContext in project trino by trinodb.

the class TestDeltaLakeGlueMetastore method setUp.

@BeforeClass
public void setUp() {
    tempDir = Files.createTempDir();
    String temporaryLocation = tempDir.toURI().toString();
    Map<String, String> config = ImmutableMap.<String, String>builder().put("hive.metastore", "glue").put("delta.hide-non-delta-lake-tables", "true").buildOrThrow();
    Bootstrap app = new Bootstrap(// connector dependencies
    new JsonModule(), binder -> {
        ConnectorContext context = new TestingConnectorContext();
        binder.bind(CatalogName.class).toInstance(new CatalogName("test"));
        binder.bind(TypeManager.class).toInstance(context.getTypeManager());
        binder.bind(NodeManager.class).toInstance(context.getNodeManager());
        binder.bind(PageIndexerFactory.class).toInstance(context.getPageIndexerFactory());
        binder.bind(NodeVersion.class).toInstance(new NodeVersion("test_version"));
    }, // connector modules
    new DeltaLakeMetastoreModule(), new DeltaLakeModule(), // test setup
    binder -> {
        binder.bind(HdfsEnvironment.class).toInstance(HDFS_ENVIRONMENT);
    });
    Injector injector = app.doNotInitializeLogging().setRequiredConfigurationProperties(config).initialize();
    lifeCycleManager = injector.getInstance(LifeCycleManager.class);
    metastoreClient = injector.getInstance(GlueHiveMetastore.class);
    metadataFactory = injector.getInstance(DeltaLakeMetadataFactory.class);
    session = TestingConnectorSession.builder().setPropertyMetadata(injector.getInstance(DeltaLakeSessionProperties.class).getSessionProperties()).build();
    databaseName = "test_delta_glue" + randomName();
    metastoreClient.createDatabase(Database.builder().setDatabaseName(databaseName).setOwnerName(Optional.of("public")).setOwnerType(Optional.of(ROLE)).setLocation(Optional.of(temporaryLocation)).build());
}
Also used : DeltaLakeModule(io.trino.plugin.deltalake.DeltaLakeModule) TestingConnectorContext(io.trino.testing.TestingConnectorContext) DeltaLakeMetadataFactory(io.trino.plugin.deltalake.DeltaLakeMetadataFactory) JsonModule(io.airlift.json.JsonModule) PageIndexerFactory(io.trino.spi.PageIndexerFactory) HdfsEnvironment(io.trino.plugin.hive.HdfsEnvironment) GlueHiveMetastore(io.trino.plugin.hive.metastore.glue.GlueHiveMetastore) NodeVersion(io.trino.plugin.hive.NodeVersion) NodeManager(io.trino.spi.NodeManager) LifeCycleManager(io.airlift.bootstrap.LifeCycleManager) Injector(com.google.inject.Injector) ConnectorContext(io.trino.spi.connector.ConnectorContext) TestingConnectorContext(io.trino.testing.TestingConnectorContext) DeltaLakeMetastoreModule(io.trino.plugin.deltalake.metastore.DeltaLakeMetastoreModule) Bootstrap(io.airlift.bootstrap.Bootstrap) TypeManager(io.trino.spi.type.TypeManager) CatalogName(io.trino.plugin.base.CatalogName) BeforeClass(org.testng.annotations.BeforeClass)

Example 39 with TestingConnectorContext

use of io.trino.testing.TestingConnectorContext in project trino by trinodb.

the class TestDeltaLakeMetadata method setUp.

@BeforeClass
public void setUp() throws IOException {
    temporaryCatalogDirectory = createTempDirectory("HiveCatalog").toFile();
    Map<String, String> config = ImmutableMap.<String, String>builder().put("hive.metastore", "file").put("hive.metastore.catalog.dir", temporaryCatalogDirectory.getPath()).buildOrThrow();
    Bootstrap app = new Bootstrap(// connector dependencies
    new JsonModule(), binder -> {
        ConnectorContext context = new TestingConnectorContext();
        binder.bind(NodeVersion.class).toInstance(new NodeVersion(context.getNodeManager().getCurrentNode().getVersion()));
        binder.bind(CatalogName.class).toInstance(new CatalogName("test"));
        binder.bind(TypeManager.class).toInstance(context.getTypeManager());
        binder.bind(NodeManager.class).toInstance(context.getNodeManager());
        binder.bind(PageIndexerFactory.class).toInstance(context.getPageIndexerFactory());
    }, // connector modules
    new DeltaLakeMetastoreModule(), new DeltaLakeModule(), // test setup
    binder -> {
        binder.bind(HdfsEnvironment.class).toInstance(HDFS_ENVIRONMENT);
    }, new AbstractModule() {

        @Provides
        public DeltaLakeMetastore getDeltaLakeMetastore(@RawHiveMetastoreFactory HiveMetastoreFactory hiveMetastoreFactory, TransactionLogAccess transactionLogAccess, TypeManager typeManager, CachingDeltaLakeStatisticsAccess statistics) {
            return new HiveMetastoreBackedDeltaLakeMetastore(hiveMetastoreFactory.createMetastore(Optional.empty()), transactionLogAccess, typeManager, statistics);
        }
    });
    Injector injector = app.doNotInitializeLogging().setRequiredConfigurationProperties(config).initialize();
    deltaLakeMetadataFactory = injector.getInstance(DeltaLakeMetadataFactory.class);
    injector.getInstance(DeltaLakeMetastore.class).createDatabase(Database.builder().setDatabaseName(DATABASE_NAME).setOwnerName(Optional.of("test")).setOwnerType(Optional.of(USER)).setLocation(Optional.empty()).build());
}
Also used : HiveMetastoreFactory(io.trino.plugin.hive.metastore.HiveMetastoreFactory) RawHiveMetastoreFactory(io.trino.plugin.hive.metastore.RawHiveMetastoreFactory) TransactionLogAccess(io.trino.plugin.deltalake.transactionlog.TransactionLogAccess) TestingConnectorContext(io.trino.testing.TestingConnectorContext) Provides(com.google.inject.Provides) DeltaLakeMetastore(io.trino.plugin.deltalake.metastore.DeltaLakeMetastore) HiveMetastoreBackedDeltaLakeMetastore(io.trino.plugin.deltalake.metastore.HiveMetastoreBackedDeltaLakeMetastore) JsonModule(io.airlift.json.JsonModule) PageIndexerFactory(io.trino.spi.PageIndexerFactory) HdfsEnvironment(io.trino.plugin.hive.HdfsEnvironment) AbstractModule(com.google.inject.AbstractModule) NodeVersion(io.trino.plugin.hive.NodeVersion) NodeManager(io.trino.spi.NodeManager) HiveMetastoreBackedDeltaLakeMetastore(io.trino.plugin.deltalake.metastore.HiveMetastoreBackedDeltaLakeMetastore) Injector(com.google.inject.Injector) ConnectorContext(io.trino.spi.connector.ConnectorContext) TestingConnectorContext(io.trino.testing.TestingConnectorContext) DeltaLakeMetastoreModule(io.trino.plugin.deltalake.metastore.DeltaLakeMetastoreModule) Bootstrap(io.airlift.bootstrap.Bootstrap) TypeManager(io.trino.spi.type.TypeManager) CatalogName(io.trino.plugin.base.CatalogName) CachingDeltaLakeStatisticsAccess(io.trino.plugin.deltalake.statistics.CachingDeltaLakeStatisticsAccess) BeforeClass(org.testng.annotations.BeforeClass)

Example 40 with TestingConnectorContext

use of io.trino.testing.TestingConnectorContext in project trino by trinodb.

the class TestTransactionLogAccess method setupTransactionLogAccess.

private void setupTransactionLogAccess(String tableName, Path tableLocation) throws IOException {
    TestingConnectorContext context = new TestingConnectorContext();
    TypeManager typeManager = context.getTypeManager();
    HdfsConfig hdfsConfig = new HdfsConfig();
    HdfsConfiguration hdfsConfiguration = new HiveHdfsConfiguration(new HdfsConfigurationInitializer(hdfsConfig), ImmutableSet.of());
    HdfsEnvironment hdfsEnvironment = new HdfsEnvironment(hdfsConfiguration, hdfsConfig, new NoHdfsAuthentication());
    FileFormatDataSourceStats fileFormatDataSourceStats = new FileFormatDataSourceStats();
    transactionLogAccess = new TrackingTransactionLogAccess(tableName, tableLocation, SESSION, typeManager, new CheckpointSchemaManager(typeManager), new DeltaLakeConfig(), fileFormatDataSourceStats, hdfsEnvironment, new ParquetReaderConfig());
    DeltaLakeTableHandle tableHandle = new DeltaLakeTableHandle("schema", tableName, "location", // ignored
    Optional.empty(), TupleDomain.none(), TupleDomain.none(), Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty(), 0);
    tableSnapshot = transactionLogAccess.loadSnapshot(tableHandle.getSchemaTableName(), tableLocation, SESSION);
}
Also used : HdfsConfigurationInitializer(io.trino.plugin.hive.HdfsConfigurationInitializer) HiveHdfsConfiguration(io.trino.plugin.hive.HiveHdfsConfiguration) HdfsConfig(io.trino.plugin.hive.HdfsConfig) FileFormatDataSourceStats(io.trino.plugin.hive.FileFormatDataSourceStats) TestingConnectorContext(io.trino.testing.TestingConnectorContext) HiveHdfsConfiguration(io.trino.plugin.hive.HiveHdfsConfiguration) HdfsConfiguration(io.trino.plugin.hive.HdfsConfiguration) NoHdfsAuthentication(io.trino.plugin.hive.authentication.NoHdfsAuthentication) HdfsEnvironment(io.trino.plugin.hive.HdfsEnvironment) CheckpointSchemaManager(io.trino.plugin.deltalake.transactionlog.checkpoint.CheckpointSchemaManager) TypeManager(io.trino.spi.type.TypeManager) ParquetReaderConfig(io.trino.plugin.hive.parquet.ParquetReaderConfig)

Aggregations

TestingConnectorContext (io.trino.testing.TestingConnectorContext)64 ConnectorFactory (io.trino.spi.connector.ConnectorFactory)56 Test (org.testng.annotations.Test)53 Plugin (io.trino.spi.Plugin)22 Connector (io.trino.spi.connector.Connector)14 HdfsEnvironment (io.trino.plugin.hive.HdfsEnvironment)5 TypeManager (io.trino.spi.type.TypeManager)5 File (java.io.File)4 Path (java.nio.file.Path)4 BeforeClass (org.testng.annotations.BeforeClass)4 TransactionLogAccess (io.trino.plugin.deltalake.transactionlog.TransactionLogAccess)3 CheckpointSchemaManager (io.trino.plugin.deltalake.transactionlog.checkpoint.CheckpointSchemaManager)3 FileFormatDataSourceStats (io.trino.plugin.hive.FileFormatDataSourceStats)3 HdfsConfig (io.trino.plugin.hive.HdfsConfig)3 HdfsConfiguration (io.trino.plugin.hive.HdfsConfiguration)3 HdfsConfigurationInitializer (io.trino.plugin.hive.HdfsConfigurationInitializer)3 HiveHdfsConfiguration (io.trino.plugin.hive.HiveHdfsConfiguration)3 NodeVersion (io.trino.plugin.hive.NodeVersion)3 NoHdfsAuthentication (io.trino.plugin.hive.authentication.NoHdfsAuthentication)3 ParquetReaderConfig (io.trino.plugin.hive.parquet.ParquetReaderConfig)3