Search in sources :

Example 16 with ConnectorIdentity

use of io.trino.spi.security.ConnectorIdentity in project trino by trinodb.

the class BaseJdbcClient method renameSchema.

@Override
public void renameSchema(ConnectorSession session, String schemaName, String newSchemaName) {
    ConnectorIdentity identity = session.getIdentity();
    try (Connection connection = connectionFactory.openConnection(session)) {
        String remoteSchemaName = identifierMapping.toRemoteSchemaName(identity, connection, schemaName);
        String newRemoteSchemaName = identifierMapping.toRemoteSchemaName(identity, connection, newSchemaName);
        execute(connection, renameSchemaSql(remoteSchemaName, newRemoteSchemaName));
    } catch (SQLException e) {
        throw new TrinoException(JDBC_ERROR, e);
    }
}
Also used : SQLException(java.sql.SQLException) Connection(java.sql.Connection) ConnectorIdentity(io.trino.spi.security.ConnectorIdentity) TrinoException(io.trino.spi.TrinoException)

Example 17 with ConnectorIdentity

use of io.trino.spi.security.ConnectorIdentity in project trino by trinodb.

the class BaseJdbcClient method getTableNames.

@Override
public List<SchemaTableName> getTableNames(ConnectorSession session, Optional<String> schema) {
    try (Connection connection = connectionFactory.openConnection(session)) {
        ConnectorIdentity identity = session.getIdentity();
        Optional<String> remoteSchema = schema.map(schemaName -> identifierMapping.toRemoteSchemaName(identity, connection, schemaName));
        if (remoteSchema.isPresent() && !filterSchema(remoteSchema.get())) {
            return ImmutableList.of();
        }
        try (ResultSet resultSet = getTables(connection, remoteSchema, Optional.empty())) {
            ImmutableList.Builder<SchemaTableName> list = ImmutableList.builder();
            while (resultSet.next()) {
                String remoteSchemaFromResultSet = getTableSchemaName(resultSet);
                String tableSchema = identifierMapping.fromRemoteSchemaName(remoteSchemaFromResultSet);
                String tableName = identifierMapping.fromRemoteTableName(remoteSchemaFromResultSet, resultSet.getString("TABLE_NAME"));
                if (filterSchema(tableSchema)) {
                    list.add(new SchemaTableName(tableSchema, tableName));
                }
            }
            return list.build();
        }
    } catch (SQLException e) {
        throw new TrinoException(JDBC_ERROR, e);
    }
}
Also used : SQLException(java.sql.SQLException) ImmutableList.toImmutableList(com.google.common.collect.ImmutableList.toImmutableList) ImmutableList(com.google.common.collect.ImmutableList) Connection(java.sql.Connection) ResultSet(java.sql.ResultSet) ConnectorIdentity(io.trino.spi.security.ConnectorIdentity) TrinoException(io.trino.spi.TrinoException) SchemaTableName(io.trino.spi.connector.SchemaTableName)

Example 18 with ConnectorIdentity

use of io.trino.spi.security.ConnectorIdentity in project trino by trinodb.

the class BaseJdbcClient method getTableHandle.

@Override
public Optional<JdbcTableHandle> getTableHandle(ConnectorSession session, SchemaTableName schemaTableName) {
    try (Connection connection = connectionFactory.openConnection(session)) {
        ConnectorIdentity identity = session.getIdentity();
        String remoteSchema = identifierMapping.toRemoteSchemaName(identity, connection, schemaTableName.getSchemaName());
        String remoteTable = identifierMapping.toRemoteTableName(identity, connection, remoteSchema, schemaTableName.getTableName());
        try (ResultSet resultSet = getTables(connection, Optional.of(remoteSchema), Optional.of(remoteTable))) {
            List<JdbcTableHandle> tableHandles = new ArrayList<>();
            while (resultSet.next()) {
                tableHandles.add(new JdbcTableHandle(schemaTableName, getRemoteTable(resultSet), getTableComment(resultSet)));
            }
            if (tableHandles.isEmpty()) {
                return Optional.empty();
            }
            if (tableHandles.size() > 1) {
                throw new TrinoException(NOT_SUPPORTED, "Multiple tables matched: " + schemaTableName);
            }
            return Optional.of(getOnlyElement(tableHandles));
        }
    } catch (SQLException e) {
        throw new TrinoException(JDBC_ERROR, e);
    }
}
Also used : SQLException(java.sql.SQLException) Connection(java.sql.Connection) ResultSet(java.sql.ResultSet) ArrayList(java.util.ArrayList) ConnectorIdentity(io.trino.spi.security.ConnectorIdentity) TrinoException(io.trino.spi.TrinoException)

Example 19 with ConnectorIdentity

use of io.trino.spi.security.ConnectorIdentity in project trino by trinodb.

the class TestDeltaLakePerTransactionMetastoreCache method createQueryRunner.

private DistributedQueryRunner createQueryRunner(boolean enablePerTransactionHiveMetastoreCaching) throws Exception {
    boolean createdDeltaLake = false;
    if (dockerizedMinioDataLake == null) {
        // share environment between testcases to speed things up
        dockerizedMinioDataLake = createDockerizedMinioDataLakeForDeltaLake(BUCKET_NAME);
        createdDeltaLake = true;
    }
    Session session = testSessionBuilder().setCatalog(DELTA_CATALOG).setSchema("default").build();
    DistributedQueryRunner queryRunner = DistributedQueryRunner.builder(session).build();
    queryRunner.installPlugin(new Plugin() {

        @Override
        public Iterable<ConnectorFactory> getConnectorFactories() {
            return ImmutableList.of(new ConnectorFactory() {

                @Override
                public String getName() {
                    return TEST_DELTA_CONNECTOR_NAME;
                }

                @Override
                public Connector create(String catalogName, Map<String, String> config, ConnectorContext context) {
                    return InternalDeltaLakeConnectorFactory.createConnector(catalogName, config, context, new AbstractConfigurationAwareModule() {

                        @Override
                        protected void setup(Binder binder) {
                            newOptionalBinder(binder, ThriftMetastoreClientFactory.class).setDefault().to(DefaultThriftMetastoreClientFactory.class).in(Scopes.SINGLETON);
                            binder.bind(MetastoreLocator.class).to(StaticMetastoreLocator.class).in(Scopes.SINGLETON);
                            configBinder(binder).bindConfig(StaticMetastoreConfig.class);
                            configBinder(binder).bindConfig(ThriftMetastoreConfig.class);
                            binder.bind(ThriftMetastore.class).to(ThriftHiveMetastore.class).in(Scopes.SINGLETON);
                            newExporter(binder).export(ThriftMetastore.class).as((generator) -> generator.generatedNameOf(ThriftHiveMetastore.class));
                            install(new ThriftMetastoreAuthenticationModule());
                            binder.bind(Boolean.class).annotatedWith(HideNonDeltaLakeTables.class).toInstance(false);
                            binder.bind(BridgingHiveMetastoreFactory.class).in(Scopes.SINGLETON);
                        }

                        @Provides
                        @Singleton
                        @RawHiveMetastoreFactory
                        public HiveMetastoreFactory getCountingHiveMetastoreFactory(BridgingHiveMetastoreFactory bridgingHiveMetastoreFactory) {
                            return new HiveMetastoreFactory() {

                                @Override
                                public boolean isImpersonationEnabled() {
                                    return false;
                                }

                                @Override
                                public HiveMetastore createMetastore(Optional<ConnectorIdentity> identity) {
                                    HiveMetastore bridgingHiveMetastore = bridgingHiveMetastoreFactory.createMetastore(identity);
                                    // bind HiveMetastore which counts method executions
                                    return Reflection.newProxy(HiveMetastore.class, (proxy, method, args) -> {
                                        String methodName = method.getName();
                                        long count = hiveMetastoreInvocationCounts.getOrDefault(methodName, 0L);
                                        hiveMetastoreInvocationCounts.put(methodName, count + 1);
                                        return method.invoke(bridgingHiveMetastore, args);
                                    });
                                }
                            };
                        }
                    });
                }
            });
        }
    });
    ImmutableMap.Builder<String, String> deltaLakeProperties = ImmutableMap.builder();
    deltaLakeProperties.put("hive.metastore.uri", dockerizedMinioDataLake.getTestingHadoop().getMetastoreAddress());
    deltaLakeProperties.put("hive.s3.aws-access-key", MINIO_ACCESS_KEY);
    deltaLakeProperties.put("hive.s3.aws-secret-key", MINIO_SECRET_KEY);
    deltaLakeProperties.put("hive.s3.endpoint", dockerizedMinioDataLake.getMinioAddress());
    deltaLakeProperties.put("hive.s3.path-style-access", "true");
    // use test value so we do not get clash with default bindings)
    deltaLakeProperties.put("hive.metastore", "test");
    if (!enablePerTransactionHiveMetastoreCaching) {
        // almost disable the cache; 0 is not allowed as config property value
        deltaLakeProperties.put("hive.per-transaction-metastore-cache-maximum-size", "1");
    }
    queryRunner.createCatalog(DELTA_CATALOG, TEST_DELTA_CONNECTOR_NAME, deltaLakeProperties.buildOrThrow());
    if (createdDeltaLake) {
        List<TpchTable<? extends TpchEntity>> tpchTables = List.of(TpchTable.NATION, TpchTable.REGION);
        tpchTables.forEach(table -> {
            String tableName = table.getTableName();
            dockerizedMinioDataLake.copyResources("io/trino/plugin/deltalake/testing/resources/databricks/" + tableName, tableName);
            queryRunner.execute(format("CREATE TABLE %s.%s.%s (dummy int) WITH (location = 's3://%s/%3$s')", DELTA_CATALOG, "default", tableName, BUCKET_NAME));
        });
    }
    return queryRunner;
}
Also used : BridgingHiveMetastoreFactory(io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastoreFactory) DELTA_CATALOG(io.trino.plugin.deltalake.DeltaLakeQueryRunner.DELTA_CATALOG) Assertions.assertThat(org.assertj.core.api.Assertions.assertThat) MINIO_SECRET_KEY(io.trino.plugin.deltalake.util.MinioContainer.MINIO_SECRET_KEY) Test(org.testng.annotations.Test) ConnectorContext(io.trino.spi.connector.ConnectorContext) Singleton(javax.inject.Singleton) StaticMetastoreConfig(io.trino.plugin.hive.metastore.thrift.StaticMetastoreConfig) MINIO_ACCESS_KEY(io.trino.plugin.deltalake.util.MinioContainer.MINIO_ACCESS_KEY) ThriftMetastoreClientFactory(io.trino.plugin.hive.metastore.thrift.ThriftMetastoreClientFactory) HiveMetastoreFactory(io.trino.plugin.hive.metastore.HiveMetastoreFactory) AbstractConfigurationAwareModule(io.airlift.configuration.AbstractConfigurationAwareModule) HiveMetastore(io.trino.plugin.hive.metastore.HiveMetastore) DistributedQueryRunner(io.trino.testing.DistributedQueryRunner) ImmutableList(com.google.common.collect.ImmutableList) Binder(com.google.inject.Binder) DockerizedMinioDataLake(io.trino.plugin.deltalake.util.DockerizedMinioDataLake) Map(java.util.Map) ExportBinder.newExporter(org.weakref.jmx.guice.ExportBinder.newExporter) DefaultThriftMetastoreClientFactory(io.trino.plugin.hive.metastore.thrift.DefaultThriftMetastoreClientFactory) ConnectorFactory(io.trino.spi.connector.ConnectorFactory) OptionalBinder.newOptionalBinder(com.google.inject.multibindings.OptionalBinder.newOptionalBinder) TpchEntity(io.trino.tpch.TpchEntity) TpchTable(io.trino.tpch.TpchTable) StaticMetastoreLocator(io.trino.plugin.hive.metastore.thrift.StaticMetastoreLocator) AfterClass(org.testng.annotations.AfterClass) ConnectorIdentity(io.trino.spi.security.ConnectorIdentity) ImmutableMap(com.google.common.collect.ImmutableMap) ThriftHiveMetastore(io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastore) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) Reflection(com.google.common.reflect.Reflection) ThriftMetastoreAuthenticationModule(io.trino.plugin.hive.metastore.thrift.ThriftMetastoreAuthenticationModule) Scopes(com.google.inject.Scopes) RawHiveMetastoreFactory(io.trino.plugin.hive.metastore.RawHiveMetastoreFactory) String.format(java.lang.String.format) Plugin(io.trino.spi.Plugin) List(java.util.List) TestingSession.testSessionBuilder(io.trino.testing.TestingSession.testSessionBuilder) Provides(com.google.inject.Provides) DeltaLakeDockerizedMinioDataLake.createDockerizedMinioDataLakeForDeltaLake(io.trino.plugin.deltalake.DeltaLakeDockerizedMinioDataLake.createDockerizedMinioDataLakeForDeltaLake) MetastoreLocator(io.trino.plugin.hive.metastore.thrift.MetastoreLocator) ThriftMetastore(io.trino.plugin.hive.metastore.thrift.ThriftMetastore) ThriftMetastoreConfig(io.trino.plugin.hive.metastore.thrift.ThriftMetastoreConfig) Optional(java.util.Optional) ConfigBinder.configBinder(io.airlift.configuration.ConfigBinder.configBinder) Session(io.trino.Session) Connector(io.trino.spi.connector.Connector) DistributedQueryRunner(io.trino.testing.DistributedQueryRunner) RawHiveMetastoreFactory(io.trino.plugin.hive.metastore.RawHiveMetastoreFactory) StaticMetastoreLocator(io.trino.plugin.hive.metastore.thrift.StaticMetastoreLocator) MetastoreLocator(io.trino.plugin.hive.metastore.thrift.MetastoreLocator) TpchTable(io.trino.tpch.TpchTable) ThriftMetastoreConfig(io.trino.plugin.hive.metastore.thrift.ThriftMetastoreConfig) Binder(com.google.inject.Binder) OptionalBinder.newOptionalBinder(com.google.inject.multibindings.OptionalBinder.newOptionalBinder) ConfigBinder.configBinder(io.airlift.configuration.ConfigBinder.configBinder) ConnectorFactory(io.trino.spi.connector.ConnectorFactory) ConnectorContext(io.trino.spi.connector.ConnectorContext) AbstractConfigurationAwareModule(io.airlift.configuration.AbstractConfigurationAwareModule) ThriftMetastoreAuthenticationModule(io.trino.plugin.hive.metastore.thrift.ThriftMetastoreAuthenticationModule) BridgingHiveMetastoreFactory(io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastoreFactory) HiveMetastoreFactory(io.trino.plugin.hive.metastore.HiveMetastoreFactory) RawHiveMetastoreFactory(io.trino.plugin.hive.metastore.RawHiveMetastoreFactory) HiveMetastore(io.trino.plugin.hive.metastore.HiveMetastore) ThriftHiveMetastore(io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastore) ThriftMetastore(io.trino.plugin.hive.metastore.thrift.ThriftMetastore) ConnectorIdentity(io.trino.spi.security.ConnectorIdentity) Provides(com.google.inject.Provides) TpchEntity(io.trino.tpch.TpchEntity) ImmutableMap(com.google.common.collect.ImmutableMap) BridgingHiveMetastoreFactory(io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastoreFactory) Singleton(javax.inject.Singleton) StaticMetastoreConfig(io.trino.plugin.hive.metastore.thrift.StaticMetastoreConfig) Map(java.util.Map) ImmutableMap(com.google.common.collect.ImmutableMap) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) Session(io.trino.Session) Plugin(io.trino.spi.Plugin)

Example 20 with ConnectorIdentity

use of io.trino.spi.security.ConnectorIdentity in project trino by trinodb.

the class TestFileSystemCache method testFileSystemCache.

@Test
public void testFileSystemCache() throws IOException {
    HdfsEnvironment environment = new HdfsEnvironment(new HiveHdfsConfiguration(new HdfsConfigurationInitializer(new HdfsConfig()), ImmutableSet.of()), new HdfsConfig(), new ImpersonatingHdfsAuthentication(new SimpleHadoopAuthentication(), new SimpleUserNameProvider()));
    ConnectorIdentity userId = ConnectorIdentity.ofUser("user");
    ConnectorIdentity otherUserId = ConnectorIdentity.ofUser("other_user");
    FileSystem fs1 = getFileSystem(environment, userId);
    FileSystem fs2 = getFileSystem(environment, userId);
    assertSame(fs1, fs2);
    FileSystem fs3 = getFileSystem(environment, otherUserId);
    assertNotSame(fs1, fs3);
    FileSystem fs4 = getFileSystem(environment, otherUserId);
    assertSame(fs3, fs4);
    FileSystem.closeAll();
    FileSystem fs5 = getFileSystem(environment, userId);
    assertNotSame(fs5, fs1);
}
Also used : SimpleHadoopAuthentication(io.trino.plugin.hive.authentication.SimpleHadoopAuthentication) SimpleUserNameProvider(io.trino.plugin.hive.authentication.SimpleUserNameProvider) ImpersonatingHdfsAuthentication(io.trino.plugin.hive.authentication.ImpersonatingHdfsAuthentication) FileSystem(org.apache.hadoop.fs.FileSystem) ConnectorIdentity(io.trino.spi.security.ConnectorIdentity) Test(org.testng.annotations.Test)

Aggregations

ConnectorIdentity (io.trino.spi.security.ConnectorIdentity)21 TrinoException (io.trino.spi.TrinoException)16 Type (io.trino.spi.type.Type)12 Map (java.util.Map)11 Optional (java.util.Optional)11 ImmutableList (com.google.common.collect.ImmutableList)10 Connection (java.sql.Connection)10 SchemaTableName (io.trino.spi.connector.SchemaTableName)9 List (java.util.List)9 Set (java.util.Set)9 Objects.requireNonNull (java.util.Objects.requireNonNull)8 Inject (javax.inject.Inject)8 ImmutableMap (com.google.common.collect.ImmutableMap)7 ImmutableSet (com.google.common.collect.ImmutableSet)7 ImmutableSet.toImmutableSet (com.google.common.collect.ImmutableSet.toImmutableSet)7 NOT_SUPPORTED (io.trino.spi.StandardErrorCode.NOT_SUPPORTED)7 String.format (java.lang.String.format)7 Preconditions.checkArgument (com.google.common.base.Preconditions.checkArgument)6 ImmutableList.toImmutableList (com.google.common.collect.ImmutableList.toImmutableList)6 ConnectorSession (io.trino.spi.connector.ConnectorSession)6