Search in sources :

Example 31 with ConnectorMetadata

use of com.facebook.presto.spi.connector.ConnectorMetadata in project presto by prestodb.

the class AbstractTestHiveClientS3 method testGetRecordsS3.

@Test
public void testGetRecordsS3() throws Exception {
    try (Transaction transaction = newTransaction()) {
        ConnectorMetadata metadata = transaction.getMetadata();
        ConnectorSession session = newSession();
        ConnectorTableHandle table = getTableHandle(metadata, tableS3);
        List<ColumnHandle> columnHandles = ImmutableList.copyOf(metadata.getColumnHandles(session, table).values());
        Map<String, Integer> columnIndex = indexColumns(columnHandles);
        List<ConnectorTableLayoutResult> tableLayoutResults = metadata.getTableLayouts(session, table, new Constraint<>(TupleDomain.all(), bindings -> true), Optional.empty());
        HiveTableLayoutHandle layoutHandle = (HiveTableLayoutHandle) getOnlyElement(tableLayoutResults).getTableLayout().getHandle();
        assertEquals(layoutHandle.getPartitions().get().size(), 1);
        ConnectorSplitSource splitSource = splitManager.getSplits(transaction.getTransactionHandle(), session, layoutHandle);
        long sum = 0;
        for (ConnectorSplit split : getAllSplits(splitSource)) {
            try (ConnectorPageSource pageSource = pageSourceProvider.createPageSource(transaction.getTransactionHandle(), session, split, columnHandles)) {
                MaterializedResult result = materializeSourceDataStream(session, pageSource, getTypes(columnHandles));
                for (MaterializedRow row : result) {
                    sum += (Long) row.getField(columnIndex.get("t_bigint"));
                }
            }
        }
        assertEquals(sum, 78300);
    }
}
Also used : HiveTestUtils.getDefaultHiveDataStreamFactories(com.facebook.presto.hive.HiveTestUtils.getDefaultHiveDataStreamFactories) ConnectorSplitSource(com.facebook.presto.spi.ConnectorSplitSource) HiveTestUtils.getDefaultHiveFileWriterFactories(com.facebook.presto.hive.HiveTestUtils.getDefaultHiveFileWriterFactories) FileSystem(org.apache.hadoop.fs.FileSystem) TypeRegistry(com.facebook.presto.type.TypeRegistry) Test(org.testng.annotations.Test) AbstractTestHiveClient.getAllSplits(com.facebook.presto.hive.AbstractTestHiveClient.getAllSplits) BIGINT(com.facebook.presto.spi.type.BigintType.BIGINT) MaterializedResult.materializeSourceDataStream(com.facebook.presto.testing.MaterializedResult.materializeSourceDataStream) Preconditions.checkArgument(com.google.common.base.Preconditions.checkArgument) SchemaTableName(com.facebook.presto.spi.SchemaTableName) ExtendedHiveMetastore(com.facebook.presto.hive.metastore.ExtendedHiveMetastore) BoundedExecutor(io.airlift.concurrent.BoundedExecutor) Map(java.util.Map) ConnectorPageSink(com.facebook.presto.spi.ConnectorPageSink) Path(org.apache.hadoop.fs.Path) ConnectorPageSourceProvider(com.facebook.presto.spi.connector.ConnectorPageSourceProvider) ENGLISH(java.util.Locale.ENGLISH) Assert.assertFalse(org.testng.Assert.assertFalse) AbstractTestHiveClient.filterNonHiddenColumnHandles(com.facebook.presto.hive.AbstractTestHiveClient.filterNonHiddenColumnHandles) ConnectorSplitManager(com.facebook.presto.spi.connector.ConnectorSplitManager) ImmutableMap(com.google.common.collect.ImmutableMap) PrincipalPrivileges(com.facebook.presto.hive.metastore.PrincipalPrivileges) BeforeClass(org.testng.annotations.BeforeClass) Collection(java.util.Collection) UUID(java.util.UUID) String.format(java.lang.String.format) ConnectorSession(com.facebook.presto.spi.ConnectorSession) TupleDomain(com.facebook.presto.spi.predicate.TupleDomain) List(java.util.List) ColumnMetadata(com.facebook.presto.spi.ColumnMetadata) BridgingHiveMetastore(com.facebook.presto.hive.metastore.BridgingHiveMetastore) TYPE_MANAGER(com.facebook.presto.hive.HiveTestUtils.TYPE_MANAGER) Optional(java.util.Optional) NoHdfsAuthentication(com.facebook.presto.hive.authentication.NoHdfsAuthentication) JsonCodec(io.airlift.json.JsonCodec) ConnectorMetadata(com.facebook.presto.spi.connector.ConnectorMetadata) Table(com.facebook.presto.hive.metastore.Table) Slice(io.airlift.slice.Slice) HiveTransaction(com.facebook.presto.hive.AbstractTestHiveClient.HiveTransaction) Database(com.facebook.presto.hive.metastore.Database) HiveTestUtils.getTypes(com.facebook.presto.hive.HiveTestUtils.getTypes) MoreExecutors.newDirectExecutorService(com.google.common.util.concurrent.MoreExecutors.newDirectExecutorService) Assert.assertEquals(org.testng.Assert.assertEquals) ConnectorTableHandle(com.facebook.presto.spi.ConnectorTableHandle) ConnectorOutputTableHandle(com.facebook.presto.spi.ConnectorOutputTableHandle) ImmutableList(com.google.common.collect.ImmutableList) Threads.daemonThreadsNamed(io.airlift.concurrent.Threads.daemonThreadsNamed) ConnectorPageSinkProvider(com.facebook.presto.spi.connector.ConnectorPageSinkProvider) ImmutableMultimap(com.google.common.collect.ImmutableMultimap) AbstractTestHiveClient.createTableProperties(com.facebook.presto.hive.AbstractTestHiveClient.createTableProperties) ExecutorService(java.util.concurrent.ExecutorService) ConnectorTableMetadata(com.facebook.presto.spi.ConnectorTableMetadata) AfterClass(org.testng.annotations.AfterClass) CachingHiveMetastore(com.facebook.presto.hive.metastore.CachingHiveMetastore) Transaction(com.facebook.presto.hive.AbstractTestHiveClient.Transaction) Throwables(com.google.common.base.Throwables) Constraint(com.facebook.presto.spi.Constraint) Iterables.getOnlyElement(com.google.common.collect.Iterables.getOnlyElement) TestingConnectorSession(com.facebook.presto.testing.TestingConnectorSession) MoreFutures.getFutureValue(io.airlift.concurrent.MoreFutures.getFutureValue) HostAndPort(com.google.common.net.HostAndPort) ConnectorSplit(com.facebook.presto.spi.ConnectorSplit) ConnectorTableLayoutResult(com.facebook.presto.spi.ConnectorTableLayoutResult) MaterializedResult(com.facebook.presto.testing.MaterializedResult) Assertions.assertEqualsIgnoreOrder(io.airlift.testing.Assertions.assertEqualsIgnoreOrder) ConnectorPageSource(com.facebook.presto.spi.ConnectorPageSource) TableNotFoundException(com.facebook.presto.spi.TableNotFoundException) ColumnHandle(com.facebook.presto.spi.ColumnHandle) Executors.newCachedThreadPool(java.util.concurrent.Executors.newCachedThreadPool) MaterializedRow(com.facebook.presto.testing.MaterializedRow) AbstractTestHiveClient.filterNonHiddenColumnMetadata(com.facebook.presto.hive.AbstractTestHiveClient.filterNonHiddenColumnMetadata) Assert.assertTrue(org.testng.Assert.assertTrue) HadoopFileStatus.isDirectory(com.facebook.presto.hadoop.HadoopFileStatus.isDirectory) GroupByHashPageIndexerFactory(com.facebook.presto.GroupByHashPageIndexerFactory) ThriftHiveMetastore(com.facebook.presto.hive.metastore.ThriftHiveMetastore) JoinCompiler(com.facebook.presto.sql.gen.JoinCompiler) HiveTestUtils.getDefaultHiveRecordCursorProvider(com.facebook.presto.hive.HiveTestUtils.getDefaultHiveRecordCursorProvider) ColumnHandle(com.facebook.presto.spi.ColumnHandle) ConnectorSplitSource(com.facebook.presto.spi.ConnectorSplitSource) ConnectorPageSource(com.facebook.presto.spi.ConnectorPageSource) ConnectorTableHandle(com.facebook.presto.spi.ConnectorTableHandle) HiveTransaction(com.facebook.presto.hive.AbstractTestHiveClient.HiveTransaction) Transaction(com.facebook.presto.hive.AbstractTestHiveClient.Transaction) ConnectorTableLayoutResult(com.facebook.presto.spi.ConnectorTableLayoutResult) ConnectorSession(com.facebook.presto.spi.ConnectorSession) TestingConnectorSession(com.facebook.presto.testing.TestingConnectorSession) ConnectorMetadata(com.facebook.presto.spi.connector.ConnectorMetadata) MaterializedResult(com.facebook.presto.testing.MaterializedResult) ConnectorSplit(com.facebook.presto.spi.ConnectorSplit) MaterializedRow(com.facebook.presto.testing.MaterializedRow) Test(org.testng.annotations.Test)

Example 32 with ConnectorMetadata

use of com.facebook.presto.spi.connector.ConnectorMetadata in project presto by prestodb.

the class AccumuloConnector method getMetadata.

@Override
public ConnectorMetadata getMetadata(ConnectorTransactionHandle transactionHandle) {
    ConnectorMetadata metadata = transactions.get(transactionHandle);
    checkArgument(metadata != null, "no such transaction: %s", transactionHandle);
    return metadata;
}
Also used : ConnectorMetadata(com.facebook.presto.spi.connector.ConnectorMetadata)

Example 33 with ConnectorMetadata

use of com.facebook.presto.spi.connector.ConnectorMetadata in project presto by prestodb.

the class HiveConnector method getMetadata.

@Override
public ConnectorMetadata getMetadata(ConnectorTransactionHandle transaction) {
    ConnectorMetadata metadata = transactionManager.get(transaction);
    checkArgument(metadata != null, "no such transaction: %s", transaction);
    return new ClassLoaderSafeConnectorMetadata(metadata, classLoader);
}
Also used : ClassLoaderSafeConnectorMetadata(com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorMetadata) ConnectorMetadata(com.facebook.presto.spi.connector.ConnectorMetadata) ClassLoaderSafeConnectorMetadata(com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorMetadata)

Example 34 with ConnectorMetadata

use of com.facebook.presto.spi.connector.ConnectorMetadata in project presto by prestodb.

the class HiveTransactionManager method put.

public void put(ConnectorTransactionHandle transactionHandle, ConnectorMetadata metadata) {
    ConnectorMetadata previousValue = transactions.putIfAbsent(transactionHandle, metadata);
    checkState(previousValue == null);
}
Also used : ConnectorMetadata(com.facebook.presto.spi.connector.ConnectorMetadata)

Example 35 with ConnectorMetadata

use of com.facebook.presto.spi.connector.ConnectorMetadata in project presto by prestodb.

the class MetadataManager method createView.

@Override
public void createView(Session session, QualifiedObjectName viewName, String viewData, boolean replace) {
    CatalogMetadata catalogMetadata = getCatalogMetadataForWrite(session, viewName.getCatalogName());
    ConnectorId connectorId = catalogMetadata.getConnectorId();
    ConnectorMetadata metadata = catalogMetadata.getMetadata();
    metadata.createView(session.toConnectorSession(connectorId), viewName.asSchemaTableName(), viewData, replace);
}
Also used : ConnectorMetadata(com.facebook.presto.spi.connector.ConnectorMetadata) ConnectorId(com.facebook.presto.connector.ConnectorId)

Aggregations

ConnectorMetadata (com.facebook.presto.spi.connector.ConnectorMetadata)100 ConnectorSession (com.facebook.presto.spi.ConnectorSession)49 Test (org.testng.annotations.Test)48 ConnectorTableHandle (com.facebook.presto.spi.ConnectorTableHandle)42 TestingConnectorSession (com.facebook.presto.testing.TestingConnectorSession)39 ConnectorId (com.facebook.presto.connector.ConnectorId)38 ColumnHandle (com.facebook.presto.spi.ColumnHandle)33 ConnectorTableMetadata (com.facebook.presto.spi.ConnectorTableMetadata)33 SchemaTableName (com.facebook.presto.spi.SchemaTableName)31 MaterializedResult (com.facebook.presto.testing.MaterializedResult)28 ColumnMetadata (com.facebook.presto.spi.ColumnMetadata)26 ConnectorOutputTableHandle (com.facebook.presto.spi.ConnectorOutputTableHandle)25 ConnectorInsertTableHandle (com.facebook.presto.spi.ConnectorInsertTableHandle)24 ConnectorPageSink (com.facebook.presto.spi.ConnectorPageSink)24 ConnectorPageSource (com.facebook.presto.spi.ConnectorPageSource)24 PrestoException (com.facebook.presto.spi.PrestoException)24 Path (org.apache.hadoop.fs.Path)24 Constraint (com.facebook.presto.spi.Constraint)23 SchemaTablePrefix (com.facebook.presto.spi.SchemaTablePrefix)23 MaterializedRow (com.facebook.presto.testing.MaterializedRow)23