Search in sources :

Example 1 with HetuMetastore

use of io.prestosql.spi.metastore.HetuMetastore in project hetu-core by openlookeng.

the class TestHeuristicIndexClient method testDeleteSelectedColumnsHelper.

@Test
public void testDeleteSelectedColumnsHelper() throws IOException {
    String tableName = "catalog.schema.UT_test";
    try (TempFolder folder = new TempFolder()) {
        folder.create();
        File tableFolder = new File(folder.getRoot().getPath(), tableName);
        assertTrue(tableFolder.mkdir());
        File columnFolder = new File(tableFolder, "test_column");
        assertTrue(columnFolder.mkdirs());
        File indexTypeFolder = new File(columnFolder, "BLOOM");
        assertTrue(indexTypeFolder.mkdirs());
        assertTrue(new File(indexTypeFolder, "testIndex.index").createNewFile());
        HetuFileSystemClient fs = new HetuLocalFileSystemClient(new LocalConfig(new Properties()), folder.getRoot().toPath());
        HetuMetastore testMetaStore = new HetuFsMetastore(new HetuFsMetastoreConfig().setHetuFileSystemMetastorePath(folder.getRoot().getPath()), fs);
        HeuristicIndexClient client = new HeuristicIndexClient(fs, testMetaStore, folder.getRoot().toPath());
        client.addIndexRecord(new CreateIndexMetadata("idx1", tableName, "BLOOM", 0L, Collections.singletonList(new Pair<>("test_column", VARCHAR)), Collections.emptyList(), new Properties(), "user", CreateIndexMetadata.Level.UNDEFINED));
        client.deleteIndex("idx1", Collections.emptyList());
        assertFalse(indexTypeFolder.exists());
    }
}
Also used : CreateIndexMetadata(io.prestosql.spi.connector.CreateIndexMetadata) HetuFsMetastoreConfig(io.hetu.core.metastore.hetufilesystem.HetuFsMetastoreConfig) LocalConfig(io.hetu.core.filesystem.LocalConfig) Properties(java.util.Properties) HetuFsMetastore(io.hetu.core.metastore.hetufilesystem.HetuFsMetastore) HetuMetastore(io.prestosql.spi.metastore.HetuMetastore) HetuFileSystemClient(io.prestosql.spi.filesystem.HetuFileSystemClient) TempFolder(io.hetu.core.common.filesystem.TempFolder) File(java.io.File) HetuLocalFileSystemClient(io.hetu.core.filesystem.HetuLocalFileSystemClient) Test(org.testng.annotations.Test)

Example 2 with HetuMetastore

use of io.prestosql.spi.metastore.HetuMetastore in project hetu-core by openlookeng.

the class TestIndexRecordManager method testDelete.

@Test
public void testDelete() throws IOException {
    try (TempFolder folder = new TempFolder()) {
        folder.create();
        HetuMetastore testMetaStore = new HetuFsMetastore(new HetuFsMetastoreConfig().setHetuFileSystemMetastorePath(folder.getRoot().getPath()), FILE_SYSTEM_CLIENT);
        IndexRecordManager indexRecordManager = new IndexRecordManager(testMetaStore);
        indexRecordManager.addIndexRecord("1", "testUser", "testCatalog.testSchema.testTable", new String[] { "testColumn" }, "minmax", 0L, Collections.emptyList(), Arrays.asList("cp=1"));
        indexRecordManager.addIndexRecord("2", "testUser", "testCatalog.testSchema.testTable", new String[] { "testColumn" }, "bloom", 0L, Collections.emptyList(), Arrays.asList("cp=1"));
        assertNotNull(indexRecordManager.lookUpIndexRecord("1"));
        assertEquals(indexRecordManager.getIndexRecords().size(), 2);
        // Delete 1
        indexRecordManager.deleteIndexRecord("1", Collections.emptyList());
        assertNull(indexRecordManager.lookUpIndexRecord("1"));
        assertNotNull(indexRecordManager.lookUpIndexRecord("2"));
        assertEquals(indexRecordManager.getIndexRecords().size(), 1);
        // Delete 1 again
        indexRecordManager.deleteIndexRecord("1", Collections.emptyList());
        assertNull(indexRecordManager.lookUpIndexRecord("1"));
        assertNotNull(indexRecordManager.lookUpIndexRecord("2"));
        assertEquals(indexRecordManager.getIndexRecords().size(), 1);
        // Delete 2
        indexRecordManager.deleteIndexRecord("2", Collections.emptyList());
        assertNull(indexRecordManager.lookUpIndexRecord("2"));
        assertEquals(indexRecordManager.getIndexRecords().size(), 0);
    }
}
Also used : HetuFsMetastoreConfig(io.hetu.core.metastore.hetufilesystem.HetuFsMetastoreConfig) TempFolder(io.hetu.core.common.filesystem.TempFolder) HetuFsMetastore(io.hetu.core.metastore.hetufilesystem.HetuFsMetastore) HetuMetastore(io.prestosql.spi.metastore.HetuMetastore) JdbcHetuMetastore(io.hetu.core.metastore.jdbc.JdbcHetuMetastore) Test(org.testng.annotations.Test)

Example 3 with HetuMetastore

use of io.prestosql.spi.metastore.HetuMetastore in project hetu-core by openlookeng.

the class TestIndexRecordManager method testIndexRecordAddLookUpHelper.

private void testIndexRecordAddLookUpHelper(String name, String user, String table, String[] columns, String indexType, List<String> indexProperties, List<String> partitions) throws IOException {
    try (TempFolder folder = new TempFolder()) {
        folder.create();
        HetuMetastore testMetaStore = new HetuFsMetastore(new HetuFsMetastoreConfig().setHetuFileSystemMetastorePath(folder.getRoot().getPath()), FILE_SYSTEM_CLIENT);
        IndexRecordManager indexRecordManager = new IndexRecordManager(testMetaStore);
        IndexRecord expected = new IndexRecord(name, user, table, columns, indexType, 0L, indexProperties, partitions);
        indexRecordManager.addIndexRecord(name, user, table, columns, indexType, 0L, indexProperties, partitions);
        IndexRecord actual1 = indexRecordManager.lookUpIndexRecord(name);
        assertNotNull(actual1);
        assertEquals(actual1, expected);
        IndexRecord actual2 = indexRecordManager.lookUpIndexRecord(table, columns, indexType);
        assertNotNull(actual2);
        assertEquals(actual2, expected);
    }
}
Also used : HetuFsMetastoreConfig(io.hetu.core.metastore.hetufilesystem.HetuFsMetastoreConfig) TempFolder(io.hetu.core.common.filesystem.TempFolder) HetuFsMetastore(io.hetu.core.metastore.hetufilesystem.HetuFsMetastore) IndexRecord(io.prestosql.spi.heuristicindex.IndexRecord) HetuMetastore(io.prestosql.spi.metastore.HetuMetastore) JdbcHetuMetastore(io.hetu.core.metastore.jdbc.JdbcHetuMetastore)

Example 4 with HetuMetastore

use of io.prestosql.spi.metastore.HetuMetastore in project hetu-core by openlookeng.

the class MemoryConnectorFactory method create.

@Override
public Connector create(String catalogName, Map<String, String> requiredConfig, ConnectorContext context) {
    requireNonNull(requiredConfig, "requiredConfig is null");
    if (context.getHetuMetastore() == null) {
        throw new IllegalStateException("HetuMetastore must be configured to use the Memory Connector. Please refer to HetuMetastore docs for details.");
    }
    try {
        // A plugin is not required to use Guice; it is just very convenient
        Bootstrap app = new Bootstrap(binder -> {
            binder.bind(PageSorter.class).toInstance(context.getPageSorter());
            binder.bind(FunctionMetadataManager.class).toInstance(context.getFunctionMetadataManager());
            binder.bind(StandardFunctionResolution.class).toInstance(context.getStandardFunctionResolution());
            binder.bind(TypeManager.class).toInstance(context.getTypeManager());
            binder.bind(RowExpressionService.class).toInstance(context.getRowExpressionService());
            binder.bind(DeterminismEvaluator.class).toInstance(context.getRowExpressionService().getDeterminismEvaluator());
            binder.bind(HetuMetastore.class).toInstance(context.getHetuMetastore());
        }, new MBeanModule(), new MBeanServerModule(), new JsonModule(), new MemoryModule(context.getTypeManager(), context.getNodeManager(), new PagesSerdeFactory(context.getBlockEncodingSerde(), false).createPagesSerde()));
        Injector injector = app.strictConfig().doNotInitializeLogging().setRequiredConfigurationProperties(requiredConfig).initialize();
        synchronized (MemoryThreadManager.class) {
            if (!MemoryThreadManager.isSharedThreadPoolInitilized()) {
                MemoryThreadManager.initSharedThreadPool(injector.getInstance(MemoryConfig.class).getThreadPoolSize());
            }
        }
        MemoryConnector memoryConnector = injector.getInstance(MemoryConnector.class);
        memoryConnector.scheduleRefreshJob();
        return memoryConnector;
    } catch (Exception e) {
        throwIfUnchecked(e);
        throw new RuntimeException(e);
    }
}
Also used : MBeanModule(org.weakref.jmx.guice.MBeanModule) DeterminismEvaluator(io.prestosql.spi.relation.DeterminismEvaluator) JsonModule(io.airlift.json.JsonModule) FunctionMetadataManager(io.prestosql.spi.function.FunctionMetadataManager) HetuMetastore(io.prestosql.spi.metastore.HetuMetastore) PagesSerdeFactory(io.hetu.core.transport.execution.buffer.PagesSerdeFactory) RowExpressionService(io.prestosql.spi.relation.RowExpressionService) MBeanServerModule(io.prestosql.plugin.base.jmx.MBeanServerModule) Injector(com.google.inject.Injector) Bootstrap(io.airlift.bootstrap.Bootstrap) PageSorter(io.prestosql.spi.PageSorter) TypeManager(io.prestosql.spi.type.TypeManager) StandardFunctionResolution(io.prestosql.spi.function.StandardFunctionResolution)

Example 5 with HetuMetastore

use of io.prestosql.spi.metastore.HetuMetastore in project hetu-core by openlookeng.

the class CubeManager method getMetaStore.

public synchronized Optional<CubeMetaStore> getMetaStore(String name) {
    CubeMetaStore cubeMetaStore = this.cubeMetaStores.get(name);
    if (cubeMetaStore != null) {
        return Optional.of(cubeMetaStore);
    }
    CubeProvider cubeProvider = this.cubeProviders.get(name);
    if (cubeProvider != null) {
        HetuMetastore hetuMetastore = metaStoreManager.getHetuMetastore();
        if (hetuMetastore == null) {
            return Optional.empty();
        }
        cubeMetaStore = cubeProvider.getCubeMetaStore(metaStoreManager.getHetuMetastore(), properties);
        this.cubeMetaStores.put(name, cubeMetaStore);
        return Optional.of(cubeMetaStore);
    }
    return Optional.empty();
}
Also used : CubeMetaStore(io.hetu.core.spi.cube.io.CubeMetaStore) CubeProvider(io.prestosql.spi.cube.CubeProvider) HetuMetastore(io.prestosql.spi.metastore.HetuMetastore)

Aggregations

HetuMetastore (io.prestosql.spi.metastore.HetuMetastore)5 TempFolder (io.hetu.core.common.filesystem.TempFolder)3 HetuFsMetastore (io.hetu.core.metastore.hetufilesystem.HetuFsMetastore)3 HetuFsMetastoreConfig (io.hetu.core.metastore.hetufilesystem.HetuFsMetastoreConfig)3 JdbcHetuMetastore (io.hetu.core.metastore.jdbc.JdbcHetuMetastore)2 Test (org.testng.annotations.Test)2 Injector (com.google.inject.Injector)1 Bootstrap (io.airlift.bootstrap.Bootstrap)1 JsonModule (io.airlift.json.JsonModule)1 HetuLocalFileSystemClient (io.hetu.core.filesystem.HetuLocalFileSystemClient)1 LocalConfig (io.hetu.core.filesystem.LocalConfig)1 CubeMetaStore (io.hetu.core.spi.cube.io.CubeMetaStore)1 PagesSerdeFactory (io.hetu.core.transport.execution.buffer.PagesSerdeFactory)1 MBeanServerModule (io.prestosql.plugin.base.jmx.MBeanServerModule)1 PageSorter (io.prestosql.spi.PageSorter)1 CreateIndexMetadata (io.prestosql.spi.connector.CreateIndexMetadata)1 CubeProvider (io.prestosql.spi.cube.CubeProvider)1 HetuFileSystemClient (io.prestosql.spi.filesystem.HetuFileSystemClient)1 FunctionMetadataManager (io.prestosql.spi.function.FunctionMetadataManager)1 StandardFunctionResolution (io.prestosql.spi.function.StandardFunctionResolution)1