Search in sources :

Example 1 with OpenLooKengDeserializer

use of com.huawei.boostkit.omnidata.decode.impl.OpenLooKengDeserializer in project boostkit-bigdata by kunpengcompute.

the class ParquetPageSourceFactory method createParquetPushDownPageSource.

public HivePushDownPageSource createParquetPushDownPageSource(Path path, long start, long length, com.huawei.boostkit.omnidata.model.Predicate predicate) {
    AggregatedMemoryContext systemMemoryUsage = newSimpleAggregatedMemoryContext();
    Properties transProperties = new Properties();
    transProperties.put(OMNIDATA_CLIENT_TARGET_LIST, omniDataServerTarget);
    DataSource parquetPushDownDataSource = new com.huawei.boostkit.omnidata.model.datasource.hdfs.HdfsParquetDataSource(path.toString(), start, length, false);
    TaskSource readTaskInfo = new TaskSource(parquetPushDownDataSource, predicate, TaskSource.ONE_MEGABYTES);
    DataReader<Page> dataReader = DataReaderFactory.create(transProperties, readTaskInfo, new OpenLooKengDeserializer());
    return new HivePushDownPageSource(dataReader, systemMemoryUsage);
}
Also used : HdfsParquetDataSource.buildHdfsParquetDataSource(io.prestosql.plugin.hive.parquet.HdfsParquetDataSource.buildHdfsParquetDataSource) HivePushDownPageSource(io.prestosql.plugin.hive.HivePushDownPageSource) Page(io.prestosql.spi.Page) HiveSessionProperties(io.prestosql.plugin.hive.HiveSessionProperties) Properties(java.util.Properties) AggregatedMemoryContext(io.prestosql.memory.context.AggregatedMemoryContext) AggregatedMemoryContext.newSimpleAggregatedMemoryContext(io.prestosql.memory.context.AggregatedMemoryContext.newSimpleAggregatedMemoryContext) TaskSource(com.huawei.boostkit.omnidata.model.TaskSource) HdfsParquetDataSource.buildHdfsParquetDataSource(io.prestosql.plugin.hive.parquet.HdfsParquetDataSource.buildHdfsParquetDataSource) DataSource(com.huawei.boostkit.omnidata.model.datasource.DataSource) ParquetDataSource(io.prestosql.parquet.ParquetDataSource) OpenLooKengDeserializer(com.huawei.boostkit.omnidata.decode.impl.OpenLooKengDeserializer)

Example 2 with OpenLooKengDeserializer

use of com.huawei.boostkit.omnidata.decode.impl.OpenLooKengDeserializer in project boostkit-bigdata by kunpengcompute.

the class OrcPageSourceFactory method createOrcPushDownPageSource.

public HivePushDownPageSource createOrcPushDownPageSource(Path path, long start, long length, Predicate predicate) {
    AggregatedMemoryContext systemMemoryUsage = newSimpleAggregatedMemoryContext();
    Properties transProperties = new Properties();
    transProperties.put(OMNIDATA_CLIENT_TARGET_LIST, omniDataServerTarget);
    DataSource orcPushDownDataSource = new com.huawei.boostkit.omnidata.model.datasource.hdfs.HdfsOrcDataSource(path.toString(), start, length, false);
    TaskSource readTaskInfo = new TaskSource(orcPushDownDataSource, predicate, TaskSource.ONE_MEGABYTES);
    DataReader<Page> dataReader = DataReaderFactory.create(transProperties, readTaskInfo, new OpenLooKengDeserializer());
    return new HivePushDownPageSource(dataReader, systemMemoryUsage);
}
Also used : HivePushDownPageSource(io.prestosql.plugin.hive.HivePushDownPageSource) Page(io.prestosql.spi.Page) OrcCacheProperties(io.prestosql.orc.OrcCacheProperties) Properties(java.util.Properties) HiveSessionProperties(io.prestosql.plugin.hive.HiveSessionProperties) AggregatedMemoryContext(io.prestosql.memory.context.AggregatedMemoryContext) AggregatedMemoryContext.newSimpleAggregatedMemoryContext(io.prestosql.memory.context.AggregatedMemoryContext.newSimpleAggregatedMemoryContext) TaskSource(com.huawei.boostkit.omnidata.model.TaskSource) DataSource(com.huawei.boostkit.omnidata.model.datasource.DataSource) OrcDataSource(io.prestosql.orc.OrcDataSource) OpenLooKengDeserializer(com.huawei.boostkit.omnidata.decode.impl.OpenLooKengDeserializer)

Aggregations

OpenLooKengDeserializer (com.huawei.boostkit.omnidata.decode.impl.OpenLooKengDeserializer)2 TaskSource (com.huawei.boostkit.omnidata.model.TaskSource)2 DataSource (com.huawei.boostkit.omnidata.model.datasource.DataSource)2 AggregatedMemoryContext (io.prestosql.memory.context.AggregatedMemoryContext)2 AggregatedMemoryContext.newSimpleAggregatedMemoryContext (io.prestosql.memory.context.AggregatedMemoryContext.newSimpleAggregatedMemoryContext)2 HivePushDownPageSource (io.prestosql.plugin.hive.HivePushDownPageSource)2 HiveSessionProperties (io.prestosql.plugin.hive.HiveSessionProperties)2 Page (io.prestosql.spi.Page)2 Properties (java.util.Properties)2 OrcCacheProperties (io.prestosql.orc.OrcCacheProperties)1 OrcDataSource (io.prestosql.orc.OrcDataSource)1 ParquetDataSource (io.prestosql.parquet.ParquetDataSource)1 HdfsParquetDataSource.buildHdfsParquetDataSource (io.prestosql.plugin.hive.parquet.HdfsParquetDataSource.buildHdfsParquetDataSource)1