Search in sources :

Example 1 with HdfsOrcDataSource

use of com.huawei.boostkit.omnidata.model.datasource.hdfs.HdfsOrcDataSource in project boostkit-bigdata by kunpengcompute.

the class DataIoAdapter method initDataSource.

private DataSource initDataSource(PageCandidate pageCandidate) throws UnsupportedOperationException {
    DataSource dataSource;
    String fileFormat = pageCandidate.getFileFormat();
    Long fileStartPos = pageCandidate.getStartPos();
    Long fileLen = pageCandidate.getSplitLen();
    if ("ORC".equalsIgnoreCase(fileFormat)) {
        dataSource = new HdfsOrcDataSource(filePath, fileStartPos, fileLen, false);
    } else if ("PARQUET".equalsIgnoreCase(fileFormat)) {
        dataSource = new HdfsParquetDataSource(filePath, fileStartPos, fileLen, false);
    } else {
        throw new UnsupportedOperationException("unsupported data format : " + fileFormat);
    }
    return dataSource;
}
Also used : HdfsParquetDataSource(com.huawei.boostkit.omnidata.model.datasource.hdfs.HdfsParquetDataSource) OptionalLong(java.util.OptionalLong) HdfsOrcDataSource(com.huawei.boostkit.omnidata.model.datasource.hdfs.HdfsOrcDataSource) HdfsParquetDataSource(com.huawei.boostkit.omnidata.model.datasource.hdfs.HdfsParquetDataSource) HdfsOrcDataSource(com.huawei.boostkit.omnidata.model.datasource.hdfs.HdfsOrcDataSource) DataSource(com.huawei.boostkit.omnidata.model.datasource.DataSource)

Aggregations

DataSource (com.huawei.boostkit.omnidata.model.datasource.DataSource)1 HdfsOrcDataSource (com.huawei.boostkit.omnidata.model.datasource.hdfs.HdfsOrcDataSource)1 HdfsParquetDataSource (com.huawei.boostkit.omnidata.model.datasource.hdfs.HdfsParquetDataSource)1 OptionalLong (java.util.OptionalLong)1