Search in sources :

Example 16 with OrcReaderOptions

use of io.trino.orc.OrcReaderOptions in project trino by trinodb.

the class TestOriginalFilesUtils method testGetPrecedingRowCountSingleFile.

@Test
public void testGetPrecedingRowCountSingleFile() {
    List<OriginalFileInfo> originalFileInfoList = new ArrayList<>();
    originalFileInfoList.add(new OriginalFileInfo("000001_0", 730));
    long rowCountResult = OriginalFilesUtils.getPrecedingRowCount(originalFileInfoList, new Path(tablePath + "/000001_0"), HDFS_ENVIRONMENT, SESSION.getIdentity(), new OrcReaderOptions(), config, new FileFormatDataSourceStats());
    assertEquals(rowCountResult, 0, "Original file should have 0 as the starting row count");
}
Also used : Path(org.apache.hadoop.fs.Path) OriginalFileInfo(io.trino.plugin.hive.AcidInfo.OriginalFileInfo) OrcReaderOptions(io.trino.orc.OrcReaderOptions) ArrayList(java.util.ArrayList) Test(org.testng.annotations.Test)

Example 17 with OrcReaderOptions

use of io.trino.orc.OrcReaderOptions in project trino by trinodb.

the class TestOrcDeleteDeltaPageSource method testReadingDeletedRows.

@Test
public void testReadingDeletedRows() throws Exception {
    File deleteDeltaFile = new File(Resources.getResource("fullacid_delete_delta_test/delete_delta_0000004_0000004_0000/bucket_00000").toURI());
    OrcDeleteDeltaPageSourceFactory pageSourceFactory = new OrcDeleteDeltaPageSourceFactory(new OrcReaderOptions(), ConnectorIdentity.ofUser("test"), new JobConf(new Configuration(false)), HDFS_ENVIRONMENT, new FileFormatDataSourceStats());
    ConnectorPageSource pageSource = pageSourceFactory.createPageSource(new Path(deleteDeltaFile.toURI()), deleteDeltaFile.length()).orElseThrow();
    MaterializedResult materializedRows = MaterializedResult.materializeSourceDataStream(SESSION, pageSource, ImmutableList.of(BIGINT, INTEGER, BIGINT));
    assertEquals(materializedRows.getRowCount(), 1);
    AcidOutputFormat.Options bucketOptions = new AcidOutputFormat.Options(new Configuration(false)).bucket(0);
    assertEquals(materializedRows.getMaterializedRows().get(0), new MaterializedRow(5, 2L, BucketCodec.V1.encode(bucketOptions), 0L));
}
Also used : Path(org.apache.hadoop.fs.Path) OrcReaderOptions(io.trino.orc.OrcReaderOptions) Configuration(org.apache.hadoop.conf.Configuration) FileFormatDataSourceStats(io.trino.plugin.hive.FileFormatDataSourceStats) ConnectorPageSource(io.trino.spi.connector.ConnectorPageSource) MaterializedResult(io.trino.testing.MaterializedResult) File(java.io.File) JobConf(org.apache.hadoop.mapred.JobConf) MaterializedRow(io.trino.testing.MaterializedRow) AcidOutputFormat(org.apache.hadoop.hive.ql.io.AcidOutputFormat) Test(org.testng.annotations.Test)

Aggregations

OrcReaderOptions (io.trino.orc.OrcReaderOptions)17 ConnectorSession (io.trino.spi.connector.ConnectorSession)10 Test (org.testng.annotations.Test)10 Path (org.apache.hadoop.fs.Path)8 FileFormatDataSourceStats (io.trino.plugin.hive.FileFormatDataSourceStats)7 List (java.util.List)7 ImmutableList.toImmutableList (com.google.common.collect.ImmutableList.toImmutableList)6 ConnectorPageSource (io.trino.spi.connector.ConnectorPageSource)6 Type (io.trino.spi.type.Type)6 IOException (java.io.IOException)6 ArrayList (java.util.ArrayList)6 Objects.requireNonNull (java.util.Objects.requireNonNull)6 Optional (java.util.Optional)6 Collectors.toList (java.util.stream.Collectors.toList)6 Configuration (org.apache.hadoop.conf.Configuration)6 ImmutableList (com.google.common.collect.ImmutableList)5 ImmutableMap (com.google.common.collect.ImmutableMap)5 OrcDataSource (io.trino.orc.OrcDataSource)5 OrcDataSourceId (io.trino.orc.OrcDataSourceId)5 OrcWriterOptions (io.trino.orc.OrcWriterOptions)5