Search in sources :

Example 6 with OrcFileInfo

use of io.trino.plugin.raptor.legacy.storage.OrcFileRewriter.OrcFileInfo in project trino by trinodb.

the class RaptorStorageManager method rewriteShard.

@VisibleForTesting
Collection<Slice> rewriteShard(long transactionId, OptionalInt bucketNumber, UUID shardUuid, BitSet rowsToDelete) {
    if (rowsToDelete.isEmpty()) {
        return ImmutableList.of();
    }
    UUID newShardUuid = UUID.randomUUID();
    File input = storageService.getStorageFile(shardUuid);
    File output = storageService.getStagingFile(newShardUuid);
    OrcFileInfo info = rewriteFile(input, output, rowsToDelete);
    long rowCount = info.getRowCount();
    if (rowCount == 0) {
        return shardDelta(shardUuid, Optional.empty());
    }
    shardRecorder.recordCreatedShard(transactionId, newShardUuid);
    // submit for backup and wait until it finishes
    getFutureValue(backupManager.submit(newShardUuid, output));
    Set<String> nodes = ImmutableSet.of(nodeId);
    long uncompressedSize = info.getUncompressedSize();
    ShardInfo shard = createShardInfo(newShardUuid, bucketNumber, output, nodes, rowCount, uncompressedSize);
    writeShard(newShardUuid);
    return shardDelta(shardUuid, Optional.of(shard));
}
Also used : OrcFileInfo(io.trino.plugin.raptor.legacy.storage.OrcFileRewriter.OrcFileInfo) UUID(java.util.UUID) File(java.io.File) ShardInfo(io.trino.plugin.raptor.legacy.metadata.ShardInfo) VisibleForTesting(com.google.common.annotations.VisibleForTesting)

Aggregations

OrcFileInfo (io.trino.plugin.raptor.legacy.storage.OrcFileRewriter.OrcFileInfo)6 File (java.io.File)6 ArrayType (io.trino.spi.type.ArrayType)5 DecimalType (io.trino.spi.type.DecimalType)5 Type (io.trino.spi.type.Type)5 VarcharType.createVarcharType (io.trino.spi.type.VarcharType.createVarcharType)5 BitSet (java.util.BitSet)5 Test (org.testng.annotations.Test)5 Page (io.trino.spi.Page)3 OrcDataSource (io.trino.orc.OrcDataSource)2 OrcRecordReader (io.trino.orc.OrcRecordReader)2 OrcTestingUtil.fileOrcDataSource (io.trino.plugin.raptor.legacy.storage.OrcTestingUtil.fileOrcDataSource)2 Block (io.trino.spi.block.Block)2 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 ShardInfo (io.trino.plugin.raptor.legacy.metadata.ShardInfo)1 BigDecimal (java.math.BigDecimal)1 UUID (java.util.UUID)1