Examples with ParquetReadOptions org.apache.parquet.ParquetReadOptions used on opensource projects

Example 1 with ParquetReadOptions

use of org.apache.parquet.ParquetReadOptions in project parquet-mr by apache.

the class ParquetFileReader method readFooter.

/**
 * Reads the meta data block in the footer of the file using provided input stream
 * @param file a {@link InputFile} to read
 * @param filter the filter to apply to row groups
 * @return the metadata blocks in the footer
 * @throws IOException if an error occurs while reading the file
 * @deprecated will be removed in 2.0.0;
 *             use {@link ParquetFileReader#open(InputFile, ParquetReadOptions)}
 */
@Deprecated
public static final ParquetMetadata readFooter(InputFile file, MetadataFilter filter) throws IOException {
    ParquetReadOptions options;
    if (file instanceof HadoopInputFile) {
        HadoopInputFile hadoopFile = (HadoopInputFile) file;
        options = HadoopReadOptions.builder(hadoopFile.getConfiguration(), hadoopFile.getPath()).withMetadataFilter(filter).build();
    } else {
        options = ParquetReadOptions.builder().withMetadataFilter(filter).build();
    }
    try (SeekableInputStream in = file.newStream()) {
        return readFooter(file, options, in);
    }
}

Example 2 with ParquetReadOptions

use of org.apache.parquet.ParquetReadOptions in project parquet-mr by apache.

the class ColumnEncryptorTest method getParquetMetadata.

private ParquetMetadata getParquetMetadata(FileDecryptionProperties decryptionProperties) throws IOException {
    ParquetMetadata metaData;
    ParquetReadOptions readOptions = ParquetReadOptions.builder().withDecryption(decryptionProperties).build();
    InputFile file = HadoopInputFile.fromPath(new Path(outputFile), conf);
    try (SeekableInputStream in = file.newStream()) {
        metaData = ParquetFileReader.readFooter(file, readOptions, in);
    }
    return metaData;
}

Example 3 with ParquetReadOptions

use of org.apache.parquet.ParquetReadOptions in project pxf by greenplum-db.

the class ParquetWriteTest method validateFooter.

private MessageType validateFooter(Path parquetFile, int numCols, int numRows) throws IOException {
    ParquetReadOptions parquetReadOptions = HadoopReadOptions.builder(configuration).build();
    HadoopInputFile inputFile = HadoopInputFile.fromPath(parquetFile, configuration);
    try (ParquetFileReader parquetFileReader = ParquetFileReader.open(inputFile, parquetReadOptions)) {
        FileMetaData metadata = parquetFileReader.getFileMetaData();
        ParquetMetadata readFooter = parquetFileReader.getFooter();
        // one block
        assertEquals(1, readFooter.getBlocks().size());
        BlockMetaData block0 = readFooter.getBlocks().get(0);
        // one column
        assertEquals(numCols, block0.getColumns().size());
        // 10 rows in this block
        assertEquals(numRows, block0.getRowCount());
        ColumnChunkMetaData column0 = block0.getColumns().get(0);
        assertEquals(CompressionCodecName.SNAPPY, column0.getCodec());
        return metadata.getSchema();
    }
}

Example 4 with ParquetReadOptions

use of org.apache.parquet.ParquetReadOptions in project drill by apache.

the class TestParquetReaderConfig method testReadOptions.

@Test
public void testReadOptions() {
    // set enableStringsSignedMinMax to true
    ParquetReaderConfig readerConfig = new ParquetReaderConfig(false, false, false, true, true);
    ParquetReadOptions readOptions = readerConfig.toReadOptions();
    assertTrue(readOptions.useSignedStringMinMax());
    // set enableStringsSignedMinMax to false
    readerConfig = new ParquetReaderConfig(false, false, false, true, false);
    readOptions = readerConfig.toReadOptions();
    assertFalse(readOptions.useSignedStringMinMax());
}

Example 5 with ParquetReadOptions

use of org.apache.parquet.ParquetReadOptions in project parquet-mr by apache.

the class ColumnEncryptorTest method verifyOffsetIndexes.

private void verifyOffsetIndexes() throws IOException {
    ParquetReadOptions readOptions = HadoopReadOptions.builder(conf).withDecryption(EncDecProperties.getFileDecryptionProperties()).build();
    try (TransParquetFileReader inReader = createFileReader(inputFile.getFileName());
        TransParquetFileReader outReader = createFileReader(outputFile)) {
        ParquetMetadata inMetaData = getMetadata(readOptions, inputFile.getFileName(), inReader);
        ParquetMetadata outMetaData = getMetadata(readOptions, outputFile, outReader);
        compareOffsetIndexes(inReader, outReader, inMetaData, outMetaData);
    }
}