Search in sources :

Example 1 with CompositeTap

use of cascading.tap.CompositeTap in project parquet-mr by apache.

the class ParquetTupleScheme method readSchema.

private MessageType readSchema(FlowProcess<? extends JobConf> flowProcess, Tap tap) {
    try {
        Hfs hfs;
        if (tap instanceof CompositeTap)
            hfs = (Hfs) ((CompositeTap) tap).getChildTaps().next();
        else
            hfs = (Hfs) tap;
        List<Footer> footers = getFooters(flowProcess, hfs);
        if (footers.isEmpty()) {
            throw new TapException("Could not read Parquet metadata at " + hfs.getPath());
        } else {
            return footers.get(0).getParquetMetadata().getFileMetaData().getSchema();
        }
    } catch (IOException e) {
        throw new TapException(e);
    }
}
Also used : Hfs(cascading.tap.hadoop.Hfs) CompositeTap(cascading.tap.CompositeTap) Footer(org.apache.parquet.hadoop.Footer) TapException(cascading.tap.TapException) IOException(java.io.IOException)

Example 2 with CompositeTap

use of cascading.tap.CompositeTap in project parquet-mr by apache.

the class ParquetTupleScheme method readSchema.

private MessageType readSchema(FlowProcess<JobConf> flowProcess, Tap tap) {
    try {
        Hfs hfs;
        if (tap instanceof CompositeTap)
            hfs = (Hfs) ((CompositeTap) tap).getChildTaps().next();
        else
            hfs = (Hfs) tap;
        List<Footer> footers = getFooters(flowProcess, hfs);
        if (footers.isEmpty()) {
            throw new TapException("Could not read Parquet metadata at " + hfs.getPath());
        } else {
            return footers.get(0).getParquetMetadata().getFileMetaData().getSchema();
        }
    } catch (IOException e) {
        throw new TapException(e);
    }
}
Also used : Hfs(cascading.tap.hadoop.Hfs) CompositeTap(cascading.tap.CompositeTap) Footer(org.apache.parquet.hadoop.Footer) TapException(cascading.tap.TapException) IOException(java.io.IOException)

Aggregations

CompositeTap (cascading.tap.CompositeTap)2 TapException (cascading.tap.TapException)2 Hfs (cascading.tap.hadoop.Hfs)2 IOException (java.io.IOException)2 Footer (org.apache.parquet.hadoop.Footer)2