Examples with CompressionCodec org.apache.hadoop.io.compress.CompressionCodec used on opensource projects

Example 56 with CompressionCodec

use of org.apache.hadoop.io.compress.CompressionCodec in project hive by apache.

the class Utilities method createRCFileWriter.

/**
 * Create a RCFile output stream based on job configuration Uses user supplied compression flag
 * (rather than obtaining it from the Job Configuration).
 *
 * @param jc
 *          Job configuration
 * @param fs
 *          File System to create file in
 * @param file
 *          Path to be created
 * @return output stream over the created rcfile
 */
public static RCFile.Writer createRCFileWriter(JobConf jc, FileSystem fs, Path file, boolean isCompressed, Progressable progressable) throws IOException {
    CompressionCodec codec = null;
    if (isCompressed) {
        Class<?> codecClass = FileOutputFormat.getOutputCompressorClass(jc, DefaultCodec.class);
        codec = (CompressionCodec) ReflectionUtil.newInstance(codecClass, jc);
    }
    return new RCFile.Writer(fs, jc, file, progressable, codec);
}

Example 57 with CompressionCodec

use of org.apache.hadoop.io.compress.CompressionCodec in project hive by apache.

the class RCFileOutputFormat method getRecordWriter.

/**
 * {@inheritDoc}
 */
@Override
public RecordWriter<WritableComparable, BytesRefArrayWritable> getRecordWriter(FileSystem ignored, JobConf job, String name, Progressable progress) throws IOException {
    Path outputPath = getWorkOutputPath(job);
    FileSystem fs = outputPath.getFileSystem(job);
    Path file = new Path(outputPath, name);
    CompressionCodec codec = null;
    if (getCompressOutput(job)) {
        Class<?> codecClass = getOutputCompressorClass(job, DefaultCodec.class);
        codec = (CompressionCodec) ReflectionUtils.newInstance(codecClass, job);
    }
    final RCFile.Writer out = new RCFile.Writer(fs, job, file, progress, codec);
    return new RecordWriter<WritableComparable, BytesRefArrayWritable>() {

        @Override
        public void close(Reporter reporter) throws IOException {
            out.close();
        }

        @Override
        public void write(WritableComparable key, BytesRefArrayWritable value) throws IOException {
            out.append(value);
        }
    };
}

Example 58 with CompressionCodec

use of org.apache.hadoop.io.compress.CompressionCodec in project hive by apache.

the class PerformTestRCFileAndSeqFile method main.

public static void main(String[] args) throws Exception {
    int count = 1000;
    String file = null;
    try {
        for (int i = 0; i < args.length; ++i) {
            // parse command line
            if (args[i] == null) {
                continue;
            } else if (args[i].equals("-count")) {
                count = Integer.parseInt(args[++i]);
            } else {
                // file is required parameter
                file = args[i];
            }
        }
        // change it to choose the appropriate file system
        boolean isLocalFS = true;
        PerformTestRCFileAndSeqFile testcase = new PerformTestRCFileAndSeqFile(isLocalFS, file);
        // change these parameters
        boolean checkCorrect = true;
        CompressionCodec codec = new DefaultCodec();
        testcase.columnMaxSize = 30;
        // testcase.testWithColumnNumber(count, 2, checkCorrect, codec);
        // testcase.testWithColumnNumber(count, 10, checkCorrect, codec);
        // testcase.testWithColumnNumber(count, 25, checkCorrect, codec);
        testcase.testWithColumnNumber(count, 40, checkCorrect, codec);
    // testcase.testWithColumnNumber(count, 50, checkCorrect, codec);
    // testcase.testWithColumnNumber(count, 80, checkCorrect, codec);
    } finally {
    }
}

Example 59 with CompressionCodec

use of org.apache.hadoop.io.compress.CompressionCodec in project shifu by ShifuML.

the class ShifuFileUtils method getCompressInputStream.

private static InputStream getCompressInputStream(FSDataInputStream fdis, Path path) throws IOException {
    String name = path.getName();
    if (name.toLowerCase().endsWith(".gz")) {
        return new GZIPInputStream(fdis);
    } else if (name.toLowerCase().endsWith(".bz2")) {
        return new BZip2CompressorInputStream(fdis);
    } else if (name.toLowerCase().endsWith(".snappy")) {
        Configuration conf = new Configuration();
        CompressionCodecFactory ccf = new CompressionCodecFactory(conf);
        CompressionCodec codec = ccf.getCodecByClassName(SnappyCodec.class.getName());
        return codec.createInputStream(fdis);
    } else {
        return fdis;
    }
}

Example 60 with CompressionCodec

use of org.apache.hadoop.io.compress.CompressionCodec in project shifu by ShifuML.

the class HdfsGlobalFile method openPartFileAsStream.

private InputStream openPartFileAsStream(FileStatus fileStatus) throws IOException {
    CompressionCodecFactory compressionFactory = new CompressionCodecFactory(new Configuration());
    InputStream is = null;
    FileSystem fs = ShifuFileUtils.getFileSystemBySourceType(sourceType);
    CompressionCodec codec = compressionFactory.getCodec(fileStatus.getPath());
    if (codec != null) {
        is = codec.createInputStream(fs.open(fileStatus.getPath()));
    } else {
        is = fs.open(fileStatus.getPath());
    }
    return is;
}