Search in sources :

Example 1 with ZstdCompressorOutputStream

use of org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream in project flink by apache.

the class GenericCsvInputFormatTest method createTempZStandardFile.

private FileInputSplit createTempZStandardFile(String content) throws IOException {
    File tempFile = File.createTempFile("test_contents", "tmp.zst");
    tempFile.deleteOnExit();
    DataOutputStream dos = new DataOutputStream(new ZstdCompressorOutputStream(new FileOutputStream(tempFile)));
    dos.writeBytes(content);
    dos.close();
    return new FileInputSplit(0, new Path(tempFile.toURI().toString()), 0, tempFile.length(), new String[] { "localhost" });
}
Also used : ZstdCompressorOutputStream(org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream) Path(org.apache.flink.core.fs.Path) FileInputSplit(org.apache.flink.core.fs.FileInputSplit) DataOutputStream(java.io.DataOutputStream) FileOutputStream(java.io.FileOutputStream) DelimitedInputFormatTest.createTempFile(org.apache.flink.api.common.io.DelimitedInputFormatTest.createTempFile) File(java.io.File)

Example 2 with ZstdCompressorOutputStream

use of org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream in project druid by druid-io.

the class CompressionUtilsTest method testDecompressZstd.

@Test
public void testDecompressZstd() throws IOException {
    final File tmpDir = temporaryFolder.newFolder("testDecompressZstd");
    final File zstdFile = new File(tmpDir, testFile.getName() + ".zst");
    Assert.assertFalse(zstdFile.exists());
    try (final OutputStream out = new ZstdCompressorOutputStream(new FileOutputStream(zstdFile))) {
        ByteStreams.copy(new FileInputStream(testFile), out);
    }
    try (final InputStream inputStream = CompressionUtils.decompress(new FileInputStream(zstdFile), zstdFile.getName())) {
        assertGoodDataStream(inputStream);
    }
}
Also used : ZstdCompressorOutputStream(org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream) GZIPInputStream(java.util.zip.GZIPInputStream) FilterInputStream(java.io.FilterInputStream) ByteArrayInputStream(java.io.ByteArrayInputStream) FileInputStream(java.io.FileInputStream) CountingInputStream(com.google.common.io.CountingInputStream) InputStream(java.io.InputStream) ZipOutputStream(java.util.zip.ZipOutputStream) ByteArrayOutputStream(java.io.ByteArrayOutputStream) BZip2CompressorOutputStream(org.apache.commons.compress.compressors.bzip2.BZip2CompressorOutputStream) ZstdCompressorOutputStream(org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream) OutputStream(java.io.OutputStream) FramedSnappyCompressorOutputStream(org.apache.commons.compress.compressors.snappy.FramedSnappyCompressorOutputStream) XZCompressorOutputStream(org.apache.commons.compress.compressors.xz.XZCompressorOutputStream) FileOutputStream(java.io.FileOutputStream) FilterOutputStream(java.io.FilterOutputStream) GZIPOutputStream(java.util.zip.GZIPOutputStream) FileOutputStream(java.io.FileOutputStream) File(java.io.File) FileInputStream(java.io.FileInputStream) Test(org.junit.Test)

Aggregations

File (java.io.File)2 FileOutputStream (java.io.FileOutputStream)2 ZstdCompressorOutputStream (org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream)2 CountingInputStream (com.google.common.io.CountingInputStream)1 ByteArrayInputStream (java.io.ByteArrayInputStream)1 ByteArrayOutputStream (java.io.ByteArrayOutputStream)1 DataOutputStream (java.io.DataOutputStream)1 FileInputStream (java.io.FileInputStream)1 FilterInputStream (java.io.FilterInputStream)1 FilterOutputStream (java.io.FilterOutputStream)1 InputStream (java.io.InputStream)1 OutputStream (java.io.OutputStream)1 GZIPInputStream (java.util.zip.GZIPInputStream)1 GZIPOutputStream (java.util.zip.GZIPOutputStream)1 ZipOutputStream (java.util.zip.ZipOutputStream)1 BZip2CompressorOutputStream (org.apache.commons.compress.compressors.bzip2.BZip2CompressorOutputStream)1 FramedSnappyCompressorOutputStream (org.apache.commons.compress.compressors.snappy.FramedSnappyCompressorOutputStream)1 XZCompressorOutputStream (org.apache.commons.compress.compressors.xz.XZCompressorOutputStream)1 DelimitedInputFormatTest.createTempFile (org.apache.flink.api.common.io.DelimitedInputFormatTest.createTempFile)1 FileInputSplit (org.apache.flink.core.fs.FileInputSplit)1