Search in sources :

Example 16 with Lz4Compressor

use of io.airlift.compress.lz4.Lz4Compressor in project hadoop by apache.

the class TestLz4CompressorDecompressor method testCompressorSetInputAIOBException.

//test on ArrayIndexOutOfBoundsException in {@code compressor.setInput()}
@Test
public void testCompressorSetInputAIOBException() {
    try {
        Lz4Compressor compressor = new Lz4Compressor();
        compressor.setInput(new byte[] {}, -5, 10);
        fail("testCompressorSetInputAIOBException error !!!");
    } catch (ArrayIndexOutOfBoundsException ex) {
    // expected
    } catch (Exception ex) {
        fail("testCompressorSetInputAIOBException ex error !!!");
    }
}
Also used : Lz4Compressor(org.apache.hadoop.io.compress.lz4.Lz4Compressor) IOException(java.io.IOException) Test(org.junit.Test)

Example 17 with Lz4Compressor

use of io.airlift.compress.lz4.Lz4Compressor in project hadoop by apache.

the class TestLz4CompressorDecompressor method testSetInputWithBytesSizeMoreThenDefaultLz4CompressorByfferSize.

// test Lz4Compressor compressor.compress()  
@Test
public void testSetInputWithBytesSizeMoreThenDefaultLz4CompressorByfferSize() {
    int BYTES_SIZE = 1024 * 64 + 1;
    try {
        Lz4Compressor compressor = new Lz4Compressor();
        byte[] bytes = generate(BYTES_SIZE);
        assertTrue("needsInput error !!!", compressor.needsInput());
        compressor.setInput(bytes, 0, bytes.length);
        byte[] emptyBytes = new byte[BYTES_SIZE];
        int csize = compressor.compress(emptyBytes, 0, bytes.length);
        assertTrue("testSetInputWithBytesSizeMoreThenDefaultLz4CompressorByfferSize error !!!", csize != 0);
    } catch (Exception ex) {
        fail("testSetInputWithBytesSizeMoreThenDefaultLz4CompressorByfferSize ex error !!!");
    }
}
Also used : Lz4Compressor(org.apache.hadoop.io.compress.lz4.Lz4Compressor) IOException(java.io.IOException) Test(org.junit.Test)

Example 18 with Lz4Compressor

use of io.airlift.compress.lz4.Lz4Compressor in project hadoop by apache.

the class TestLz4CompressorDecompressor method testCompressorCompressNullPointerException.

//test on NullPointerException in {@code compressor.compress()}  
@Test
public void testCompressorCompressNullPointerException() {
    try {
        Lz4Compressor compressor = new Lz4Compressor();
        byte[] bytes = generate(1024 * 6);
        compressor.setInput(bytes, 0, bytes.length);
        compressor.compress(null, 0, 0);
        fail("testCompressorCompressNullPointerException error !!!");
    } catch (NullPointerException ex) {
    // expected
    } catch (Exception e) {
        fail("testCompressorCompressNullPointerException ex error !!!");
    }
}
Also used : Lz4Compressor(org.apache.hadoop.io.compress.lz4.Lz4Compressor) IOException(java.io.IOException) Test(org.junit.Test)

Example 19 with Lz4Compressor

use of io.airlift.compress.lz4.Lz4Compressor in project hadoop by apache.

the class TestLz4CompressorDecompressor method testCompressorSetInputNullPointerException.

//test on NullPointerException in {@code compressor.setInput()} 
@Test
public void testCompressorSetInputNullPointerException() {
    try {
        Lz4Compressor compressor = new Lz4Compressor();
        compressor.setInput(null, 0, 10);
        fail("testCompressorSetInputNullPointerException error !!!");
    } catch (NullPointerException ex) {
    // expected
    } catch (Exception e) {
        fail("testCompressorSetInputNullPointerException ex error !!!");
    }
}
Also used : Lz4Compressor(org.apache.hadoop.io.compress.lz4.Lz4Compressor) IOException(java.io.IOException) Test(org.junit.Test)

Example 20 with Lz4Compressor

use of io.airlift.compress.lz4.Lz4Compressor in project vespa by vespa-engine.

the class Compressor method compress.

/**
 * Compresses some data
 *
 * @param requestedCompression the desired compression type, which will be used if the data is deemed suitable.
 *                             Not all the existing types are actually supported.
 * @param data the data to compress. This array is only read by this method.
 * @param uncompressedSize uncompressedSize the size in bytes of the data array. If this is not present, it is
 *                         assumed that the size is the same as the data array size, i.e that it is completely
 *                         filled with uncompressed data.
 * @return the compression result
 * @throws IllegalArgumentException if the compression type is not supported
 */
public Compression compress(CompressionType requestedCompression, byte[] data, Optional<Integer> uncompressedSize) {
    switch(requestedCompression) {
        case NONE:
            data = uncompressedSize.isPresent() ? Arrays.copyOf(data, uncompressedSize.get()) : data;
            return new Compression(CompressionType.NONE, data.length, data);
        case LZ4:
            int dataSize = uncompressedSize.isPresent() ? uncompressedSize.get() : data.length;
            if (dataSize < compressMinSizeBytes)
                return new Compression(CompressionType.INCOMPRESSIBLE, dataSize, data);
            LZ4Compressor compressor = level < 7 ? factory.fastCompressor() : factory.highCompressor();
            byte[] compressedData = compressor.compress(data, 0, dataSize);
            if (compressedData.length + 8 >= dataSize * compressionThresholdFactor)
                return new Compression(CompressionType.INCOMPRESSIBLE, dataSize, data);
            return new Compression(CompressionType.LZ4, dataSize, compressedData);
        default:
            throw new IllegalArgumentException(requestedCompression + " is not supported");
    }
}
Also used : LZ4Compressor(net.jpountz.lz4.LZ4Compressor)

Aggregations

IOException (java.io.IOException)11 LZ4Compressor (net.jpountz.lz4.LZ4Compressor)10 Lz4Compressor (org.apache.hadoop.io.compress.lz4.Lz4Compressor)10 Test (org.junit.Test)10 Lz4Decompressor (org.apache.hadoop.io.compress.lz4.Lz4Decompressor)5 ByteBuffer (java.nio.ByteBuffer)3 Lz4Compressor (io.airlift.compress.lz4.Lz4Compressor)2 ByteArrayOutputStream (java.io.ByteArrayOutputStream)2 DataInputStream (java.io.DataInputStream)2 DataOutputStream (java.io.DataOutputStream)2 LZ4Exception (net.jpountz.lz4.LZ4Exception)2 BlockCompressorStream (org.apache.hadoop.io.compress.BlockCompressorStream)2 BlockDecompressorStream (org.apache.hadoop.io.compress.BlockDecompressorStream)2 DeflateCompressor (com.facebook.presto.orc.zlib.DeflateCompressor)1 InflateDecompressor (com.facebook.presto.orc.zlib.InflateDecompressor)1 ZstdJniCompressor (com.facebook.presto.orc.zstd.ZstdJniCompressor)1 ZstdJniDecompressor (com.facebook.presto.orc.zstd.ZstdJniDecompressor)1 PrestoException (com.facebook.presto.spi.PrestoException)1 PageCompressor (com.facebook.presto.spi.page.PageCompressor)1 PageDecompressor (com.facebook.presto.spi.page.PageDecompressor)1