Search in sources :

Example 31 with DrillBuf

use of io.netty.buffer.DrillBuf in project drill by axbaretto.

the class BufferedDirectBufInputStream method close.

public void close() throws IOException {
    DrillBuf buffer;
    InputStream inp;
    synchronized (this) {
        try {
            if ((inp = in) != null) {
                in = null;
                inp.close();
            }
        } catch (IOException e) {
            throw e;
        } finally {
            if ((buffer = this.internalBuffer) != null) {
                this.internalBuffer = null;
                buffer.release();
            }
            if ((buffer = this.tempBuffer) != null) {
                this.tempBuffer = null;
                buffer.release();
            }
        }
    }
}
Also used : InputStream(java.io.InputStream) IOException(java.io.IOException) DrillBuf(io.netty.buffer.DrillBuf)

Example 32 with DrillBuf

use of io.netty.buffer.DrillBuf in project drill by axbaretto.

the class TestAllocators method testTransfer.

@Test
public void testTransfer() throws Exception {
    final Properties props = new Properties() {

        {
            put(RootAllocatorFactory.TOP_LEVEL_MAX_ALLOC, "1049600");
        }
    };
    final DrillConfig config = DrillConfig.create(props);
    BufferAllocator a = RootAllocatorFactory.newRoot(config);
    BufferAllocator a1 = a.newChildAllocator("a1", 0, Integer.MAX_VALUE);
    BufferAllocator a2 = a.newChildAllocator("a2", 0, Integer.MAX_VALUE);
    DrillBuf buf1 = a1.buffer(1_000_000);
    DrillBuf buf2 = a2.buffer(1_000);
    DrillBuf buf3 = buf1.transferOwnership(a2).buffer;
    buf1.release();
    buf2.release();
    buf3.release();
    a1.close();
    a2.close();
    a.close();
}
Also used : DrillConfig(org.apache.drill.common.config.DrillConfig) Properties(java.util.Properties) DrillBuf(io.netty.buffer.DrillBuf) MemoryTest(org.apache.drill.categories.MemoryTest) DrillTest(org.apache.drill.test.DrillTest) Test(org.junit.Test)

Example 33 with DrillBuf

use of io.netty.buffer.DrillBuf in project drill by axbaretto.

the class TestLenientAllocation method testLenientLimit.

@Test
public void testLenientLimit() {
    LogFixtureBuilder logBuilder = LogFixture.builder().logger(Accountant.class, Level.WARN);
    try (LogFixture logFixture = logBuilder.build()) {
        // Test can't run without assertions
        assertTrue(AssertionUtil.isAssertionsEnabled());
        // Create a child allocator
        BufferAllocator allocator = fixture.allocator().newChildAllocator("test", 10 * ONE_MEG, 128 * ONE_MEG);
        ((Accountant) allocator).forceLenient();
        // Allocate most of the available memory
        DrillBuf buf1 = allocator.buffer(64 * ONE_MEG);
        // Oops, we did our math wrong; allocate too large a buffer.
        DrillBuf buf2 = allocator.buffer(128 * ONE_MEG);
        try {
            allocator.buffer(64 * ONE_MEG);
            fail();
        } catch (OutOfMemoryException e) {
        // Expected
        }
        // Clean up
        buf1.close();
        buf2.close();
        allocator.close();
    }
}
Also used : LogFixture(org.apache.drill.test.LogFixture) Accountant(org.apache.drill.exec.memory.Accountant) LogFixtureBuilder(org.apache.drill.test.LogFixture.LogFixtureBuilder) OutOfMemoryException(org.apache.drill.exec.exception.OutOfMemoryException) BufferAllocator(org.apache.drill.exec.memory.BufferAllocator) DrillBuf(io.netty.buffer.DrillBuf) SubOperatorTest(org.apache.drill.test.SubOperatorTest) Test(org.junit.Test)

Example 34 with DrillBuf

use of io.netty.buffer.DrillBuf in project drill by axbaretto.

the class TestLenientAllocation method testStrict.

/**
 * Test that the allocator is normally strict in debug mode.
 */
@Test
public void testStrict() {
    LogFixtureBuilder logBuilder = LogFixture.builder().logger(Accountant.class, Level.WARN);
    try (LogFixture logFixture = logBuilder.build()) {
        // Test can't run without assertions
        assertTrue(AssertionUtil.isAssertionsEnabled());
        // Create a child allocator
        BufferAllocator allocator = fixture.allocator().newChildAllocator("test", 10 * 1024, 128 * 1024);
        // Allocate most of the available memory
        DrillBuf buf1 = allocator.buffer(64 * 1024);
        try {
            allocator.buffer(128 * 1024);
            fail();
        } catch (OutOfMemoryException e) {
        // Expected
        }
        // Clean up
        buf1.close();
        allocator.close();
    }
}
Also used : LogFixture(org.apache.drill.test.LogFixture) LogFixtureBuilder(org.apache.drill.test.LogFixture.LogFixtureBuilder) OutOfMemoryException(org.apache.drill.exec.exception.OutOfMemoryException) BufferAllocator(org.apache.drill.exec.memory.BufferAllocator) DrillBuf(io.netty.buffer.DrillBuf) SubOperatorTest(org.apache.drill.test.SubOperatorTest) Test(org.junit.Test)

Example 35 with DrillBuf

use of io.netty.buffer.DrillBuf in project drill by axbaretto.

the class TestLenientAllocation method testLenient.

/**
 * Use a test-time hack to force the allocator to be lenient,
 * regardless of whether we are in debug mode or not.
 */
@Test
public void testLenient() {
    LogFixtureBuilder logBuilder = LogFixture.builder().logger(Accountant.class, Level.WARN);
    try (LogFixture logFixture = logBuilder.build()) {
        // Test can't run without assertions
        assertTrue(AssertionUtil.isAssertionsEnabled());
        // Create a child allocator
        BufferAllocator allocator = fixture.allocator().newChildAllocator("test", 10 * 1024, 128 * 1024);
        ((Accountant) allocator).forceLenient();
        // Allocate most of the available memory
        DrillBuf buf1 = allocator.buffer(64 * 1024);
        // Oops, we did our math wrong; allocate too large a buffer.
        DrillBuf buf2 = allocator.buffer(128 * 1024);
        assertEquals(192 * 1024, allocator.getAllocatedMemory());
        // We keep making mistakes.
        DrillBuf buf3 = allocator.buffer(32 * 1024);
        // Right up to the hard limit
        DrillBuf buf4 = allocator.buffer(32 * 1024);
        assertEquals(256 * 1024, allocator.getAllocatedMemory());
        try {
            allocator.buffer(8);
            fail();
        } catch (OutOfMemoryException e) {
        // Expected
        }
        // Recover from our excesses
        buf2.close();
        buf3.close();
        buf4.close();
        assertEquals(64 * 1024, allocator.getAllocatedMemory());
        // We're back in the good graces of the allocator,
        // can allocate more.
        DrillBuf buf5 = allocator.buffer(8);
        // Clean up
        buf1.close();
        buf5.close();
        allocator.close();
    }
}
Also used : LogFixture(org.apache.drill.test.LogFixture) Accountant(org.apache.drill.exec.memory.Accountant) LogFixtureBuilder(org.apache.drill.test.LogFixture.LogFixtureBuilder) OutOfMemoryException(org.apache.drill.exec.exception.OutOfMemoryException) BufferAllocator(org.apache.drill.exec.memory.BufferAllocator) DrillBuf(io.netty.buffer.DrillBuf) SubOperatorTest(org.apache.drill.test.SubOperatorTest) Test(org.junit.Test)

Aggregations

DrillBuf (io.netty.buffer.DrillBuf)187 Test (org.junit.Test)63 MemoryTest (org.apache.drill.categories.MemoryTest)38 SelectionVector4 (org.apache.drill.exec.record.selection.SelectionVector4)22 ValueVector (org.apache.drill.exec.vector.ValueVector)18 BaseTest (org.apache.drill.test.BaseTest)18 MaterializedField (org.apache.drill.exec.record.MaterializedField)16 IOException (java.io.IOException)13 VectorTest (org.apache.drill.categories.VectorTest)13 OutOfMemoryException (org.apache.drill.exec.exception.OutOfMemoryException)13 ExecTest (org.apache.drill.exec.ExecTest)11 BufferAllocator (org.apache.drill.exec.memory.BufferAllocator)11 VectorContainer (org.apache.drill.exec.record.VectorContainer)10 Stopwatch (com.google.common.base.Stopwatch)9 ByteBuffer (java.nio.ByteBuffer)9 BatchSchema (org.apache.drill.exec.record.BatchSchema)9 UnlikelyTest (org.apache.drill.categories.UnlikelyTest)8 UserBitShared (org.apache.drill.exec.proto.UserBitShared)8 SerializedField (org.apache.drill.exec.proto.UserBitShared.SerializedField)8 DrillTest (org.apache.drill.test.DrillTest)8