Example 11 with TaskWrapper
use of org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.TaskWrapper in project hive by apache.
the class TestTaskExecutorService method testPreemptionQueueComparator.
@Test(timeout = 5000)
public void testPreemptionQueueComparator() throws InterruptedException {
TaskWrapper r1 = createTaskWrapper(createSubmitWorkRequestProto(1, 2, 100, 200, false), false, 100000);
TaskWrapper r2 = createTaskWrapper(createSubmitWorkRequestProto(2, 4, 200, 300, false), false, 100000);
TaskWrapper r3 = createTaskWrapper(createSubmitWorkRequestProto(3, 6, 300, 400, false), false, 1000000);
TaskWrapper r4 = createTaskWrapper(createSubmitWorkRequestProto(4, 8, 400, 500, false), false, 1000000);
TaskWrapper r5 = createTaskWrapper(createSubmitWorkRequestProto(5, 2, 100, 200, false), true, 1000000);
TaskWrapper r6 = createTaskWrapper(createSubmitWorkRequestProto(6, 8, 400, 500, true), false, 1000000);
BlockingQueue<TaskWrapper> queue = new PriorityBlockingQueue<>(6, new TaskExecutorService.PreemptionQueueComparator());
queue.offer(r6);
queue.offer(r5);
queue.offer(r1);
assertEquals(r1, queue.peek());
queue.offer(r2);
assertEquals(r1, queue.peek());
queue.offer(r3);
assertEquals(r1, queue.peek());
queue.offer(r4);
assertEquals(r1, queue.take());
assertEquals(r2, queue.take());
assertEquals(r3, queue.take());
assertEquals(r4, queue.take());
assertEquals(r5, queue.take());
assertEquals(r6, queue.take());
}
Also used :
TaskExecutorTestHelpers.createTaskWrapper(org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorTestHelpers.createTaskWrapper)
TaskWrapper(org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.TaskWrapper)
PriorityBlockingQueue(java.util.concurrent.PriorityBlockingQueue)
Test(org.junit.Test)
Example 12 with TaskWrapper
use of org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.TaskWrapper in project hive by apache.
the class TestFirstInFirstOutComparator method testWaitQueueComparatorCanFinish.
@Test(timeout = 60000)
public void testWaitQueueComparatorCanFinish() throws InterruptedException {
// Test that only the fixed property (...ForQueue) is used in order determination, not the dynamic call.
TaskWrapper r1 = createTaskWrapper(createSubmitWorkRequestProto(1, 1, 0, 10, 100, 2), true, false, 100000);
TaskWrapper r2 = createTaskWrapper(createSubmitWorkRequestProto(2, 1, 0, 10, 100, 1), false, true, 100000);
TaskWrapper r3 = createTaskWrapper(createSubmitWorkRequestProto(3, 1, 0, 10, 100, 5), true, true, 100000);
EvictingPriorityBlockingQueue<TaskWrapper> queue = new EvictingPriorityBlockingQueue<TaskWrapper>(new FirstInFirstOutComparator(), 4);
assertNull(queue.offer(r1, 0));
assertNull(queue.offer(r2, 0));
assertNull(queue.offer(r3, 0));
assertEquals(r2, queue.take());
assertEquals(r3, queue.take());
assertEquals(r1, queue.take());
}
Also used :
TaskExecutorTestHelpers.createTaskWrapper(org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorTestHelpers.createTaskWrapper)
TaskWrapper(org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.TaskWrapper)
EvictingPriorityBlockingQueue(org.apache.hadoop.hive.llap.daemon.impl.EvictingPriorityBlockingQueue)
Test(org.junit.Test)
Example 13 with TaskWrapper
use of org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.TaskWrapper in project hive by apache.
the class TestFirstInFirstOutComparator method testWaitQueueComparator.
@Test(timeout = 60000)
public void testWaitQueueComparator() throws InterruptedException {
TaskWrapper r1 = createTaskWrapper(createRequest(1, 2, 5, 100), false, 100000);
TaskWrapper r2 = createTaskWrapper(createRequest(2, 4, 4, 200), false, 100000);
TaskWrapper r3 = createTaskWrapper(createRequest(3, 6, 3, 300), false, 1000000);
TaskWrapper r4 = createTaskWrapper(createRequest(4, 8, 2, 400), false, 1000000);
TaskWrapper r5 = createTaskWrapper(createRequest(5, 10, 1, 500), false, 1000000);
EvictingPriorityBlockingQueue<TaskWrapper> queue = new EvictingPriorityBlockingQueue<>(new FirstInFirstOutComparator(), 4);
assertNull(queue.offer(r1, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r2, 0));
assertEquals(r2, queue.peek());
assertNull(queue.offer(r3, 0));
assertEquals(r3, queue.peek());
assertNull(queue.offer(r4, 0));
assertEquals(r4, queue.peek());
// this offer will be accepted and r1 evicted
assertEquals(r1, queue.offer(r5, 0));
assertEquals(r5, queue.take());
assertEquals(r4, queue.take());
assertEquals(r3, queue.take());
assertEquals(r2, queue.take());
r1 = createTaskWrapper(createRequest(1, 2, 5, 100), true, 100000);
r2 = createTaskWrapper(createRequest(2, 4, 4, 200), true, 100000);
r3 = createTaskWrapper(createRequest(3, 6, 3, 300), true, 1000000);
r4 = createTaskWrapper(createRequest(4, 8, 2, 400), true, 1000000);
r5 = createTaskWrapper(createRequest(5, 10, 1, 500), true, 1000000);
queue = new EvictingPriorityBlockingQueue<TaskWrapper>(new FirstInFirstOutComparator(), 4);
assertNull(queue.offer(r1, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r2, 0));
assertEquals(r2, queue.peek());
assertNull(queue.offer(r3, 0));
assertEquals(r3, queue.peek());
assertNull(queue.offer(r4, 0));
assertEquals(r4, queue.peek());
// this offer will be accpeted and r1 evicted
assertEquals(r1, queue.offer(r5, 0));
assertEquals(r5, queue.take());
assertEquals(r4, queue.take());
assertEquals(r3, queue.take());
assertEquals(r2, queue.take());
r1 = createTaskWrapper(createRequest(1, 1, 5, 100), true, 100000);
r2 = createTaskWrapper(createRequest(2, 1, 4, 200), false, 100000);
r3 = createTaskWrapper(createRequest(3, 1, 3, 300), true, 1000000);
r4 = createTaskWrapper(createRequest(4, 1, 2, 400), false, 1000000);
r5 = createTaskWrapper(createRequest(5, 10, 1, 500), true, 1000000);
queue = new EvictingPriorityBlockingQueue<TaskWrapper>(new FirstInFirstOutComparator(), 4);
assertNull(queue.offer(r1, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r2, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r3, 0));
assertEquals(r3, queue.peek());
assertNull(queue.offer(r4, 0));
assertEquals(r3, queue.peek());
// offer accepted and r2 gets evicted
assertEquals(r2, queue.offer(r5, 0));
assertEquals(r5, queue.take());
assertEquals(r3, queue.take());
assertEquals(r1, queue.take());
assertEquals(r4, queue.take());
r1 = createTaskWrapper(createRequest(1, 2, 5, 100), true, 100000);
r2 = createTaskWrapper(createRequest(2, 4, 4, 200), false, 100000);
r3 = createTaskWrapper(createRequest(3, 6, 3, 300), true, 1000000);
r4 = createTaskWrapper(createRequest(4, 8, 2, 400), false, 1000000);
r5 = createTaskWrapper(createRequest(5, 10, 1, 500), true, 1000000);
queue = new EvictingPriorityBlockingQueue<TaskWrapper>(new FirstInFirstOutComparator(), 4);
assertNull(queue.offer(r1, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r2, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r3, 0));
assertEquals(r3, queue.peek());
assertNull(queue.offer(r4, 0));
assertEquals(r3, queue.peek());
// offer accepted and r2 gets evicted
assertEquals(r2, queue.offer(r5, 0));
assertEquals(r5, queue.take());
assertEquals(r3, queue.take());
assertEquals(r1, queue.take());
assertEquals(r4, queue.take());
r1 = createTaskWrapper(createRequest(1, 2, 5, 100), true, 100000);
r2 = createTaskWrapper(createRequest(2, 4, 4, 200), false, 100000);
r3 = createTaskWrapper(createRequest(3, 6, 3, 300), false, 1000000);
r4 = createTaskWrapper(createRequest(4, 8, 2, 400), false, 1000000);
r5 = createTaskWrapper(createRequest(5, 10, 1, 500), true, 1000000);
queue = new EvictingPriorityBlockingQueue<TaskWrapper>(new FirstInFirstOutComparator(), 4);
assertNull(queue.offer(r1, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r2, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r3, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r4, 0));
assertEquals(r1, queue.peek());
// offer accepted and r2 gets evicted
assertEquals(r2, queue.offer(r5, 0));
assertEquals(r5, queue.take());
assertEquals(r1, queue.take());
assertEquals(r4, queue.take());
assertEquals(r3, queue.take());
r1 = createTaskWrapper(createRequest(1, 2, 5, 100), false, 100000);
r2 = createTaskWrapper(createRequest(2, 4, 4, 200), true, 100000);
r3 = createTaskWrapper(createRequest(3, 6, 3, 300), true, 1000000);
r4 = createTaskWrapper(createRequest(4, 8, 2, 400), true, 1000000);
r5 = createTaskWrapper(createRequest(5, 10, 1, 500), true, 1000000);
queue = new EvictingPriorityBlockingQueue<TaskWrapper>(new FirstInFirstOutComparator(), 4);
assertNull(queue.offer(r1, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r2, 0));
assertEquals(r2, queue.peek());
assertNull(queue.offer(r3, 0));
assertEquals(r3, queue.peek());
assertNull(queue.offer(r4, 0));
assertEquals(r4, queue.peek());
// offer accepted, r1 evicted
assertEquals(r1, queue.offer(r5, 0));
assertEquals(r5, queue.take());
assertEquals(r4, queue.take());
assertEquals(r3, queue.take());
assertEquals(r2, queue.take());
r1 = createTaskWrapper(createRequest(1, 2, 5, 100), false, 100000);
r2 = createTaskWrapper(createRequest(2, 4, 4, 200), true, 100000);
r3 = createTaskWrapper(createRequest(3, 6, 3, 300), true, 1000000);
r4 = createTaskWrapper(createRequest(4, 8, 2, 400), true, 1000000);
r5 = createTaskWrapper(createRequest(5, 10, 2, 500), true, 1000000);
queue = new EvictingPriorityBlockingQueue<TaskWrapper>(new FirstInFirstOutComparator(), 4);
assertNull(queue.offer(r1, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r2, 0));
assertEquals(r2, queue.peek());
assertNull(queue.offer(r3, 0));
assertEquals(r3, queue.peek());
assertNull(queue.offer(r4, 0));
assertEquals(r4, queue.peek());
// offer accepted, r1 evicted
assertEquals(r1, queue.offer(r5, 0));
assertEquals(r5, queue.take());
assertEquals(r4, queue.take());
assertEquals(r3, queue.take());
assertEquals(r2, queue.take());
}
Also used :
TaskExecutorTestHelpers.createTaskWrapper(org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorTestHelpers.createTaskWrapper)
TaskWrapper(org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.TaskWrapper)
EvictingPriorityBlockingQueue(org.apache.hadoop.hive.llap.daemon.impl.EvictingPriorityBlockingQueue)
Test(org.junit.Test)
Example 14 with TaskWrapper
use of org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.TaskWrapper in project hive by apache.
the class TestShortestJobFirstComparator method testWaitQueueComparator.
@Test(timeout = 60000)
public void testWaitQueueComparator() throws InterruptedException {
TaskWrapper r1 = createTaskWrapper(createSubmitWorkRequestProto(1, 2, 100, 200, "q1"), false, 100000);
TaskWrapper r2 = createTaskWrapper(createSubmitWorkRequestProto(2, 4, 200, 300, "q2"), false, 100000);
TaskWrapper r3 = createTaskWrapper(createSubmitWorkRequestProto(3, 6, 300, 400, "q3"), false, 1000000);
TaskWrapper r4 = createTaskWrapper(createSubmitWorkRequestProto(4, 8, 400, 500, "q4"), false, 1000000);
TaskWrapper r5 = createTaskWrapper(createSubmitWorkRequestProto(5, 10, 500, 600, "q5"), false, 1000000);
EvictingPriorityBlockingQueue<TaskWrapper> queue = new EvictingPriorityBlockingQueue<>(new ShortestJobFirstComparator(), 4);
assertNull(queue.offer(r1, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r2, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r3, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r4, 0));
assertEquals(r1, queue.peek());
// this offer will be rejected
assertEquals(r5, queue.offer(r5, 0));
assertEquals(r1, queue.take());
assertEquals(r2, queue.take());
assertEquals(r3, queue.take());
assertEquals(r4, queue.take());
r1 = createTaskWrapper(createSubmitWorkRequestProto(1, 2, 100, 200, "q1"), true, 100000);
r2 = createTaskWrapper(createSubmitWorkRequestProto(2, 4, 200, 300, "q2"), true, 100000);
r3 = createTaskWrapper(createSubmitWorkRequestProto(3, 6, 300, 400, "q3"), true, 1000000);
r4 = createTaskWrapper(createSubmitWorkRequestProto(4, 8, 400, 500, "q4"), true, 1000000);
r5 = createTaskWrapper(createSubmitWorkRequestProto(5, 10, 500, 600, "q5"), true, 1000000);
queue = new EvictingPriorityBlockingQueue<TaskWrapper>(new ShortestJobFirstComparator(), 4);
assertNull(queue.offer(r1, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r2, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r3, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r4, 0));
assertEquals(r1, queue.peek());
// this offer will be rejected
assertEquals(r5, queue.offer(r5, 0));
assertEquals(r1, queue.take());
assertEquals(r2, queue.take());
assertEquals(r3, queue.take());
assertEquals(r4, queue.take());
r1 = createTaskWrapper(createSubmitWorkRequestProto(1, 1, 100, 1000, "q1"), true, 100000);
r2 = createTaskWrapper(createSubmitWorkRequestProto(2, 1, 200, 900, "q2"), false, 100000);
r3 = createTaskWrapper(createSubmitWorkRequestProto(3, 1, 300, 800, "q3"), true, 1000000);
r4 = createTaskWrapper(createSubmitWorkRequestProto(4, 1, 400, 700, "q4"), false, 1000000);
r5 = createTaskWrapper(createSubmitWorkRequestProto(5, 10, 500, 600, "q5"), true, 1000000);
queue = new EvictingPriorityBlockingQueue<TaskWrapper>(new ShortestJobFirstComparator(), 4);
assertNull(queue.offer(r1, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r2, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r3, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r4, 0));
assertEquals(r1, queue.peek());
// offer accepted and r4 gets evicted
assertEquals(r4, queue.offer(r5, 0));
assertEquals(r1, queue.take());
assertEquals(r3, queue.take());
assertEquals(r5, queue.take());
assertEquals(r2, queue.take());
r1 = createTaskWrapper(createSubmitWorkRequestProto(1, 2, 100, 200, "q1"), true, 100000);
r2 = createTaskWrapper(createSubmitWorkRequestProto(2, 4, 200, 300, "q2"), false, 100000);
r3 = createTaskWrapper(createSubmitWorkRequestProto(3, 6, 300, 400, "q3"), true, 1000000);
r4 = createTaskWrapper(createSubmitWorkRequestProto(4, 8, 400, 500, "q4"), false, 1000000);
r5 = createTaskWrapper(createSubmitWorkRequestProto(5, 10, 500, 600, "q5"), true, 1000000);
queue = new EvictingPriorityBlockingQueue<TaskWrapper>(new ShortestJobFirstComparator(), 4);
assertNull(queue.offer(r1, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r2, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r3, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r4, 0));
assertEquals(r1, queue.peek());
// offer accepted and r4 gets evicted
assertEquals(r4, queue.offer(r5, 0));
assertEquals(r1, queue.take());
assertEquals(r3, queue.take());
assertEquals(r5, queue.take());
assertEquals(r2, queue.take());
r1 = createTaskWrapper(createSubmitWorkRequestProto(1, 2, 100, 200, "q1"), true, 100000);
r2 = createTaskWrapper(createSubmitWorkRequestProto(2, 4, 200, 300, "q2"), false, 100000);
r3 = createTaskWrapper(createSubmitWorkRequestProto(3, 6, 300, 400, "q3"), false, 1000000);
r4 = createTaskWrapper(createSubmitWorkRequestProto(4, 8, 400, 500, "q4"), false, 1000000);
r5 = createTaskWrapper(createSubmitWorkRequestProto(5, 10, 500, 600, "q5"), true, 1000000);
queue = new EvictingPriorityBlockingQueue<TaskWrapper>(new ShortestJobFirstComparator(), 4);
assertNull(queue.offer(r1, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r2, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r3, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r4, 0));
assertEquals(r1, queue.peek());
// offer accepted and r4 gets evicted
assertEquals(r4, queue.offer(r5, 0));
assertEquals(r1, queue.take());
assertEquals(r5, queue.take());
assertEquals(r2, queue.take());
assertEquals(r3, queue.take());
r1 = createTaskWrapper(createSubmitWorkRequestProto(1, 2, 100, 200, "q1"), false, 100000);
r2 = createTaskWrapper(createSubmitWorkRequestProto(2, 4, 200, 300, "q2"), true, 100000);
r3 = createTaskWrapper(createSubmitWorkRequestProto(3, 6, 300, 400, "q3"), true, 1000000);
r4 = createTaskWrapper(createSubmitWorkRequestProto(4, 8, 400, 500, "q4"), true, 1000000);
r5 = createTaskWrapper(createSubmitWorkRequestProto(5, 10, 500, 600, "q5"), true, 1000000);
queue = new EvictingPriorityBlockingQueue<TaskWrapper>(new ShortestJobFirstComparator(), 4);
assertNull(queue.offer(r1, 0));
assertEquals(r1, queue.peek());
assertNull(queue.offer(r2, 0));
assertEquals(r2, queue.peek());
assertNull(queue.offer(r3, 0));
assertEquals(r2, queue.peek());
assertNull(queue.offer(r4, 0));
assertEquals(r2, queue.peek());
// offer accepted, r1 evicted
assertEquals(r1, queue.offer(r5, 0));
assertEquals(r2, queue.take());
assertEquals(r3, queue.take());
assertEquals(r4, queue.take());
assertEquals(r5, queue.take());
}
Also used :
TaskExecutorTestHelpers.createTaskWrapper(org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorTestHelpers.createTaskWrapper)
TaskWrapper(org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.TaskWrapper)
EvictingPriorityBlockingQueue(org.apache.hadoop.hive.llap.daemon.impl.EvictingPriorityBlockingQueue)
Test(org.junit.Test)
Example 15 with TaskWrapper
use of org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.TaskWrapper in project hive by apache.
the class TestShortestJobFirstComparator method testWaitQueueComparatorCanFinish.
@Test(timeout = 60000)
public void testWaitQueueComparatorCanFinish() throws InterruptedException {
// Test that only the fixed property (...ForQueue) is used in order determination, not the dynamic call.
TaskWrapper r1 = createTaskWrapper(createSubmitWorkRequestProto(1, 1, 0, 10, 100, 2), true, false, 100000);
TaskWrapper r2 = createTaskWrapper(createSubmitWorkRequestProto(2, 1, 0, 10, 100, 1), false, true, 100000);
TaskWrapper r3 = createTaskWrapper(createSubmitWorkRequestProto(3, 1, 0, 10, 100, 5), true, true, 100000);
EvictingPriorityBlockingQueue<TaskWrapper> queue = new EvictingPriorityBlockingQueue<>(new ShortestJobFirstComparator(), 4);
assertNull(queue.offer(r1, 0));
assertNull(queue.offer(r2, 0));
assertNull(queue.offer(r3, 0));
assertEquals(r2, queue.take());
assertEquals(r3, queue.take());
assertEquals(r1, queue.take());
}
Also used :
TaskExecutorTestHelpers.createTaskWrapper(org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorTestHelpers.createTaskWrapper)
TaskWrapper(org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.TaskWrapper)
EvictingPriorityBlockingQueue(org.apache.hadoop.hive.llap.daemon.impl.EvictingPriorityBlockingQueue)
Test(org.junit.Test)
Aggregations
TaskWrapper (org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.TaskWrapper)15
TaskExecutorTestHelpers.createTaskWrapper (org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorTestHelpers.createTaskWrapper)15
Test (org.junit.Test)15
EvictingPriorityBlockingQueue (org.apache.hadoop.hive.llap.daemon.impl.EvictingPriorityBlockingQueue)11
Configuration (org.apache.hadoop.conf.Configuration)3
MockRequest (org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorTestHelpers.MockRequest)3
TaskExecutorTestHelpers.createMockRequest (org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorTestHelpers.createMockRequest)3
ShortestJobFirstComparator (org.apache.hadoop.hive.llap.daemon.impl.comparator.ShortestJobFirstComparator)3
PriorityBlockingQueue (java.util.concurrent.PriorityBlockingQueue)1
