Search in sources :

Example 1 with QueueQueryParallelizer

use of org.apache.drill.exec.planner.fragment.QueueQueryParallelizer in project drill by apache.

the class TestMemoryCalculator method TestSingleMajorFragmentWithProjectAndScan.

@Test
public void TestSingleMajorFragmentWithProjectAndScan() throws Exception {
    List<DrillbitEndpoint> activeEndpoints = getEndpoints(2, new HashSet<>());
    Map<DrillbitEndpoint, NodeResource> resources = activeEndpoints.stream().collect(Collectors.toMap(x -> x, x -> NodeResource.create()));
    String sql = "SELECT * from cp.`tpch/nation.parquet`";
    SimpleParallelizer parallelizer = new QueueQueryParallelizer(false, queryContext);
    PlanningSet planningSet = preparePlanningSet(activeEndpoints, DEFAULT_SLICE_TARGET, resources, sql, parallelizer);
    parallelizer.adjustMemory(planningSet, createSet(planningSet.getRootWrapper()), activeEndpoints);
    assertTrue("memory requirement is different", Iterables.all(resources.entrySet(), (e) -> e.getValue().getMemory() == 30));
}
Also used : NodeResource(org.apache.drill.exec.planner.cost.NodeResource) ArgumentMatchers.any(org.mockito.ArgumentMatchers.any) DrillbitEndpoint(org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint) QueueQueryParallelizer(org.apache.drill.exec.planner.fragment.QueueQueryParallelizer) HashMap(java.util.HashMap) ArrayList(java.util.ArrayList) PopUnitTestBase(org.apache.drill.exec.pop.PopUnitTestBase) HashSet(java.util.HashSet) DrillbitContext(org.apache.drill.exec.server.DrillbitContext) Iterables(org.apache.drill.shaded.guava.com.google.common.collect.Iterables) Map(java.util.Map) NodeResource(org.apache.drill.exec.planner.cost.NodeResource) SimpleParallelizer(org.apache.drill.exec.planner.fragment.SimpleParallelizer) ClientFixture(org.apache.drill.test.ClientFixture) QueryContext(org.apache.drill.exec.ops.QueryContext) AfterClass(org.junit.AfterClass) PlanTestBase(org.apache.drill.PlanTestBase) Iterator(java.util.Iterator) ClusterFixture(org.apache.drill.test.ClusterFixture) EmbeddedQueryQueue(org.apache.drill.exec.work.foreman.rm.EmbeddedQueryQueue) Set(java.util.Set) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) Mockito.when(org.mockito.Mockito.when) Collectors(java.util.stream.Collectors) PlanningSet(org.apache.drill.exec.planner.fragment.PlanningSet) List(java.util.List) Wrapper(org.apache.drill.exec.planner.fragment.Wrapper) PhysicalPlanReader(org.apache.drill.exec.planner.PhysicalPlanReader) Fragment(org.apache.drill.exec.planner.fragment.Fragment) UserBitShared(org.apache.drill.exec.proto.UserBitShared) UserProtos(org.apache.drill.exec.proto.UserProtos) ExecConstants(org.apache.drill.exec.ExecConstants) ClusterFixtureBuilder(org.apache.drill.test.ClusterFixtureBuilder) UserSession(org.apache.drill.exec.rpc.user.UserSession) Mockito.mock(org.mockito.Mockito.mock) DrillbitEndpoint(org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint) SimpleParallelizer(org.apache.drill.exec.planner.fragment.SimpleParallelizer) QueueQueryParallelizer(org.apache.drill.exec.planner.fragment.QueueQueryParallelizer) PlanningSet(org.apache.drill.exec.planner.fragment.PlanningSet) Test(org.junit.Test)

Example 2 with QueueQueryParallelizer

use of org.apache.drill.exec.planner.fragment.QueueQueryParallelizer in project drill by apache.

the class TestMemoryCalculator method TestTwoMajorFragmentWithSortyProjectAndScan.

@Test
public void TestTwoMajorFragmentWithSortyProjectAndScan() throws Exception {
    List<DrillbitEndpoint> activeEndpoints = getEndpoints(2, new HashSet<>());
    Map<DrillbitEndpoint, NodeResource> resources = activeEndpoints.stream().collect(Collectors.toMap(x -> x, x -> NodeResource.create()));
    String sql = "SELECT * from cp.`tpch/lineitem.parquet` order by dept_id";
    SimpleParallelizer parallelizer = new QueueQueryParallelizer(false, queryContext);
    PlanningSet planningSet = preparePlanningSet(activeEndpoints, 2, resources, sql, parallelizer);
    parallelizer.adjustMemory(planningSet, createSet(planningSet.getRootWrapper()), activeEndpoints);
    assertTrue("memory requirement is different", Iterables.all(resources.entrySet(), (e) -> e.getValue().getMemory() == 481490));
}
Also used : NodeResource(org.apache.drill.exec.planner.cost.NodeResource) ArgumentMatchers.any(org.mockito.ArgumentMatchers.any) DrillbitEndpoint(org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint) QueueQueryParallelizer(org.apache.drill.exec.planner.fragment.QueueQueryParallelizer) HashMap(java.util.HashMap) ArrayList(java.util.ArrayList) PopUnitTestBase(org.apache.drill.exec.pop.PopUnitTestBase) HashSet(java.util.HashSet) DrillbitContext(org.apache.drill.exec.server.DrillbitContext) Iterables(org.apache.drill.shaded.guava.com.google.common.collect.Iterables) Map(java.util.Map) NodeResource(org.apache.drill.exec.planner.cost.NodeResource) SimpleParallelizer(org.apache.drill.exec.planner.fragment.SimpleParallelizer) ClientFixture(org.apache.drill.test.ClientFixture) QueryContext(org.apache.drill.exec.ops.QueryContext) AfterClass(org.junit.AfterClass) PlanTestBase(org.apache.drill.PlanTestBase) Iterator(java.util.Iterator) ClusterFixture(org.apache.drill.test.ClusterFixture) EmbeddedQueryQueue(org.apache.drill.exec.work.foreman.rm.EmbeddedQueryQueue) Set(java.util.Set) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) Mockito.when(org.mockito.Mockito.when) Collectors(java.util.stream.Collectors) PlanningSet(org.apache.drill.exec.planner.fragment.PlanningSet) List(java.util.List) Wrapper(org.apache.drill.exec.planner.fragment.Wrapper) PhysicalPlanReader(org.apache.drill.exec.planner.PhysicalPlanReader) Fragment(org.apache.drill.exec.planner.fragment.Fragment) UserBitShared(org.apache.drill.exec.proto.UserBitShared) UserProtos(org.apache.drill.exec.proto.UserProtos) ExecConstants(org.apache.drill.exec.ExecConstants) ClusterFixtureBuilder(org.apache.drill.test.ClusterFixtureBuilder) UserSession(org.apache.drill.exec.rpc.user.UserSession) Mockito.mock(org.mockito.Mockito.mock) DrillbitEndpoint(org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint) SimpleParallelizer(org.apache.drill.exec.planner.fragment.SimpleParallelizer) QueueQueryParallelizer(org.apache.drill.exec.planner.fragment.QueueQueryParallelizer) PlanningSet(org.apache.drill.exec.planner.fragment.PlanningSet) Test(org.junit.Test)

Example 3 with QueueQueryParallelizer

use of org.apache.drill.exec.planner.fragment.QueueQueryParallelizer in project drill by apache.

the class TestMemoryCalculator method TestSingleMajorFragmentWithGroupByProjectAndScan.

@Test
public void TestSingleMajorFragmentWithGroupByProjectAndScan() throws Exception {
    List<DrillbitEndpoint> activeEndpoints = getEndpoints(2, new HashSet<>());
    Map<DrillbitEndpoint, NodeResource> resources = activeEndpoints.stream().collect(Collectors.toMap(x -> x, x -> NodeResource.create()));
    String sql = "SELECT dept_id, count(*) from cp.`tpch/lineitem.parquet` group by dept_id";
    SimpleParallelizer parallelizer = new QueueQueryParallelizer(false, queryContext);
    PlanningSet planningSet = preparePlanningSet(activeEndpoints, DEFAULT_SLICE_TARGET, resources, sql, parallelizer);
    parallelizer.adjustMemory(planningSet, createSet(planningSet.getRootWrapper()), activeEndpoints);
    assertTrue("memory requirement is different", Iterables.all(resources.entrySet(), (e) -> e.getValue().getMemory() == 529570));
}
Also used : NodeResource(org.apache.drill.exec.planner.cost.NodeResource) ArgumentMatchers.any(org.mockito.ArgumentMatchers.any) DrillbitEndpoint(org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint) QueueQueryParallelizer(org.apache.drill.exec.planner.fragment.QueueQueryParallelizer) HashMap(java.util.HashMap) ArrayList(java.util.ArrayList) PopUnitTestBase(org.apache.drill.exec.pop.PopUnitTestBase) HashSet(java.util.HashSet) DrillbitContext(org.apache.drill.exec.server.DrillbitContext) Iterables(org.apache.drill.shaded.guava.com.google.common.collect.Iterables) Map(java.util.Map) NodeResource(org.apache.drill.exec.planner.cost.NodeResource) SimpleParallelizer(org.apache.drill.exec.planner.fragment.SimpleParallelizer) ClientFixture(org.apache.drill.test.ClientFixture) QueryContext(org.apache.drill.exec.ops.QueryContext) AfterClass(org.junit.AfterClass) PlanTestBase(org.apache.drill.PlanTestBase) Iterator(java.util.Iterator) ClusterFixture(org.apache.drill.test.ClusterFixture) EmbeddedQueryQueue(org.apache.drill.exec.work.foreman.rm.EmbeddedQueryQueue) Set(java.util.Set) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) Mockito.when(org.mockito.Mockito.when) Collectors(java.util.stream.Collectors) PlanningSet(org.apache.drill.exec.planner.fragment.PlanningSet) List(java.util.List) Wrapper(org.apache.drill.exec.planner.fragment.Wrapper) PhysicalPlanReader(org.apache.drill.exec.planner.PhysicalPlanReader) Fragment(org.apache.drill.exec.planner.fragment.Fragment) UserBitShared(org.apache.drill.exec.proto.UserBitShared) UserProtos(org.apache.drill.exec.proto.UserProtos) ExecConstants(org.apache.drill.exec.ExecConstants) ClusterFixtureBuilder(org.apache.drill.test.ClusterFixtureBuilder) UserSession(org.apache.drill.exec.rpc.user.UserSession) Mockito.mock(org.mockito.Mockito.mock) DrillbitEndpoint(org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint) SimpleParallelizer(org.apache.drill.exec.planner.fragment.SimpleParallelizer) QueueQueryParallelizer(org.apache.drill.exec.planner.fragment.QueueQueryParallelizer) PlanningSet(org.apache.drill.exec.planner.fragment.PlanningSet) Test(org.junit.Test)

Aggregations

ArrayList (java.util.ArrayList)3 HashMap (java.util.HashMap)3 HashSet (java.util.HashSet)3 Iterator (java.util.Iterator)3 List (java.util.List)3 Map (java.util.Map)3 Set (java.util.Set)3 Collectors (java.util.stream.Collectors)3 PlanTestBase (org.apache.drill.PlanTestBase)3 ExecConstants (org.apache.drill.exec.ExecConstants)3 QueryContext (org.apache.drill.exec.ops.QueryContext)3 PhysicalPlanReader (org.apache.drill.exec.planner.PhysicalPlanReader)3 NodeResource (org.apache.drill.exec.planner.cost.NodeResource)3 Fragment (org.apache.drill.exec.planner.fragment.Fragment)3 PlanningSet (org.apache.drill.exec.planner.fragment.PlanningSet)3 QueueQueryParallelizer (org.apache.drill.exec.planner.fragment.QueueQueryParallelizer)3 SimpleParallelizer (org.apache.drill.exec.planner.fragment.SimpleParallelizer)3 Wrapper (org.apache.drill.exec.planner.fragment.Wrapper)3 PopUnitTestBase (org.apache.drill.exec.pop.PopUnitTestBase)3 DrillbitEndpoint (org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint)3