Search in sources :

Example 31 with FragmentContextImpl

use of org.apache.drill.exec.ops.FragmentContextImpl in project drill by apache.

the class TestHashJoin method testHJMockScanCommon.

private void testHJMockScanCommon(String physicalPlan, int expectedRows) throws Throwable {
    final DrillbitContext bitContext = mockDrillbitContext();
    final UserClientConnection connection = Mockito.mock(UserClientConnection.class);
    final PhysicalPlanReader reader = PhysicalPlanReaderTestFactory.defaultPhysicalPlanReader(c);
    final PhysicalPlan plan = reader.readPhysicalPlan(Files.asCharSource(DrillFileUtils.getResourceAsFile(physicalPlan), Charsets.UTF_8).read());
    final FunctionImplementationRegistry registry = new FunctionImplementationRegistry(c);
    final FragmentContextImpl context = new FragmentContextImpl(bitContext, PlanFragment.getDefaultInstance(), connection, registry);
    final SimpleRootExec exec = new SimpleRootExec(ImplCreator.getExec(context, (FragmentRoot) plan.getSortedOperators(false).iterator().next()));
    int totalRecordCount = 0;
    while (exec.next()) {
        totalRecordCount += exec.getRecordCount();
    }
    exec.close();
    assertEquals(expectedRows, totalRecordCount);
    if (context.getExecutorState().getFailureCause() != null) {
        throw context.getExecutorState().getFailureCause();
    }
    assertTrue(!context.getExecutorState().isFailed());
}
Also used : DrillbitContext(org.apache.drill.exec.server.DrillbitContext) SimpleRootExec(org.apache.drill.exec.physical.impl.SimpleRootExec) PhysicalPlan(org.apache.drill.exec.physical.PhysicalPlan) PhysicalPlanReader(org.apache.drill.exec.planner.PhysicalPlanReader) UserClientConnection(org.apache.drill.exec.rpc.UserClientConnection) FragmentContextImpl(org.apache.drill.exec.ops.FragmentContextImpl) FragmentRoot(org.apache.drill.exec.physical.base.FragmentRoot) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry)

Example 32 with FragmentContextImpl

use of org.apache.drill.exec.ops.FragmentContextImpl in project drill by apache.

the class TestMergeJoin method simpleEqualityJoin.

@Test
// this doesn't have a sort.  it also causes an infinite loop.  these may or may not be related.
@Ignore
public void simpleEqualityJoin() throws Throwable {
    final DrillbitContext bitContext = mockDrillbitContext();
    final UserClientConnection connection = Mockito.mock(UserClientConnection.class);
    final PhysicalPlanReader reader = PhysicalPlanReaderTestFactory.defaultPhysicalPlanReader(c);
    final PhysicalPlan plan = reader.readPhysicalPlan(Files.asCharSource(DrillFileUtils.getResourceAsFile("/join/merge_join.json"), Charsets.UTF_8).read());
    final FunctionImplementationRegistry registry = new FunctionImplementationRegistry(c);
    final FragmentContextImpl context = new FragmentContextImpl(bitContext, PlanFragment.getDefaultInstance(), connection, registry);
    final SimpleRootExec exec = new SimpleRootExec(ImplCreator.getExec(context, (FragmentRoot) plan.getSortedOperators(false).iterator().next()));
    int totalRecordCount = 0;
    final StringBuilder sb = new StringBuilder();
    while (exec.next()) {
        totalRecordCount += exec.getRecordCount();
        for (final ValueVector v : exec) {
            sb.append("[" + v.getField().getName() + "]        ");
        }
        sb.append("\n\n");
        for (int valueIdx = 0; valueIdx < exec.getRecordCount(); valueIdx++) {
            final List<Object> row = new ArrayList<>();
            for (final ValueVector v : exec) {
                row.add(v.getAccessor().getObject(valueIdx));
            }
            for (final Object cell : row) {
                if (cell == null) {
                    sb.append("<null>          ");
                    continue;
                }
                final int len = cell.toString().length();
                sb.append(cell);
                for (int i = 0; i < (14 - len); ++i) {
                    sb.append(" ");
                }
            }
            sb.append("\n");
        }
        sb.append("\n");
    }
    logger.info(sb.toString());
    assertEquals(100, totalRecordCount);
    if (context.getExecutorState().getFailureCause() != null) {
        throw context.getExecutorState().getFailureCause();
    }
    assertTrue(!context.getExecutorState().isFailed());
}
Also used : DrillbitContext(org.apache.drill.exec.server.DrillbitContext) PhysicalPlan(org.apache.drill.exec.physical.PhysicalPlan) PhysicalPlanReader(org.apache.drill.exec.planner.PhysicalPlanReader) ArrayList(java.util.ArrayList) FragmentContextImpl(org.apache.drill.exec.ops.FragmentContextImpl) FragmentRoot(org.apache.drill.exec.physical.base.FragmentRoot) ValueVector(org.apache.drill.exec.vector.ValueVector) SimpleRootExec(org.apache.drill.exec.physical.impl.SimpleRootExec) UserClientConnection(org.apache.drill.exec.rpc.UserClientConnection) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry) Ignore(org.junit.Ignore) SlowTest(org.apache.drill.categories.SlowTest) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Example 33 with FragmentContextImpl

use of org.apache.drill.exec.ops.FragmentContextImpl in project drill by apache.

the class TestMergeJoin method testJoinBatchSize.

@Test
public void testJoinBatchSize() throws Throwable {
    final DrillbitContext bitContext = mockDrillbitContext();
    final UserClientConnection connection = Mockito.mock(UserClientConnection.class);
    final PhysicalPlanReader reader = PhysicalPlanReaderTestFactory.defaultPhysicalPlanReader(c);
    final PhysicalPlan plan = reader.readPhysicalPlan(Files.asCharSource(DrillFileUtils.getResourceAsFile("/join/join_batchsize.json"), Charsets.UTF_8).read());
    final FunctionImplementationRegistry registry = new FunctionImplementationRegistry(c);
    final FragmentContextImpl context = new FragmentContextImpl(bitContext, PlanFragment.getDefaultInstance(), connection, registry);
    final SimpleRootExec exec = new SimpleRootExec(ImplCreator.getExec(context, (FragmentRoot) plan.getSortedOperators(false).iterator().next()));
    // skip schema batch
    exec.next();
    while (exec.next()) {
        assertEquals(100, exec.getRecordCount());
    }
    if (context.getExecutorState().getFailureCause() != null) {
        throw context.getExecutorState().getFailureCause();
    }
    assertTrue(!context.getExecutorState().isFailed());
}
Also used : DrillbitContext(org.apache.drill.exec.server.DrillbitContext) SimpleRootExec(org.apache.drill.exec.physical.impl.SimpleRootExec) PhysicalPlan(org.apache.drill.exec.physical.PhysicalPlan) PhysicalPlanReader(org.apache.drill.exec.planner.PhysicalPlanReader) UserClientConnection(org.apache.drill.exec.rpc.UserClientConnection) FragmentContextImpl(org.apache.drill.exec.ops.FragmentContextImpl) FragmentRoot(org.apache.drill.exec.physical.base.FragmentRoot) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry) SlowTest(org.apache.drill.categories.SlowTest) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Example 34 with FragmentContextImpl

use of org.apache.drill.exec.ops.FragmentContextImpl in project drill by apache.

the class TestMergeJoin method orderedEqualityInnerJoin.

@Test
@Ignore
public void orderedEqualityInnerJoin() throws Throwable {
    final DrillbitContext bitContext = mockDrillbitContext();
    final UserClientConnection connection = Mockito.mock(UserClientConnection.class);
    final PhysicalPlanReader reader = PhysicalPlanReaderTestFactory.defaultPhysicalPlanReader(c, new StoragePluginRegistryImpl(bitContext));
    final PhysicalPlan plan = reader.readPhysicalPlan(Files.asCharSource(DrillFileUtils.getResourceAsFile("/join/merge_inner_single_batch.json"), Charsets.UTF_8).read().replace("#{LEFT_FILE}", DrillFileUtils.getResourceAsFile("/join/merge_single_batch.left.json").toURI().toString()).replace("#{RIGHT_FILE}", DrillFileUtils.getResourceAsFile("/join/merge_single_batch.right.json").toURI().toString()));
    final FunctionImplementationRegistry registry = new FunctionImplementationRegistry(c);
    final FragmentContextImpl context = new FragmentContextImpl(bitContext, PlanFragment.getDefaultInstance(), connection, registry);
    final SimpleRootExec exec = new SimpleRootExec(ImplCreator.getExec(context, (FragmentRoot) plan.getSortedOperators(false).iterator().next()));
    int totalRecordCount = 0;
    final StringBuilder sb = new StringBuilder();
    while (exec.next()) {
        totalRecordCount += exec.getRecordCount();
        sb.append(String.format("got next with record count: %d (total: %d):\n", exec.getRecordCount(), totalRecordCount));
        sb.append("       t1                 t2\n");
        for (int valueIdx = 0; valueIdx < exec.getRecordCount(); valueIdx++) {
            final List<Object> row = Lists.newArrayList();
            for (final ValueVector v : exec) {
                row.add(v.getField().getName() + ":" + v.getAccessor().getObject(valueIdx));
            }
            for (final Object cell : row) {
                if (cell == null) {
                    sb.append("<null>    ");
                    continue;
                }
                final int len = cell.toString().length();
                sb.append(cell + " ");
                for (int i = 0; i < (10 - len); ++i) {
                    sb.append(" ");
                }
            }
            sb.append('\n');
        }
    }
    assertEquals(23, totalRecordCount);
    if (context.getExecutorState().getFailureCause() != null) {
        throw context.getExecutorState().getFailureCause();
    }
    assertTrue(!context.getExecutorState().isFailed());
}
Also used : DrillbitContext(org.apache.drill.exec.server.DrillbitContext) PhysicalPlan(org.apache.drill.exec.physical.PhysicalPlan) PhysicalPlanReader(org.apache.drill.exec.planner.PhysicalPlanReader) FragmentContextImpl(org.apache.drill.exec.ops.FragmentContextImpl) FragmentRoot(org.apache.drill.exec.physical.base.FragmentRoot) ValueVector(org.apache.drill.exec.vector.ValueVector) SimpleRootExec(org.apache.drill.exec.physical.impl.SimpleRootExec) StoragePluginRegistryImpl(org.apache.drill.exec.store.StoragePluginRegistryImpl) UserClientConnection(org.apache.drill.exec.rpc.UserClientConnection) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry) Ignore(org.junit.Ignore) SlowTest(org.apache.drill.categories.SlowTest) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Example 35 with FragmentContextImpl

use of org.apache.drill.exec.ops.FragmentContextImpl in project drill by apache.

the class TestMergeJoin method orderedEqualityMultiBatchJoin.

@Test
@Ignore
public void orderedEqualityMultiBatchJoin() throws Throwable {
    final DrillbitContext bitContext = mockDrillbitContext();
    final UserClientConnection connection = Mockito.mock(UserClientConnection.class);
    final PhysicalPlanReader reader = PhysicalPlanReaderTestFactory.defaultPhysicalPlanReader(c, new StoragePluginRegistryImpl(bitContext));
    final PhysicalPlan plan = reader.readPhysicalPlan(Files.asCharSource(DrillFileUtils.getResourceAsFile("/join/merge_multi_batch.json"), Charsets.UTF_8).read().replace("#{LEFT_FILE}", DrillFileUtils.getResourceAsFile("/join/merge_multi_batch.left.json").toURI().toString()).replace("#{RIGHT_FILE}", DrillFileUtils.getResourceAsFile("/join/merge_multi_batch.right.json").toURI().toString()));
    final FunctionImplementationRegistry registry = new FunctionImplementationRegistry(c);
    final FragmentContextImpl context = new FragmentContextImpl(bitContext, PlanFragment.getDefaultInstance(), connection, registry);
    final SimpleRootExec exec = new SimpleRootExec(ImplCreator.getExec(context, (FragmentRoot) plan.getSortedOperators(false).iterator().next()));
    int totalRecordCount = 0;
    final StringBuilder sb = new StringBuilder();
    while (exec.next()) {
        totalRecordCount += exec.getRecordCount();
        sb.append(String.format("got next with record count: %d (total: %d):\n", exec.getRecordCount(), totalRecordCount));
        for (int valueIdx = 0; valueIdx < exec.getRecordCount(); valueIdx++) {
            final List<Object> row = Lists.newArrayList();
            for (final ValueVector v : exec) {
                row.add(v.getField().getName() + ":" + v.getAccessor().getObject(valueIdx));
            }
            for (final Object cell : row) {
                if (cell == null) {
                    sb.append("<null>    ");
                    continue;
                }
                int len = cell.toString().length();
                sb.append(cell + " ");
                for (int i = 0; i < (10 - len); ++i) {
                    sb.append(" ");
                }
            }
            sb.append('\n');
        }
    }
    assertEquals(25, totalRecordCount);
    if (context.getExecutorState().getFailureCause() != null) {
        throw context.getExecutorState().getFailureCause();
    }
    assertTrue(!context.getExecutorState().isFailed());
}
Also used : DrillbitContext(org.apache.drill.exec.server.DrillbitContext) PhysicalPlan(org.apache.drill.exec.physical.PhysicalPlan) PhysicalPlanReader(org.apache.drill.exec.planner.PhysicalPlanReader) FragmentContextImpl(org.apache.drill.exec.ops.FragmentContextImpl) FragmentRoot(org.apache.drill.exec.physical.base.FragmentRoot) ValueVector(org.apache.drill.exec.vector.ValueVector) SimpleRootExec(org.apache.drill.exec.physical.impl.SimpleRootExec) StoragePluginRegistryImpl(org.apache.drill.exec.store.StoragePluginRegistryImpl) UserClientConnection(org.apache.drill.exec.rpc.UserClientConnection) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry) Ignore(org.junit.Ignore) SlowTest(org.apache.drill.categories.SlowTest) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Aggregations

FragmentContextImpl (org.apache.drill.exec.ops.FragmentContextImpl)98 FunctionImplementationRegistry (org.apache.drill.exec.expr.fn.FunctionImplementationRegistry)81 PhysicalPlan (org.apache.drill.exec.physical.PhysicalPlan)78 DrillbitContext (org.apache.drill.exec.server.DrillbitContext)77 FragmentRoot (org.apache.drill.exec.physical.base.FragmentRoot)74 UserClientConnection (org.apache.drill.exec.rpc.UserClientConnection)72 PhysicalPlanReader (org.apache.drill.exec.planner.PhysicalPlanReader)70 Test (org.junit.Test)66 SimpleRootExec (org.apache.drill.exec.physical.impl.SimpleRootExec)48 SchemaPath (org.apache.drill.common.expression.SchemaPath)32 OperatorTest (org.apache.drill.categories.OperatorTest)31 ExecTest (org.apache.drill.exec.ExecTest)31 ValueVector (org.apache.drill.exec.vector.ValueVector)16 BigIntVector (org.apache.drill.exec.vector.BigIntVector)12 IntVector (org.apache.drill.exec.vector.IntVector)12 Ignore (org.junit.Ignore)12 SlowTest (org.apache.drill.categories.SlowTest)10 DrillConfig (org.apache.drill.common.config.DrillConfig)8 StoragePluginRegistryImpl (org.apache.drill.exec.store.StoragePluginRegistryImpl)8 FragmentHandle (org.apache.drill.exec.proto.ExecProtos.FragmentHandle)7