Examples with MockRecordBatch org.apache.drill.exec.physical.impl.MockRecordBatch used on opensource projects

Search in sources :

Example 46 with MockRecordBatch

use of org.apache.drill.exec.physical.impl.MockRecordBatch in project drill by apache.

the class TestTopNEmitOutcome method testTopNMultipleEmptyBatchFollowedByNonEmptyBatchEmitOutcome.

@Test
public void testTopNMultipleEmptyBatchFollowedByNonEmptyBatchEmitOutcome() {
    final RowSet.SingleRowSet nonEmptyInputRowSet2 = operatorFixture.rowSetBuilder(inputSchema).addRow(2, 20, "item2").addRow(13, 130, "item13").addRow(4, 40, "item4").build();
    final RowSet.SingleRowSet expectedRowSet = operatorFixture.rowSetBuilder(inputSchema).addRow(13, 130, "item13").addRow(4, 40, "item4").addRow(2, 20, "item2").build();
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(nonEmptyInputRowSet2.container());
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    final MockRecordBatch mockInputBatch = new MockRecordBatch(operatorFixture.getFragmentContext(), opContext, inputContainer, inputOutcomes, emptyInputRowSet.container().getSchema());
    final TopN topNConfig = new TopN(null, Lists.newArrayList(ordering("id_left", RelFieldCollation.Direction.DESCENDING, RelFieldCollation.NullDirection.FIRST)), false, 10);
    final TopNBatch topNBatch = new TopNBatch(topNConfig, operatorFixture.getFragmentContext(), mockInputBatch);
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    outputRecordCount += topNBatch.getRecordCount();
    assertEquals(0, outputRecordCount);
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.EMIT);
    outputRecordCount += topNBatch.getRecordCount();
    assertEquals(3, outputRecordCount);
    // verify results
    RowSet actualRowSet = HyperRowSetImpl.fromContainer(topNBatch.getContainer(), topNBatch.getSelectionVector4());
    new RowSetComparison(expectedRowSet).verify(actualRowSet);
    // Release memory for row sets
    nonEmptyInputRowSet2.clear();
    expectedRowSet.clear();
}
Also used : RowSetComparison(org.apache.drill.test.rowSet.RowSetComparison) RowSet(org.apache.drill.exec.physical.rowSet.RowSet) MockRecordBatch(org.apache.drill.exec.physical.impl.MockRecordBatch) TopN(org.apache.drill.exec.physical.config.TopN) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Example 47 with MockRecordBatch

use of org.apache.drill.exec.physical.impl.MockRecordBatch in project drill by apache.

the class TestTopNEmitOutcome method testTopNMultipleEMITOutcome.

@Test
public void testTopNMultipleEMITOutcome() {
    final RowSet.SingleRowSet nonEmptyInputRowSet2 = operatorFixture.rowSetBuilder(inputSchema).addRow(2, 20, "item2").addRow(3, 30, "item3").build();
    inputContainer.add(nonEmptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(nonEmptyInputRowSet2.container());
    inputContainer.add(emptyInputRowSet.container());
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    final MockRecordBatch mockInputBatch = new MockRecordBatch(operatorFixture.getFragmentContext(), opContext, inputContainer, inputOutcomes, emptyInputRowSet.container().getSchema());
    final TopN topNConfig = new TopN(null, Lists.newArrayList(ordering("id_left", RelFieldCollation.Direction.DESCENDING, RelFieldCollation.NullDirection.FIRST)), false, 10);
    final TopNBatch topNBatch = new TopNBatch(topNConfig, operatorFixture.getFragmentContext(), mockInputBatch);
    // first limit evaluation
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertEquals(1, topNBatch.getRecordCount());
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(0, topNBatch.getRecordCount());
    // After seeing EMIT limit will refresh it's state and again evaluate limit on next set of input batches
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(2, topNBatch.getRecordCount());
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(0, topNBatch.getRecordCount());
    nonEmptyInputRowSet2.clear();
}
Also used : RowSet(org.apache.drill.exec.physical.rowSet.RowSet) MockRecordBatch(org.apache.drill.exec.physical.impl.MockRecordBatch) TopN(org.apache.drill.exec.physical.config.TopN) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Example 48 with MockRecordBatch

use of org.apache.drill.exec.physical.impl.MockRecordBatch in project drill by apache.

the class TestTopNEmitOutcome method testTopNMultipleInputToMultipleOutputBatch_LowerLimits.

@Test
public void testTopNMultipleInputToMultipleOutputBatch_LowerLimits() {
    final RowSet.SingleRowSet nonEmptyInputRowSet2 = operatorFixture.rowSetBuilder(inputSchema).addRow(7, 70, "item7").addRow(3, 30, "item3").addRow(13, 130, "item13").build();
    final RowSet.SingleRowSet nonEmptyInputRowSet3 = operatorFixture.rowSetBuilder(inputSchema).addRow(17, 170, "item17").addRow(3, 30, "item3").addRow(13, 130, "item13").build();
    final RowSet.SingleRowSet expectedRowSet1 = operatorFixture.rowSetBuilder(inputSchema).addRow(13, 130, "item13").addRow(7, 70, "item7").build();
    final RowSet.SingleRowSet expectedRowSet2 = operatorFixture.rowSetBuilder(inputSchema).addRow(17, 170, "item17").addRow(13, 130, "item13").build();
    inputContainer.add(nonEmptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(nonEmptyInputRowSet2.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(nonEmptyInputRowSet3.container());
    inputContainer.add(emptyInputRowSet.container());
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.OK);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.OK);
    inputOutcomes.add(RecordBatch.IterOutcome.OK);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    final MockRecordBatch mockInputBatch = new MockRecordBatch(operatorFixture.getFragmentContext(), opContext, inputContainer, inputOutcomes, emptyInputRowSet.container().getSchema());
    final TopN topNConfig = new TopN(null, Lists.newArrayList(ordering("id_left", RelFieldCollation.Direction.DESCENDING, RelFieldCollation.NullDirection.FIRST)), false, 2);
    final TopNBatch topNBatch = new TopNBatch(topNConfig, operatorFixture.getFragmentContext(), mockInputBatch);
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertEquals(2, topNBatch.getRecordCount());
    RowSet actualRowSet1 = HyperRowSetImpl.fromContainer(topNBatch.getContainer(), topNBatch.getSelectionVector4());
    new RowSetComparison(expectedRowSet1).verify(actualRowSet1);
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(0, topNBatch.getRecordCount());
    assertTrue(topNBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(2, topNBatch.getRecordCount());
    RowSet actualRowSet2 = HyperRowSetImpl.fromContainer(topNBatch.getContainer(), topNBatch.getSelectionVector4());
    new RowSetComparison(expectedRowSet2).verify(actualRowSet2);
    nonEmptyInputRowSet2.clear();
    nonEmptyInputRowSet3.clear();
    expectedRowSet1.clear();
    expectedRowSet2.clear();
}
Also used : RowSetComparison(org.apache.drill.test.rowSet.RowSetComparison) RowSet(org.apache.drill.exec.physical.rowSet.RowSet) MockRecordBatch(org.apache.drill.exec.physical.impl.MockRecordBatch) TopN(org.apache.drill.exec.physical.config.TopN) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Example 49 with MockRecordBatch

use of org.apache.drill.exec.physical.impl.MockRecordBatch in project drill by apache.

the class TestStreamingAggEmitOutcome method t17_testStreamingAggrMultipleEMITOutcome.

/**
 *   Repeats t7_testStreamingAggrMultipleEMITOutcome with no group by
 */
@Test
public void t17_testStreamingAggrMultipleEMITOutcome() {
    final RowSet.SingleRowSet nonEmptyInputRowSet2 = operatorFixture.rowSetBuilder(inputSchema).addRow(2, 20, "item2").addRow(3, 30, "item3").build();
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(nonEmptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(nonEmptyInputRowSet2.container());
    inputContainer.add(emptyInputRowSet.container());
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    final MockRecordBatch mockInputBatch = new MockRecordBatch(operatorFixture.getFragmentContext(), opContext, inputContainer, inputOutcomes, emptyInputRowSet.container().getSchema());
    final StreamingAggregate streamAggrConfig = new StreamingAggregate(null, new ArrayList<NamedExpression>(), parseExprs("sum(id_left+cost_left)", "total_sum"));
    final StreamingAggBatch strAggBatch = new StreamingAggBatch(streamAggrConfig, mockInputBatch, operatorFixture.getFragmentContext());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertEquals(1, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(0, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(1, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(1, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.NONE);
    nonEmptyInputRowSet2.clear();
}
Also used : StreamingAggregate(org.apache.drill.exec.physical.config.StreamingAggregate) NamedExpression(org.apache.drill.common.logical.data.NamedExpression) DirectRowSet(org.apache.drill.exec.physical.rowSet.DirectRowSet) RowSet(org.apache.drill.exec.physical.rowSet.RowSet) MockRecordBatch(org.apache.drill.exec.physical.impl.MockRecordBatch) StreamingAggBatch(org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Example 50 with MockRecordBatch

use of org.apache.drill.exec.physical.impl.MockRecordBatch in project drill by apache.

the class TestStreamingAggEmitOutcome method t10_1_testStreamingAggr_InputSplitToMultipleOutputBatch.

@Test
public void t10_1_testStreamingAggr_InputSplitToMultipleOutputBatch() {
    final RowSet.SingleRowSet nonEmptyInputRowSet2 = operatorFixture.rowSetBuilder(inputSchema).addRow(1, 20, "item1").build();
    final RowSet.SingleRowSet nonEmptyInputRowSet3 = operatorFixture.rowSetBuilder(inputSchema).addRow(2, 30, "item2").build();
    final RowSet.SingleRowSet nonEmptyInputRowSet4 = operatorFixture.rowSetBuilder(inputSchema).addRow(2, 40, "item2").addRow(2, 50, "item2").addRow(2, 60, "item2").addRow(2, 70, "item2").addRow(3, 100, "item3").addRow(3, 200, "item3").addRow(3, 300, "item3").addRow(3, 400, "item3").build();
    TupleMetadata resultSchema2 = new SchemaBuilder().add("name", TypeProtos.MinorType.VARCHAR).add("id", TypeProtos.MinorType.INT).add("total_count", TypeProtos.MinorType.BIGINT).buildSchema();
    final RowSet.SingleRowSet expectedRowSet1 = operatorFixture.rowSetBuilder(resultSchema2).addRow("item1", 1, (long) 2).addRow("item2", 2, (long) 5).build();
    final RowSet.SingleRowSet expectedRowSet2 = operatorFixture.rowSetBuilder(resultSchema2).addRow("item3", 3, (long) 4).build();
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(nonEmptyInputRowSet.container());
    inputContainer.add(nonEmptyInputRowSet2.container());
    inputContainer.add(nonEmptyInputRowSet3.container());
    inputContainer.add(nonEmptyInputRowSet4.container());
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.OK);
    inputOutcomes.add(RecordBatch.IterOutcome.OK);
    inputOutcomes.add(RecordBatch.IterOutcome.OK);
    final MockRecordBatch mockInputBatch = new MockRecordBatch(operatorFixture.getFragmentContext(), opContext, inputContainer, inputOutcomes, emptyInputRowSet.container().getSchema());
    final StreamingAggregate streamAggrConfig = new StreamingAggregate(null, parseExprs("name_left", "name", "id_left", "id"), parseExprs("count(cost_left)", "total_count"));
    final StreamingAggBatch strAggBatch = new StreamingAggBatch(streamAggrConfig, mockInputBatch, operatorFixture.getFragmentContext());
    strAggBatch.setMaxOutputRowCount(2);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertEquals(2, strAggBatch.getRecordCount());
    RowSet actualRowSet = DirectRowSet.fromContainer(strAggBatch.getContainer());
    new RowSetComparison(expectedRowSet1).verify(actualRowSet);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK);
    assertEquals(1, strAggBatch.getRecordCount());
    actualRowSet = DirectRowSet.fromContainer(strAggBatch.getContainer());
    new RowSetComparison(expectedRowSet2).verify(actualRowSet);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.NONE);
    nonEmptyInputRowSet2.clear();
    nonEmptyInputRowSet3.clear();
    nonEmptyInputRowSet4.clear();
    expectedRowSet1.clear();
    expectedRowSet2.clear();
}
Also used : StreamingAggregate(org.apache.drill.exec.physical.config.StreamingAggregate) RowSetComparison(org.apache.drill.test.rowSet.RowSetComparison) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) DirectRowSet(org.apache.drill.exec.physical.rowSet.DirectRowSet) RowSet(org.apache.drill.exec.physical.rowSet.RowSet) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) MockRecordBatch(org.apache.drill.exec.physical.impl.MockRecordBatch) StreamingAggBatch(org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Aggregations

MockRecordBatch (org.apache.drill.exec.physical.impl.MockRecordBatch)130 Test (org.junit.Test)121 OperatorTest (org.apache.drill.categories.OperatorTest)106 RowSet (org.apache.drill.exec.physical.rowSet.RowSet)97 DirectRowSet (org.apache.drill.exec.physical.rowSet.DirectRowSet)64 RowSetComparison (org.apache.drill.test.rowSet.RowSetComparison)54 CloseableRecordBatch (org.apache.drill.exec.record.CloseableRecordBatch)49 SubOperatorTest (org.apache.drill.test.SubOperatorTest)49 UserException (org.apache.drill.common.exceptions.UserException)39 StreamingAggregate (org.apache.drill.exec.physical.config.StreamingAggregate)25 StreamingAggBatch (org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch)25 SchemaBuilder (org.apache.drill.exec.record.metadata.SchemaBuilder)22 TupleMetadata (org.apache.drill.exec.record.metadata.TupleMetadata)20 LateralJoinPOP (org.apache.drill.exec.physical.config.LateralJoinPOP)17 ArrayList (java.util.ArrayList)13 TopN (org.apache.drill.exec.physical.config.TopN)13 VectorContainer (org.apache.drill.exec.record.VectorContainer)13 Project (org.apache.drill.exec.physical.config.Project)8 NamedExpression (org.apache.drill.common.logical.data.NamedExpression)7 Filter (org.apache.drill.exec.physical.config.Filter)6
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial