Examples with StreamingAggregate org.apache.drill.exec.physical.config.StreamingAggregate used on opensource projects

Search in sources :

Example 6 with StreamingAggregate

use of org.apache.drill.exec.physical.config.StreamingAggregate in project drill by apache.

the class TestStreamingAggEmitOutcome method t13_testStreamingAggrEmptyBatchFollowedByNonEmptyBatchEmitOutcome.

/**
 *   Repeats t3_testStreamingAggrEmptyBatchFollowedByNonEmptyBatchEmitOutcome with no group by
 */
@Test
public void t13_testStreamingAggrEmptyBatchFollowedByNonEmptyBatchEmitOutcome() {
    final RowSet.SingleRowSet nonEmptyInputRowSet2 = operatorFixture.rowSetBuilder(inputSchema).addRow(13, 130, "item13").addRow(0, 1300, "item13").addRow(2, 20, "item2").addRow(0, 2000, "item2").addRow(4, 40, "item4").addRow(0, 4000, "item4").build();
    final RowSet.SingleRowSet expectedRowSet = operatorFixture.rowSetBuilder(resultSchemaNoGroupBy).addRow((long) 7509).build();
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(nonEmptyInputRowSet2.container());
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    final MockRecordBatch mockInputBatch = new MockRecordBatch(operatorFixture.getFragmentContext(), opContext, inputContainer, inputOutcomes, emptyInputRowSet.container().getSchema());
    final StreamingAggregate streamAggrConfig = new StreamingAggregate(null, new ArrayList<NamedExpression>(), parseExprs("sum(id_left+cost_left)", "total_sum"));
    final StreamingAggBatch strAggBatch = new StreamingAggBatch(streamAggrConfig, mockInputBatch, operatorFixture.getFragmentContext());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(0, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(1, strAggBatch.getRecordCount());
    RowSet actualRowSet = DirectRowSet.fromContainer(strAggBatch.getContainer());
    new RowSetComparison(expectedRowSet).verify(actualRowSet);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.NONE);
    // Release memory for row sets
    nonEmptyInputRowSet2.clear();
    expectedRowSet.clear();
}
Also used : StreamingAggregate(org.apache.drill.exec.physical.config.StreamingAggregate) RowSetComparison(org.apache.drill.test.rowSet.RowSetComparison) NamedExpression(org.apache.drill.common.logical.data.NamedExpression) DirectRowSet(org.apache.drill.exec.physical.rowSet.DirectRowSet) RowSet(org.apache.drill.exec.physical.rowSet.RowSet) MockRecordBatch(org.apache.drill.exec.physical.impl.MockRecordBatch) StreamingAggBatch(org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Example 7 with StreamingAggregate

use of org.apache.drill.exec.physical.config.StreamingAggregate in project drill by apache.

the class TestStreamingAggEmitOutcome method t21_testStreamingAggrRunsofEmpty_NonEmpty_Sv2.

/**
 *   Repeats t21_testStreamingAggrRunsofEmpty_NonEmpty_Sv2 with no group by
 */
@Test
public void t21_testStreamingAggrRunsofEmpty_NonEmpty_Sv2() {
    TupleMetadata inputSchema_sv2 = new SchemaBuilder().add("id_left", TypeProtos.MinorType.INT).add("cost_left", TypeProtos.MinorType.INT).add("name_left", TypeProtos.MinorType.VARCHAR).buildSchema();
    final RowSet.SingleRowSet emptyRowSet_Sv2 = operatorFixture.rowSetBuilder(inputSchema_sv2).withSv2().build();
    final RowSet.SingleRowSet nonEmptyInputRowSet2 = operatorFixture.rowSetBuilder(inputSchema_sv2).addSelection(false, 2, 20, "item2").addSelection(true, 3, 30, "item3").withSv2().build();
    inputContainer.add(emptyRowSet_Sv2.container());
    inputContainer.add(emptyRowSet_Sv2.container());
    inputContainer.add(emptyRowSet_Sv2.container());
    inputContainer.add(emptyRowSet_Sv2.container());
    inputContainer.add(emptyRowSet_Sv2.container());
    inputContainer.add(nonEmptyInputRowSet2.container());
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.OK);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputContainerSv2.add(emptyRowSet_Sv2.getSv2());
    inputContainerSv2.add(emptyRowSet_Sv2.getSv2());
    inputContainerSv2.add(emptyRowSet_Sv2.getSv2());
    inputContainerSv2.add(emptyRowSet_Sv2.getSv2());
    inputContainerSv2.add(emptyRowSet_Sv2.getSv2());
    inputContainerSv2.add(nonEmptyInputRowSet2.getSv2());
    final MockRecordBatch mockInputBatch = new MockRecordBatch(operatorFixture.getFragmentContext(), opContext, inputContainer, inputOutcomes, inputContainerSv2, inputContainer.get(0).getSchema());
    final StreamingAggregate streamAggrConfig = new StreamingAggregate(null, new ArrayList<>(), parseExprs("sum(id_left+cost_left)", "total_sum"));
    final StreamingAggBatch strAggBatch = new StreamingAggBatch(streamAggrConfig, mockInputBatch, operatorFixture.getFragmentContext());
    final RowSet.SingleRowSet expectedRowSet = operatorFixture.rowSetBuilder(resultSchemaNoGroupBy).addRow((long) 33).build();
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertEquals(0, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    // For special batch.
    assertEquals(1, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(0, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(1, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(1, strAggBatch.getRecordCount());
    RowSet actualRowSet = DirectRowSet.fromContainer(strAggBatch.getContainer());
    new RowSetComparison(expectedRowSet).verify(actualRowSet);
    nonEmptyInputRowSet2.clear();
    emptyRowSet_Sv2.clear();
    expectedRowSet.clear();
}
Also used : StreamingAggregate(org.apache.drill.exec.physical.config.StreamingAggregate) RowSetComparison(org.apache.drill.test.rowSet.RowSetComparison) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) DirectRowSet(org.apache.drill.exec.physical.rowSet.DirectRowSet) RowSet(org.apache.drill.exec.physical.rowSet.RowSet) MockRecordBatch(org.apache.drill.exec.physical.impl.MockRecordBatch) StreamingAggBatch(org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Example 8 with StreamingAggregate

use of org.apache.drill.exec.physical.config.StreamingAggregate in project drill by apache.

the class TestStreamingAggEmitOutcome method t20_testStreamingAggrWithEmptyDataSet.

/**
 *   Repeats t10_testStreamingAggrWithEmptyDataSet with no group by
 */
@Test
public void t20_testStreamingAggrWithEmptyDataSet() {
    inputContainer.add(emptyInputRowSet.container());
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    final MockRecordBatch mockInputBatch = new MockRecordBatch(operatorFixture.getFragmentContext(), opContext, inputContainer, inputOutcomes, emptyInputRowSet.container().getSchema());
    final StreamingAggregate streamAggrConfig = new StreamingAggregate(null, new ArrayList<>(), parseExprs("sum(id_left+cost_left)", "total_sum"));
    final StreamingAggBatch strAggBatch = new StreamingAggBatch(streamAggrConfig, mockInputBatch, operatorFixture.getFragmentContext());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.NONE);
}
Also used : StreamingAggregate(org.apache.drill.exec.physical.config.StreamingAggregate) MockRecordBatch(org.apache.drill.exec.physical.impl.MockRecordBatch) StreamingAggBatch(org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Example 9 with StreamingAggregate

use of org.apache.drill.exec.physical.config.StreamingAggregate in project drill by apache.

the class TestStreamingAggEmitOutcome method t4_testStreamingAggrMultipleEmptyBatchFollowedByNonEmptyBatchEmitOutcome.

@Test
public void t4_testStreamingAggrMultipleEmptyBatchFollowedByNonEmptyBatchEmitOutcome() {
    final RowSet.SingleRowSet nonEmptyInputRowSet2 = operatorFixture.rowSetBuilder(inputSchema).addRow(13, 130, "item13").addRow(0, 0, "item13").addRow(1, 33000, "item13").addRow(2, 20, "item2").addRow(0, 0, "item2").addRow(1, 11000, "item2").addRow(4, 40, "item4").build();
    final RowSet.SingleRowSet expectedRowSet = operatorFixture.rowSetBuilder(resultSchema).addRow("item13", (long) 33144).addRow("item2", (long) 11023).addRow("item4", (long) 44).build();
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(nonEmptyInputRowSet2.container());
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    final MockRecordBatch mockInputBatch = new MockRecordBatch(operatorFixture.getFragmentContext(), opContext, inputContainer, inputOutcomes, emptyInputRowSet.container().getSchema());
    final StreamingAggregate streamAggrConfig = new StreamingAggregate(null, parseExprs("name_left", "name"), parseExprs("sum(id_left+cost_left)", "total_sum"));
    final StreamingAggBatch strAggBatch = new StreamingAggBatch(streamAggrConfig, mockInputBatch, operatorFixture.getFragmentContext());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(0, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(0, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(0, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(3, strAggBatch.getRecordCount());
    RowSet actualRowSet = DirectRowSet.fromContainer(strAggBatch.getContainer());
    new RowSetComparison(expectedRowSet).verify(actualRowSet);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.NONE);
    // Release memory for row sets
    nonEmptyInputRowSet2.clear();
    expectedRowSet.clear();
}
Also used : StreamingAggregate(org.apache.drill.exec.physical.config.StreamingAggregate) RowSetComparison(org.apache.drill.test.rowSet.RowSetComparison) DirectRowSet(org.apache.drill.exec.physical.rowSet.DirectRowSet) RowSet(org.apache.drill.exec.physical.rowSet.RowSet) MockRecordBatch(org.apache.drill.exec.physical.impl.MockRecordBatch) StreamingAggBatch(org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Example 10 with StreamingAggregate

use of org.apache.drill.exec.physical.config.StreamingAggregate in project drill by apache.

the class TestStreamingAggEmitOutcome method t14_testStreamingAggrMultipleEmptyBatchFollowedByNonEmptyBatchEmitOutcome.

/**
 *   Repeats t4_testStreamingAggrMultipleEmptyBatchFollowedByNonEmptyBatchEmitOutcome with no group by
 */
@Test
public void t14_testStreamingAggrMultipleEmptyBatchFollowedByNonEmptyBatchEmitOutcome() {
    final RowSet.SingleRowSet nonEmptyInputRowSet2 = operatorFixture.rowSetBuilder(inputSchema).addRow(13, 130, "item13").addRow(0, 0, "item13").addRow(1, 33000, "item13").addRow(2, 20, "item2").addRow(0, 0, "item2").addRow(1, 11000, "item2").addRow(4, 40, "item4").build();
    final RowSet.SingleRowSet expectedRowSet = operatorFixture.rowSetBuilder(resultSchemaNoGroupBy).addRow((long) 44211).build();
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(emptyInputRowSet.container());
    inputContainer.add(nonEmptyInputRowSet2.container());
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    inputOutcomes.add(RecordBatch.IterOutcome.EMIT);
    final MockRecordBatch mockInputBatch = new MockRecordBatch(operatorFixture.getFragmentContext(), opContext, inputContainer, inputOutcomes, emptyInputRowSet.container().getSchema());
    final StreamingAggregate streamAggrConfig = new StreamingAggregate(null, new ArrayList<NamedExpression>(), parseExprs("sum(id_left+cost_left)", "total_sum"));
    final StreamingAggBatch strAggBatch = new StreamingAggBatch(streamAggrConfig, mockInputBatch, operatorFixture.getFragmentContext());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.OK_NEW_SCHEMA);
    // special batch
    assertEquals(1, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    assertEquals(0, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    // special batch
    assertEquals(1, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    // special batch
    assertEquals(1, strAggBatch.getRecordCount());
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.EMIT);
    // data batch
    assertEquals(1, strAggBatch.getRecordCount());
    RowSet actualRowSet = DirectRowSet.fromContainer(strAggBatch.getContainer());
    new RowSetComparison(expectedRowSet).verify(actualRowSet);
    assertTrue(strAggBatch.next() == RecordBatch.IterOutcome.NONE);
    // Release memory for row sets
    nonEmptyInputRowSet2.clear();
    expectedRowSet.clear();
}
Also used : StreamingAggregate(org.apache.drill.exec.physical.config.StreamingAggregate) RowSetComparison(org.apache.drill.test.rowSet.RowSetComparison) NamedExpression(org.apache.drill.common.logical.data.NamedExpression) DirectRowSet(org.apache.drill.exec.physical.rowSet.DirectRowSet) RowSet(org.apache.drill.exec.physical.rowSet.RowSet) MockRecordBatch(org.apache.drill.exec.physical.impl.MockRecordBatch) StreamingAggBatch(org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch) OperatorTest(org.apache.drill.categories.OperatorTest) Test(org.junit.Test)

Aggregations

StreamingAggregate (org.apache.drill.exec.physical.config.StreamingAggregate)31 Test (org.junit.Test)29 OperatorTest (org.apache.drill.categories.OperatorTest)25 MockRecordBatch (org.apache.drill.exec.physical.impl.MockRecordBatch)25 StreamingAggBatch (org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch)25 DirectRowSet (org.apache.drill.exec.physical.rowSet.DirectRowSet)21 RowSet (org.apache.drill.exec.physical.rowSet.RowSet)21 RowSetComparison (org.apache.drill.test.rowSet.RowSetComparison)19 NamedExpression (org.apache.drill.common.logical.data.NamedExpression)7 SchemaBuilder (org.apache.drill.exec.record.metadata.SchemaBuilder)4 TupleMetadata (org.apache.drill.exec.record.metadata.TupleMetadata)4 PhysicalOperator (org.apache.drill.exec.physical.base.PhysicalOperator)2
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial