Search in sources :

Example 1 with AllocationHint

use of org.apache.drill.exec.record.VectorInitializer.AllocationHint in project drill by apache.

the class TestShortArrays method testSizer.

@Test
public void testSizer() {
    // Create a row set with less than one item, on
    // average, per array.
    TupleMetadata schema = new SchemaBuilder().add("a", MinorType.INT).addArray("b", MinorType.INT).buildSchema();
    RowSetBuilder builder = fixture.rowSetBuilder(schema).addRow(1, intArray(10));
    for (int i = 2; i <= 10; i++) {
        builder.addRow(i, intArray());
    }
    RowSet rows = builder.build();
    // Run the record batch sizer on the resulting batch.
    RecordBatchSizer sizer = new RecordBatchSizer(rows.container());
    assertEquals(2, sizer.columns().size());
    ColumnSize bCol = sizer.columns().get("b");
    assertEquals(0.1, bCol.getCardinality(), 0.01);
    assertEquals(1, bCol.getElementCount());
    // Create a vector initializer using the sizer info.
    VectorInitializer vi = sizer.buildVectorInitializer();
    AllocationHint bHint = vi.hint("b");
    assertNotNull(bHint);
    assertEquals(bHint.elementCount, bCol.getCardinality(), 0.001);
    // Create a new batch, and new vector, using the sizer and
    // initializer inferred from the previous batch.
    SingleRowSet empty = fixture.rowSet(schema);
    vi.allocateBatch(empty.container(), 100);
    assertEquals(2, empty.container().getNumberOfColumns());
    ValueVector bVector = empty.container().getValueVector(1).getValueVector();
    assertTrue(bVector instanceof RepeatedIntVector);
    assertEquals(16, ((RepeatedIntVector) bVector).getDataVector().getValueCapacity());
    rows.clear();
    empty.clear();
}
Also used : AllocationHint(org.apache.drill.exec.record.VectorInitializer.AllocationHint) ValueVector(org.apache.drill.exec.vector.ValueVector) RowSetBuilder(org.apache.drill.exec.physical.rowSet.RowSetBuilder) RecordBatchSizer(org.apache.drill.exec.record.RecordBatchSizer) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) RepeatedIntVector(org.apache.drill.exec.vector.RepeatedIntVector) ColumnSize(org.apache.drill.exec.record.RecordBatchSizer.ColumnSize) VectorInitializer(org.apache.drill.exec.record.VectorInitializer) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) RowSet(org.apache.drill.exec.physical.rowSet.RowSet) AllocationHint(org.apache.drill.exec.record.VectorInitializer.AllocationHint) SubOperatorTest(org.apache.drill.test.SubOperatorTest) Test(org.junit.Test)

Example 2 with AllocationHint

use of org.apache.drill.exec.record.VectorInitializer.AllocationHint in project drill by axbaretto.

the class TestShortArrays method testSizer.

@Test
public void testSizer() {
    // Create a row set with less than one item, on
    // average, per array.
    BatchSchema schema = new SchemaBuilder().add("a", MinorType.INT).addArray("b", MinorType.INT).build();
    RowSetBuilder builder = fixture.rowSetBuilder(schema).addRow(1, intArray(10));
    for (int i = 2; i <= 10; i++) {
        builder.addRow(i, intArray());
    }
    RowSet rows = builder.build();
    // Run the record batch sizer on the resulting batch.
    RecordBatchSizer sizer = new RecordBatchSizer(rows.container());
    assertEquals(2, sizer.columns().size());
    ColumnSize bCol = sizer.columns().get("b");
    assertEquals(0.1, bCol.getCardinality(), 0.01);
    assertEquals(1, bCol.getElementCount());
    // Create a vector initializer using the sizer info.
    VectorInitializer vi = sizer.buildVectorInitializer();
    AllocationHint bHint = vi.hint("b");
    assertNotNull(bHint);
    assertEquals(bHint.elementCount, bCol.getCardinality(), 0.001);
    // Create a new batch, and new vector, using the sizer and
    // initializer inferred from the previous batch.
    SingleRowSet empty = fixture.rowSet(schema);
    vi.allocateBatch(empty.container(), 100);
    assertEquals(2, empty.container().getNumberOfColumns());
    @SuppressWarnings("resource") ValueVector bVector = empty.container().getValueVector(1).getValueVector();
    assertTrue(bVector instanceof RepeatedIntVector);
    assertEquals(16, ((RepeatedIntVector) bVector).getDataVector().getValueCapacity());
    rows.clear();
    empty.clear();
}
Also used : SingleRowSet(org.apache.drill.test.rowSet.RowSet.SingleRowSet) RepeatedIntVector(org.apache.drill.exec.vector.RepeatedIntVector) ColumnSize(org.apache.drill.exec.record.RecordBatchSizer.ColumnSize) VectorInitializer(org.apache.drill.exec.record.VectorInitializer) SingleRowSet(org.apache.drill.test.rowSet.RowSet.SingleRowSet) RowSet(org.apache.drill.test.rowSet.RowSet) AllocationHint(org.apache.drill.exec.record.VectorInitializer.AllocationHint) AllocationHint(org.apache.drill.exec.record.VectorInitializer.AllocationHint) ValueVector(org.apache.drill.exec.vector.ValueVector) RowSetBuilder(org.apache.drill.test.rowSet.RowSetBuilder) RecordBatchSizer(org.apache.drill.exec.record.RecordBatchSizer) BatchSchema(org.apache.drill.exec.record.BatchSchema) SchemaBuilder(org.apache.drill.test.rowSet.schema.SchemaBuilder) SubOperatorTest(org.apache.drill.test.SubOperatorTest) Test(org.junit.Test)

Aggregations

RecordBatchSizer (org.apache.drill.exec.record.RecordBatchSizer)2 ColumnSize (org.apache.drill.exec.record.RecordBatchSizer.ColumnSize)2 VectorInitializer (org.apache.drill.exec.record.VectorInitializer)2 AllocationHint (org.apache.drill.exec.record.VectorInitializer.AllocationHint)2 RepeatedIntVector (org.apache.drill.exec.vector.RepeatedIntVector)2 ValueVector (org.apache.drill.exec.vector.ValueVector)2 SubOperatorTest (org.apache.drill.test.SubOperatorTest)2 Test (org.junit.Test)2 RowSet (org.apache.drill.exec.physical.rowSet.RowSet)1 SingleRowSet (org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet)1 RowSetBuilder (org.apache.drill.exec.physical.rowSet.RowSetBuilder)1 BatchSchema (org.apache.drill.exec.record.BatchSchema)1 SchemaBuilder (org.apache.drill.exec.record.metadata.SchemaBuilder)1 TupleMetadata (org.apache.drill.exec.record.metadata.TupleMetadata)1 RowSet (org.apache.drill.test.rowSet.RowSet)1 SingleRowSet (org.apache.drill.test.rowSet.RowSet.SingleRowSet)1 RowSetBuilder (org.apache.drill.test.rowSet.RowSetBuilder)1 SchemaBuilder (org.apache.drill.test.rowSet.schema.SchemaBuilder)1