Search in sources :

Example 1 with NullResultVectorCacheImpl

use of org.apache.drill.exec.physical.resultSet.impl.NullResultVectorCacheImpl in project drill by apache.

the class TestNullColumnLoader method testCustomNullType.

/**
 * Test the ability to use a type other than nullable INT for null
 * columns. This occurs, for example, in the CSV reader where no
 * column is ever INT (nullable or otherwise) and we want our null
 * columns to be (non-nullable) VARCHAR.
 */
@Test
public void testCustomNullType() {
    final List<ResolvedNullColumn> defns = new ArrayList<>();
    defns.add(makeNullCol("unspecified", null));
    defns.add(makeNullCol("nullType", MajorType.newBuilder().setMinorType(MinorType.NULL).setMode(DataMode.OPTIONAL).build()));
    // Null required is an oxymoron, so is not tested.
    // Null type array does not make sense, so is not tested.
    final ResultVectorCache cache = new NullResultVectorCacheImpl(fixture.allocator());
    final MajorType nullType = MajorType.newBuilder().setMinorType(MinorType.VARCHAR).setMode(DataMode.OPTIONAL).build();
    final NullColumnLoader staticLoader = new NullColumnLoader(cache, defns, nullType, false);
    // Create a batch
    final VectorContainer output = staticLoader.load(2);
    // Verify values and types
    final TupleMetadata expectedSchema = new SchemaBuilder().add("unspecified", nullType).add("nullType", nullType).buildSchema();
    final SingleRowSet expected = fixture.rowSetBuilder(expectedSchema).addRow(null, null).addRow(null, null).build();
    RowSetUtilities.verify(expected, fixture.wrap(output));
    staticLoader.close();
}
Also used : ResultVectorCache(org.apache.drill.exec.physical.resultSet.ResultVectorCache) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) MajorType(org.apache.drill.common.types.TypeProtos.MajorType) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) ArrayList(java.util.ArrayList) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) NullResultVectorCacheImpl(org.apache.drill.exec.physical.resultSet.impl.NullResultVectorCacheImpl) VectorContainer(org.apache.drill.exec.record.VectorContainer) SubOperatorTest(org.apache.drill.test.SubOperatorTest) Test(org.junit.Test)

Example 2 with NullResultVectorCacheImpl

use of org.apache.drill.exec.physical.resultSet.impl.NullResultVectorCacheImpl in project drill by apache.

the class TestNullColumnLoader method testDefaultValue.

/**
 * Test the ability to provide a default value for a "null" column.
 * Default values are only allowed for required "null" columns. For
 * nullable columns, NULL is already the default.
 */
@Test
public void testDefaultValue() {
    final List<ResolvedNullColumn> defns = new ArrayList<>();
    defns.add(makeNullCol("int", Types.required(MinorType.INT), "10"));
    defns.add(makeNullCol("str", Types.required(MinorType.VARCHAR), "foo"));
    defns.add(makeNullCol("dub", Types.required(MinorType.FLOAT8), "20.0"));
    final ResultVectorCache cache = new NullResultVectorCacheImpl(fixture.allocator());
    final MajorType nullType = Types.optional(MinorType.VARCHAR);
    final NullColumnLoader staticLoader = new NullColumnLoader(cache, defns, nullType, false);
    // Create a batch
    final VectorContainer output = staticLoader.load(2);
    // Verify values and types
    final TupleMetadata expectedSchema = new SchemaBuilder().add("int", MinorType.INT).add("str", MinorType.VARCHAR).add("dub", MinorType.FLOAT8).buildSchema();
    final SingleRowSet expected = fixture.rowSetBuilder(expectedSchema).addRow(10, "foo", 20.0D).addRow(10, "foo", 20.0D).build();
    RowSetUtilities.verify(expected, fixture.wrap(output));
    staticLoader.close();
}
Also used : ResultVectorCache(org.apache.drill.exec.physical.resultSet.ResultVectorCache) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) MajorType(org.apache.drill.common.types.TypeProtos.MajorType) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) ArrayList(java.util.ArrayList) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) NullResultVectorCacheImpl(org.apache.drill.exec.physical.resultSet.impl.NullResultVectorCacheImpl) VectorContainer(org.apache.drill.exec.record.VectorContainer) SubOperatorTest(org.apache.drill.test.SubOperatorTest) Test(org.junit.Test)

Example 3 with NullResultVectorCacheImpl

use of org.apache.drill.exec.physical.resultSet.impl.NullResultVectorCacheImpl in project drill by apache.

the class TestNullColumnLoader method testSchemaWithConflicts.

/**
 * Test the various conflicts that can occur:
 * <ul>
 * <li>Schema is required, but no default value for null column.</li>
 * <li>Query wants a different type than that in the schema.</li>
 * <li>Query wants a different mode than that in the schema.</li>
 * <ul>
 *
 * The type and mode provided to the builder is that which would result from
 * schema smoothing. The types and modes should usually match, but verify
 * the rules when they don't.
 * <p>
 * Defaults for null columns are ignored: null columns use NULL as the
 * null value.
 */
@Test
public void testSchemaWithConflicts() {
    // Note: upper case names in schema, lower case in "projection" list
    final TupleMetadata outputSchema = new SchemaBuilder().add("IntReq", MinorType.INT).add("StrReq", // No default
    MinorType.VARCHAR).addNullable("IntOpt", MinorType.INT).addNullable("StrOpt", MinorType.VARCHAR).buildSchema();
    outputSchema.metadata("intReq").setDefaultValue("10");
    outputSchema.metadata("intOpt").setDefaultValue("20");
    outputSchema.metadata("strOpt").setDefaultValue("bar");
    final ResultVectorCache cache = new NullResultVectorCacheImpl(fixture.allocator());
    final NullColumnBuilder builder = new NullBuilderBuilder().setNullType(Types.optional(MinorType.VARCHAR)).setOutputSchema(outputSchema).build();
    // Defined, required, no default so --> optional
    builder.add("strReq");
    builder.add("strOpt");
    // Defined, has default, but conflicting type, so default --> null, so --> optional
    builder.add("intReq", Types.required(MinorType.BIGINT));
    // Defined, has default, conflicting mode, so keep default
    builder.add("intOpt", Types.required(MinorType.INT));
    builder.build(cache);
    // Create a batch
    builder.load(2);
    // Verify values and types
    final TupleMetadata expectedSchema = new SchemaBuilder().addNullable("strReq", MinorType.VARCHAR).addNullable("strOpt", MinorType.VARCHAR).addNullable("intReq", MinorType.BIGINT).add("intOpt", MinorType.INT).buildSchema();
    final SingleRowSet expected = fixture.rowSetBuilder(expectedSchema).addRow(null, null, null, 20).addRow(null, null, null, 20).build();
    RowSetUtilities.verify(expected, fixture.wrap(builder.output()));
    builder.close();
}
Also used : ResultVectorCache(org.apache.drill.exec.physical.resultSet.ResultVectorCache) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) NullResultVectorCacheImpl(org.apache.drill.exec.physical.resultSet.impl.NullResultVectorCacheImpl) NullBuilderBuilder(org.apache.drill.exec.physical.impl.scan.project.NullColumnBuilder.NullBuilderBuilder) SubOperatorTest(org.apache.drill.test.SubOperatorTest) Test(org.junit.Test)

Example 4 with NullResultVectorCacheImpl

use of org.apache.drill.exec.physical.resultSet.impl.NullResultVectorCacheImpl in project drill by apache.

the class TestNullColumnLoader method testNullColumnBuilderWithSchema.

/**
 * Test using an output schema, along with a default value property,
 * to define a default value for missing columns.
 */
@Test
public void testNullColumnBuilderWithSchema() {
    // Note: upper case names in schema, lower case in "projection" list
    final TupleMetadata outputSchema = new SchemaBuilder().add("IntReq", MinorType.INT).add("StrReq", MinorType.VARCHAR).addNullable("IntOpt", MinorType.INT).addNullable("StrOpt", MinorType.VARCHAR).addNullable("DubOpt", // No default
    MinorType.FLOAT8).buildSchema();
    outputSchema.metadata("intReq").setDefaultValue("10");
    outputSchema.metadata("strReq").setDefaultValue("foo");
    outputSchema.metadata("intOpt").setDefaultValue("20");
    outputSchema.metadata("strOpt").setDefaultValue("bar");
    final ResultVectorCache cache = new NullResultVectorCacheImpl(fixture.allocator());
    final NullColumnBuilder builder = new NullBuilderBuilder().setNullType(Types.optional(MinorType.VARCHAR)).setOutputSchema(outputSchema).build();
    builder.add("strReq");
    builder.add("strOpt");
    builder.add("dubOpt");
    builder.add("intReq");
    builder.add("intOpt");
    builder.add("extra");
    builder.build(cache);
    // Create a batch
    builder.load(2);
    // Verify values and types
    final TupleMetadata expectedSchema = new SchemaBuilder().add("strReq", MinorType.VARCHAR).addNullable("strOpt", MinorType.VARCHAR).addNullable("dubOpt", MinorType.FLOAT8).add("intReq", MinorType.INT).addNullable("intOpt", MinorType.INT).addNullable("extra", MinorType.VARCHAR).buildSchema();
    final SingleRowSet expected = fixture.rowSetBuilder(expectedSchema).addRow("foo", null, null, 10, null, null).addRow("foo", null, null, 10, null, null).build();
    RowSetUtilities.verify(expected, fixture.wrap(builder.output()));
    builder.close();
}
Also used : ResultVectorCache(org.apache.drill.exec.physical.resultSet.ResultVectorCache) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) NullResultVectorCacheImpl(org.apache.drill.exec.physical.resultSet.impl.NullResultVectorCacheImpl) NullBuilderBuilder(org.apache.drill.exec.physical.impl.scan.project.NullColumnBuilder.NullBuilderBuilder) SubOperatorTest(org.apache.drill.test.SubOperatorTest) Test(org.junit.Test)

Example 5 with NullResultVectorCacheImpl

use of org.apache.drill.exec.physical.resultSet.impl.NullResultVectorCacheImpl in project drill by apache.

the class TestRowBatchMerger method testFlatWithNulls.

@Test
public void testFlatWithNulls() {
    // Create the first batch
    RowSetSource first = makeFirst();
    // Create null columns
    NullColumnBuilder builder = new NullBuilderBuilder().build();
    ResolvedRow resolvedTuple = new ResolvedRow(builder);
    resolvedTuple.add(new TestProjection(resolvedTuple, 1));
    resolvedTuple.add(resolvedTuple.nullBuilder().add("null1"));
    resolvedTuple.add(resolvedTuple.nullBuilder().add("null2", Types.optional(MinorType.VARCHAR)));
    resolvedTuple.add(new TestProjection(resolvedTuple, 0));
    // Build the null values
    ResultVectorCache cache = new NullResultVectorCacheImpl(fixture.allocator());
    builder.build(cache);
    builder.load(first.rowSet().rowCount());
    // Do the merge
    VectorContainer output = new VectorContainer(fixture.allocator());
    resolvedTuple.project(first.rowSet().container(), output);
    output.setRecordCount(first.rowSet().rowCount());
    RowSet result = fixture.wrap(output);
    // Verify
    TupleMetadata expectedSchema = new SchemaBuilder().add("a", MinorType.INT).addNullable("null1", MinorType.INT).addNullable("null2", MinorType.VARCHAR).add("d", MinorType.VARCHAR).buildSchema();
    SingleRowSet expected = fixture.rowSetBuilder(expectedSchema).addRow(10, null, null, "barney").addRow(20, null, null, "wilma").build();
    new RowSetComparison(expected).verifyAndClearAll(result);
    builder.close();
}
Also used : SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) RowSet(org.apache.drill.exec.physical.rowSet.RowSet) NullBuilderBuilder(org.apache.drill.exec.physical.impl.scan.project.NullColumnBuilder.NullBuilderBuilder) VectorContainer(org.apache.drill.exec.record.VectorContainer) ResultVectorCache(org.apache.drill.exec.physical.resultSet.ResultVectorCache) RowSetComparison(org.apache.drill.test.rowSet.RowSetComparison) ResolvedRow(org.apache.drill.exec.physical.impl.scan.project.ResolvedTuple.ResolvedRow) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) NullResultVectorCacheImpl(org.apache.drill.exec.physical.resultSet.impl.NullResultVectorCacheImpl) SubOperatorTest(org.apache.drill.test.SubOperatorTest) Test(org.junit.Test)

Aggregations

ResultVectorCache (org.apache.drill.exec.physical.resultSet.ResultVectorCache)15 NullResultVectorCacheImpl (org.apache.drill.exec.physical.resultSet.impl.NullResultVectorCacheImpl)15 SchemaBuilder (org.apache.drill.exec.record.metadata.SchemaBuilder)15 TupleMetadata (org.apache.drill.exec.record.metadata.TupleMetadata)15 SubOperatorTest (org.apache.drill.test.SubOperatorTest)15 Test (org.junit.Test)15 SingleRowSet (org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet)14 NullBuilderBuilder (org.apache.drill.exec.physical.impl.scan.project.NullColumnBuilder.NullBuilderBuilder)7 VectorContainer (org.apache.drill.exec.record.VectorContainer)7 EvfTest (org.apache.drill.categories.EvfTest)5 ResolvedRow (org.apache.drill.exec.physical.impl.scan.project.ResolvedTuple.ResolvedRow)4 RowSet (org.apache.drill.exec.physical.rowSet.RowSet)4 RowSetComparison (org.apache.drill.test.rowSet.RowSetComparison)4 ArrayList (java.util.ArrayList)3 MajorType (org.apache.drill.common.types.TypeProtos.MajorType)3