Search in sources :

Example 1 with ScanFixture

use of org.apache.drill.exec.physical.impl.scan.v3.ScanFixture in project drill by apache.

the class TestFileScan method testImplicitColumns.

/**
 * Basic sanity test of a couple of implicit columns, along
 * with all table columns in table order. Full testing of implicit
 * columns is done on lower-level components.
 */
@Test
public void testImplicitColumns() {
    ReaderCreator creator = negotiator -> {
        MockEarlySchemaReader reader = new MockEarlySchemaReader(negotiator);
        reader.batchLimit = 1;
        return reader;
    };
    // Select table and implicit columns.
    FileScanFixtureBuilder builder = new FileScanFixtureBuilder();
    builder.setProjection("a", "b", "filename", "suffix");
    builder.addReader(creator);
    ScanFixture scanFixture = builder.build();
    ScanOperatorExec scan = scanFixture.scanOp;
    // Expect data and implicit columns
    TupleMetadata expectedSchema = new SchemaBuilder().add("a", MinorType.INT).addNullable("b", MinorType.VARCHAR).add("filename", MinorType.VARCHAR).add("suffix", MinorType.VARCHAR).build();
    SingleRowSet expected = fixture.rowSetBuilder(expectedSchema).addRow(10, "fred", MOCK_FILE_NAME, MOCK_SUFFIX).addRow(20, "wilma", MOCK_FILE_NAME, MOCK_SUFFIX).build();
    // Schema should include implicit columns.
    assertTrue(scan.buildSchema());
    assertEquals(expected.container().getSchema(), scan.batchAccessor().schema());
    scan.batchAccessor().release();
    // Read one batch, should contain implicit columns
    assertTrue(scan.next());
    RowSetUtilities.verify(expected, fixture.wrap(scan.batchAccessor().container()));
    // EOF
    assertFalse(scan.next());
    assertEquals(0, scan.batchAccessor().rowCount());
    scanFixture.close();
}
Also used : RowSetUtilities(org.apache.drill.test.rowSet.RowSetUtilities) MetadataUtils(org.apache.drill.exec.record.metadata.MetadataUtils) Assert.assertTrue(org.junit.Assert.assertTrue) Types(org.apache.drill.common.types.Types) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) Test(org.junit.Test) ScanFixture(org.apache.drill.exec.physical.impl.scan.v3.ScanFixture) Category(org.junit.experimental.categories.Category) MaterializedField(org.apache.drill.exec.record.MaterializedField) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) RowSetLoader(org.apache.drill.exec.physical.resultSet.RowSetLoader) DataMode(org.apache.drill.common.types.TypeProtos.DataMode) Assert.assertFalse(org.junit.Assert.assertFalse) EvfTest(org.apache.drill.categories.EvfTest) MinorType(org.apache.drill.common.types.TypeProtos.MinorType) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) ScanOperatorExec(org.apache.drill.exec.physical.impl.scan.ScanOperatorExec) Assert.assertEquals(org.junit.Assert.assertEquals) BaseMockBatchReader(org.apache.drill.exec.physical.impl.scan.v3.BaseMockBatchReader) ScanFixture(org.apache.drill.exec.physical.impl.scan.v3.ScanFixture) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) ScanOperatorExec(org.apache.drill.exec.physical.impl.scan.ScanOperatorExec) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) Test(org.junit.Test) EvfTest(org.apache.drill.categories.EvfTest)

Example 2 with ScanFixture

use of org.apache.drill.exec.physical.impl.scan.v3.ScanFixture in project drill by apache.

the class TestFileScan method testEmptyProject.

@Test
public void testEmptyProject() {
    ReaderCreator creator = negotiator -> {
        MockEarlySchemaReader reader = new MockEarlySchemaReader(negotiator);
        reader.batchLimit = 1;
        return reader;
    };
    // Select no columns
    FileScanFixtureBuilder builder = new FileScanFixtureBuilder();
    builder.setProjection();
    builder.addReader(creator);
    ScanFixture scanFixture = builder.build();
    ScanOperatorExec scan = scanFixture.scanOp;
    // Expect data and implicit columns
    TupleMetadata expectedSchema = new SchemaBuilder().build();
    SingleRowSet expected = fixture.rowSetBuilder(expectedSchema).addRow().addRow().build();
    // Schema should include implicit columns.
    assertTrue(scan.buildSchema());
    assertEquals(expected.container().getSchema(), scan.batchAccessor().schema());
    scan.batchAccessor().release();
    // Read one batch, should contain implicit columns
    assertTrue(scan.next());
    RowSetUtilities.verify(expected, fixture.wrap(scan.batchAccessor().container()));
    // EOF
    assertFalse(scan.next());
    assertEquals(0, scan.batchAccessor().rowCount());
    scanFixture.close();
}
Also used : RowSetUtilities(org.apache.drill.test.rowSet.RowSetUtilities) MetadataUtils(org.apache.drill.exec.record.metadata.MetadataUtils) Assert.assertTrue(org.junit.Assert.assertTrue) Types(org.apache.drill.common.types.Types) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) Test(org.junit.Test) ScanFixture(org.apache.drill.exec.physical.impl.scan.v3.ScanFixture) Category(org.junit.experimental.categories.Category) MaterializedField(org.apache.drill.exec.record.MaterializedField) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) RowSetLoader(org.apache.drill.exec.physical.resultSet.RowSetLoader) DataMode(org.apache.drill.common.types.TypeProtos.DataMode) Assert.assertFalse(org.junit.Assert.assertFalse) EvfTest(org.apache.drill.categories.EvfTest) MinorType(org.apache.drill.common.types.TypeProtos.MinorType) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) ScanOperatorExec(org.apache.drill.exec.physical.impl.scan.ScanOperatorExec) Assert.assertEquals(org.junit.Assert.assertEquals) BaseMockBatchReader(org.apache.drill.exec.physical.impl.scan.v3.BaseMockBatchReader) ScanFixture(org.apache.drill.exec.physical.impl.scan.v3.ScanFixture) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) ScanOperatorExec(org.apache.drill.exec.physical.impl.scan.ScanOperatorExec) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) Test(org.junit.Test) EvfTest(org.apache.drill.categories.EvfTest)

Example 3 with ScanFixture

use of org.apache.drill.exec.physical.impl.scan.v3.ScanFixture in project drill by apache.

the class TestFileScan method testLateSchemaFileWildcards.

@Test
public void testLateSchemaFileWildcards() {
    // Create a mock reader, return two batches: one schema-only, another with data.
    ReaderCreator creator = negotiator -> {
        MockLateSchemaReader reader = new MockLateSchemaReader(negotiator);
        reader.batchLimit = 1;
        reader.returnDataOnFirst = true;
        return reader;
    };
    // Create the scan operator
    FileScanFixtureBuilder builder = new FileScanFixtureBuilder();
    builder.projectAllWithImplicit(3);
    builder.addReader(creator);
    ScanFixture scanFixture = builder.build();
    ScanOperatorExec scan = scanFixture.scanOp;
    // First batch: build schema. The reader helps: it returns an
    // empty first batch.
    assertTrue(scan.buildSchema());
    assertEquals(0, scan.batchAccessor().rowCount());
    // Create the expected result.
    TupleMetadata expectedSchema = new SchemaBuilder().add("a", MinorType.INT).addNullable("b", MinorType.VARCHAR).add(FileScanUtils.FULLY_QUALIFIED_NAME_COL, MinorType.VARCHAR).add(FileScanUtils.FILE_PATH_COL, MinorType.VARCHAR).add(FileScanUtils.FILE_NAME_COL, MinorType.VARCHAR).add(FileScanUtils.SUFFIX_COL, MinorType.VARCHAR).addNullable(FileScanUtils.partitionColName(0), MinorType.VARCHAR).addNullable(FileScanUtils.partitionColName(1), MinorType.VARCHAR).addNullable(FileScanUtils.partitionColName(2), MinorType.VARCHAR).buildSchema();
    SingleRowSet expected = fixture.rowSetBuilder(expectedSchema).addRow(10, "fred", MOCK_FILE_FQN, MOCK_FILE_DIR_PATH, MOCK_FILE_NAME, MOCK_SUFFIX, MOCK_DIR0, MOCK_DIR1, null).addRow(20, "wilma", MOCK_FILE_FQN, MOCK_FILE_DIR_PATH, MOCK_FILE_NAME, MOCK_SUFFIX, MOCK_DIR0, MOCK_DIR1, null).build();
    assertEquals(expected.batchSchema(), scan.batchAccessor().schema());
    // Next call, return with data.
    assertTrue(scan.next());
    RowSetUtilities.verify(expected, fixture.wrap(scan.batchAccessor().container()));
    // EOF
    assertFalse(scan.next());
    assertEquals(0, scan.batchAccessor().rowCount());
    scanFixture.close();
}
Also used : RowSetUtilities(org.apache.drill.test.rowSet.RowSetUtilities) MetadataUtils(org.apache.drill.exec.record.metadata.MetadataUtils) Assert.assertTrue(org.junit.Assert.assertTrue) Types(org.apache.drill.common.types.Types) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) Test(org.junit.Test) ScanFixture(org.apache.drill.exec.physical.impl.scan.v3.ScanFixture) Category(org.junit.experimental.categories.Category) MaterializedField(org.apache.drill.exec.record.MaterializedField) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) RowSetLoader(org.apache.drill.exec.physical.resultSet.RowSetLoader) DataMode(org.apache.drill.common.types.TypeProtos.DataMode) Assert.assertFalse(org.junit.Assert.assertFalse) EvfTest(org.apache.drill.categories.EvfTest) MinorType(org.apache.drill.common.types.TypeProtos.MinorType) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) ScanOperatorExec(org.apache.drill.exec.physical.impl.scan.ScanOperatorExec) Assert.assertEquals(org.junit.Assert.assertEquals) BaseMockBatchReader(org.apache.drill.exec.physical.impl.scan.v3.BaseMockBatchReader) ScanFixture(org.apache.drill.exec.physical.impl.scan.v3.ScanFixture) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) ScanOperatorExec(org.apache.drill.exec.physical.impl.scan.ScanOperatorExec) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) Test(org.junit.Test) EvfTest(org.apache.drill.categories.EvfTest)

Example 4 with ScanFixture

use of org.apache.drill.exec.physical.impl.scan.v3.ScanFixture in project drill by apache.

the class TestFileScan method testFullProject.

/**
 * Exercise the major project operations: subset of table
 * columns, implicit, partition, missing columns, and output
 * order (and positions) different than table. These cases
 * are more fully test on lower level components; here we verify
 * that the components are wired up correctly.
 */
@Test
public void testFullProject() {
    ReaderCreator creator = negotiator -> {
        MockEarlySchemaReader reader = new MockEarlySchemaReader(negotiator);
        reader.batchLimit = 1;
        return reader;
    };
    // Select table and implicit columns.
    FileScanFixtureBuilder builder = new FileScanFixtureBuilder();
    builder.setProjection("dir0", "b", "filename", "c", "suffix");
    builder.addReader(creator);
    ScanFixture scanFixture = builder.build();
    ScanOperatorExec scan = scanFixture.scanOp;
    // Expect data and implicit columns
    TupleMetadata expectedSchema = new SchemaBuilder().addNullable("dir0", MinorType.VARCHAR).addNullable("b", MinorType.VARCHAR).add("filename", MinorType.VARCHAR).addNullable("c", MinorType.INT).add("suffix", MinorType.VARCHAR).build();
    SingleRowSet expected = fixture.rowSetBuilder(expectedSchema).addRow(MOCK_DIR0, "fred", MOCK_FILE_NAME, null, MOCK_SUFFIX).addRow(MOCK_DIR0, "wilma", MOCK_FILE_NAME, null, MOCK_SUFFIX).build();
    // Schema should include implicit columns.
    assertTrue(scan.buildSchema());
    assertEquals(expected.container().getSchema(), scan.batchAccessor().schema());
    scan.batchAccessor().release();
    // Read one batch, should contain implicit columns
    assertTrue(scan.next());
    RowSetUtilities.verify(expected, fixture.wrap(scan.batchAccessor().container()));
    // EOF
    assertFalse(scan.next());
    assertEquals(0, scan.batchAccessor().rowCount());
    scanFixture.close();
}
Also used : RowSetUtilities(org.apache.drill.test.rowSet.RowSetUtilities) MetadataUtils(org.apache.drill.exec.record.metadata.MetadataUtils) Assert.assertTrue(org.junit.Assert.assertTrue) Types(org.apache.drill.common.types.Types) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) Test(org.junit.Test) ScanFixture(org.apache.drill.exec.physical.impl.scan.v3.ScanFixture) Category(org.junit.experimental.categories.Category) MaterializedField(org.apache.drill.exec.record.MaterializedField) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) RowSetLoader(org.apache.drill.exec.physical.resultSet.RowSetLoader) DataMode(org.apache.drill.common.types.TypeProtos.DataMode) Assert.assertFalse(org.junit.Assert.assertFalse) EvfTest(org.apache.drill.categories.EvfTest) MinorType(org.apache.drill.common.types.TypeProtos.MinorType) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) ScanOperatorExec(org.apache.drill.exec.physical.impl.scan.ScanOperatorExec) Assert.assertEquals(org.junit.Assert.assertEquals) BaseMockBatchReader(org.apache.drill.exec.physical.impl.scan.v3.BaseMockBatchReader) ScanFixture(org.apache.drill.exec.physical.impl.scan.v3.ScanFixture) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) ScanOperatorExec(org.apache.drill.exec.physical.impl.scan.ScanOperatorExec) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) Test(org.junit.Test) EvfTest(org.apache.drill.categories.EvfTest)

Example 5 with ScanFixture

use of org.apache.drill.exec.physical.impl.scan.v3.ScanFixture in project drill by apache.

the class TestFileScan method testMapProject.

@Test
public void testMapProject() {
    ReaderCreator creator = negotiator -> {
        MockMapReader reader = new MockMapReader(negotiator);
        reader.batchLimit = 1;
        return reader;
    };
    // Select one of the two map columns
    FileScanFixtureBuilder builder = new FileScanFixtureBuilder();
    builder.setProjection("m1.a");
    builder.addReader(creator);
    ScanFixture scanFixture = builder.build();
    ScanOperatorExec scan = scanFixture.scanOp;
    // Expect data and implicit columns
    TupleMetadata expectedSchema = new SchemaBuilder().addMap("m1").add("a", MinorType.INT).resumeSchema().build();
    SingleRowSet expected = fixture.rowSetBuilder(expectedSchema).addSingleCol(new Object[] { 10 }).addSingleCol(new Object[] { 20 }).build();
    assertTrue(scan.buildSchema());
    assertEquals(expected.container().getSchema(), scan.batchAccessor().schema());
    scan.batchAccessor().release();
    assertTrue(scan.next());
    RowSetUtilities.verify(expected, fixture.wrap(scan.batchAccessor().container()));
    // EOF
    assertFalse(scan.next());
    assertEquals(0, scan.batchAccessor().rowCount());
    scanFixture.close();
}
Also used : RowSetUtilities(org.apache.drill.test.rowSet.RowSetUtilities) MetadataUtils(org.apache.drill.exec.record.metadata.MetadataUtils) Assert.assertTrue(org.junit.Assert.assertTrue) Types(org.apache.drill.common.types.Types) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) Test(org.junit.Test) ScanFixture(org.apache.drill.exec.physical.impl.scan.v3.ScanFixture) Category(org.junit.experimental.categories.Category) MaterializedField(org.apache.drill.exec.record.MaterializedField) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) RowSetLoader(org.apache.drill.exec.physical.resultSet.RowSetLoader) DataMode(org.apache.drill.common.types.TypeProtos.DataMode) Assert.assertFalse(org.junit.Assert.assertFalse) EvfTest(org.apache.drill.categories.EvfTest) MinorType(org.apache.drill.common.types.TypeProtos.MinorType) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) ScanOperatorExec(org.apache.drill.exec.physical.impl.scan.ScanOperatorExec) Assert.assertEquals(org.junit.Assert.assertEquals) BaseMockBatchReader(org.apache.drill.exec.physical.impl.scan.v3.BaseMockBatchReader) ScanFixture(org.apache.drill.exec.physical.impl.scan.v3.ScanFixture) SingleRowSet(org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet) ScanOperatorExec(org.apache.drill.exec.physical.impl.scan.ScanOperatorExec) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) Test(org.junit.Test) EvfTest(org.apache.drill.categories.EvfTest)

Aggregations

EvfTest (org.apache.drill.categories.EvfTest)5 DataMode (org.apache.drill.common.types.TypeProtos.DataMode)5 MinorType (org.apache.drill.common.types.TypeProtos.MinorType)5 Types (org.apache.drill.common.types.Types)5 ScanOperatorExec (org.apache.drill.exec.physical.impl.scan.ScanOperatorExec)5 BaseMockBatchReader (org.apache.drill.exec.physical.impl.scan.v3.BaseMockBatchReader)5 ScanFixture (org.apache.drill.exec.physical.impl.scan.v3.ScanFixture)5 RowSetLoader (org.apache.drill.exec.physical.resultSet.RowSetLoader)5 SingleRowSet (org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet)5 MaterializedField (org.apache.drill.exec.record.MaterializedField)5 MetadataUtils (org.apache.drill.exec.record.metadata.MetadataUtils)5 SchemaBuilder (org.apache.drill.exec.record.metadata.SchemaBuilder)5 TupleMetadata (org.apache.drill.exec.record.metadata.TupleMetadata)5 RowSetUtilities (org.apache.drill.test.rowSet.RowSetUtilities)5 Assert.assertEquals (org.junit.Assert.assertEquals)5 Assert.assertFalse (org.junit.Assert.assertFalse)5 Assert.assertTrue (org.junit.Assert.assertTrue)5 Test (org.junit.Test)5 Category (org.junit.experimental.categories.Category)5