Examples with BeamIOSourceRel org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel used on opensource projects

Example 6 with BeamIOSourceRel

use of org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel in project beam by apache.

the class TestTableProviderWithProjectPushDown method testIOSourceRel_AllFields.

@Test
public void testIOSourceRel_AllFields() {
    String selectTableStatement = "SELECT * FROM TEST";
    BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
    PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
    assertEquals(result.getSchema(), BASIC_SCHEMA);
    PAssert.that(result).containsInAnyOrder(row(result.getSchema(), 100, 1, "one", 100), row(result.getSchema(), 200, 2, "two", 200));
    assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
    assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
    // If project push-down succeeds new BeamIOSourceRel should not output unused fields
    assertThat(beamRelNode.getInput(0).getRowType().getFieldNames(), containsInAnyOrder("unused1", "id", "name", "unused2"));
    pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
}

Example 7 with BeamIOSourceRel

use of org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel in project beam by apache.

the class TestTableProviderWithProjectPushDown method testIOSourceRel_withNoPredicate_withRename.

@Test
public void testIOSourceRel_withNoPredicate_withRename() {
    String selectTableStatement = "SELECT id as new_id, name as new_name FROM TEST";
    BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
    PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
    assertEquals(result.getSchema(), Schema.builder().addInt32Field("new_id").addStringField("new_name").build());
    PAssert.that(result).containsInAnyOrder(row(result.getSchema(), 1, "one"), row(result.getSchema(), 2, "two"));
    assertThat(beamRelNode, instanceOf(BeamIOSourceRel.class));
    // If project push-down succeeds new BeamIOSourceRel should not output unused fields
    assertThat(beamRelNode.getRowType().getFieldNames(), containsInAnyOrder("new_id", "new_name"));
    pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
}

Example 8 with BeamIOSourceRel

use of org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel in project beam by apache.

the class TestTableProviderWithProjectPushDown method testIOSourceRel_withNoPredicate.

@Test
public void testIOSourceRel_withNoPredicate() {
    String selectTableStatement = "SELECT id, name FROM TEST";
    BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
    PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
    assertEquals(result.getSchema(), Schema.builder().addInt32Field("id").addStringField("name").build());
    PAssert.that(result).containsInAnyOrder(row(result.getSchema(), 1, "one"), row(result.getSchema(), 2, "two"));
    assertThat(beamRelNode, instanceOf(BeamIOSourceRel.class));
    // If project push-down succeeds new BeamIOSourceRel should not output unused fields
    assertThat(beamRelNode.getRowType().getFieldNames(), containsInAnyOrder("id", "name"));
    pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
}

Example 9 with BeamIOSourceRel

use of org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel in project beam by apache.

the class BeamAggregateProjectMergeRuleTest method testBeamAggregateProjectMergeRule_withProjectTable_withPredicate.

@Test
public void testBeamAggregateProjectMergeRule_withProjectTable_withPredicate() {
    // When an IO supports project push-down, Projects should be merged with an IO.
    String sqlQuery = "select SUM(id) as id_sum from TEST_PROJECT where unused1=1 group by name";
    BeamRelNode beamRel = sqlEnv.parseQuery(sqlQuery);
    BeamAggregationRel aggregate = (BeamAggregationRel) beamRel.getInput(0);
    BeamCalcRel calc = (BeamCalcRel) aggregate.getInput();
    BeamIOSourceRel ioSourceRel = (BeamIOSourceRel) calc.getInput();
    // Make sure project push-down took place.
    assertThat(ioSourceRel, instanceOf(BeamPushDownIOSourceRel.class));
    assertThat(ioSourceRel.getRowType().getFieldNames(), containsInAnyOrder("name", "id", "unused1"));
}

Example 10 with BeamIOSourceRel

use of org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel in project beam by apache.

the class BeamAggregateProjectMergeRuleTest method testBeamAggregateProjectMergeRule_withFilterTable.

@Test
public void testBeamAggregateProjectMergeRule_withFilterTable() {
    // When an IO does not supports project push-down, Projects should be merged with an aggregate.
    String sqlQuery = "select SUM(id) as id_sum from TEST_FILTER group by name";
    BeamRelNode beamRel = sqlEnv.parseQuery(sqlQuery);
    BeamAggregationRel aggregate = (BeamAggregationRel) beamRel.getInput(0);
    BeamIOSourceRel ioSourceRel = (BeamIOSourceRel) aggregate.getInput();
    // Make sure project merged with an aggregate.
    assertThat(aggregate.getRowType().getFieldNames(), containsInAnyOrder("id_sum", "name"));
    // IO projects al fields.
    assertThat(ioSourceRel, instanceOf(BeamIOSourceRel.class));
    assertThat(ioSourceRel.getRowType().getFieldNames(), containsInAnyOrder("unused1", "name", "id", "unused2"));
}