Examples with BeamSqlTable org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable used on opensource projects

Example 1 with BeamSqlTable

use of org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable in project beam by apache.

the class SqlTransform method toTableMap.

@SuppressWarnings("unchecked")
private Map<String, BeamSqlTable> toTableMap(PInput inputs) {
    /**
     * A single PCollection is transformed to a table named PCOLLECTION, other input types are
     * expanded and converted to tables using the tags as names.
     */
    if (inputs instanceof PCollection) {
        PCollection<?> pCollection = (PCollection<?>) inputs;
        return ImmutableMap.of(PCOLLECTION_NAME, new BeamPCollectionTable(pCollection));
    }
    ImmutableMap.Builder<String, BeamSqlTable> tables = ImmutableMap.builder();
    for (Map.Entry<TupleTag<?>, PValue> input : inputs.expand().entrySet()) {
        PCollection<?> pCollection = (PCollection<?>) input.getValue();
        tables.put(input.getKey().getId(), new BeamPCollectionTable(pCollection));
    }
    return tables.build();
}

Example 2 with BeamSqlTable

use of org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable in project beam by apache.

the class BigQueryRowCountIT method testEmptyTable.

@Test
public void testEmptyTable() {
    BigQueryTableProvider provider = new BigQueryTableProvider();
    Table table = getTable("testTable", bigQuery.tableSpec());
    BeamSqlTable sqlTable = provider.buildBeamSqlTable(table);
    BeamTableStatistics size = sqlTable.getTableStatistics(TestPipeline.testingPipelineOptions());
    assertNotNull(size);
    assertEquals(0d, size.getRowCount(), 0.1);
}

Example 3 with BeamSqlTable

use of org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable in project beam by apache.

the class SchemaIOTableProviderWrapperTest method testBuildIOReader.

@Test
public void testBuildIOReader() {
    TestSchemaIOTableProviderWrapper provider = new TestSchemaIOTableProviderWrapper();
    BeamSqlTable beamSqlTable = provider.buildBeamSqlTable(testTable);
    PCollection<Row> result = beamSqlTable.buildIOReader(pipeline.begin());
    PAssert.that(result).containsInAnyOrder(rows);
    pipeline.run();
}

Example 4 with BeamSqlTable

use of org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable in project beam by apache.

the class KafkaTableProviderTest method testBuildBeamSqlProtoTable.

@Test
public void testBuildBeamSqlProtoTable() {
    Table table = mockProtoTable("hello", PayloadMessages.SimpleMessage.class);
    BeamSqlTable sqlTable = provider.buildBeamSqlTable(table);
    assertNotNull(sqlTable);
    assertTrue(sqlTable instanceof BeamKafkaTable);
    BeamKafkaTable kafkaTable = (BeamKafkaTable) sqlTable;
    assertEquals(LOCATION_BROKER, kafkaTable.getBootstrapServers());
    assertEquals(ImmutableList.of(LOCATION_TOPIC), kafkaTable.getTopics());
}

Example 5 with BeamSqlTable

use of org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable in project beam by apache.

the class KafkaTableProviderTest method testBuildBeamSqlAvroTable.

@Test
public void testBuildBeamSqlAvroTable() {
    Table table = mockTable("hello", "avro");
    BeamSqlTable sqlTable = provider.buildBeamSqlTable(table);
    assertNotNull(sqlTable);
    assertTrue(sqlTable instanceof BeamKafkaTable);
    BeamKafkaTable kafkaTable = (BeamKafkaTable) sqlTable;
    assertEquals(LOCATION_BROKER, kafkaTable.getBootstrapServers());
    assertEquals(ImmutableList.of(LOCATION_TOPIC), kafkaTable.getTopics());
}