Search in sources :

Example 21 with TableRow

use of com.google.api.services.bigquery.model.TableRow in project beam by apache.

the class BigQueryTornadoesTest method testFormatCounts.

@Test
@SuppressWarnings({ "rawtypes", "unchecked" })
public void testFormatCounts() throws Exception {
    DoFnTester<KV<Integer, Long>, TableRow> formatCountsFn = DoFnTester.of(new FormatCountsFn());
    KV[] empty = {};
    List<TableRow> results = formatCountsFn.processBundle(empty);
    Assert.assertTrue(results.size() == 0);
    KV[] input = { KV.of(3, 0L), KV.of(4, Long.MAX_VALUE), KV.of(5, Long.MIN_VALUE) };
    results = formatCountsFn.processBundle(input);
    Assert.assertEquals(results.size(), 3);
    Assert.assertEquals(results.get(0).get("month"), 3);
    Assert.assertEquals(results.get(0).get("tornado_count"), 0L);
    Assert.assertEquals(results.get(1).get("month"), 4);
    Assert.assertEquals(results.get(1).get("tornado_count"), Long.MAX_VALUE);
    Assert.assertEquals(results.get(2).get("month"), 5);
    Assert.assertEquals(results.get(2).get("tornado_count"), Long.MIN_VALUE);
}
Also used : FormatCountsFn(org.apache.beam.examples.cookbook.BigQueryTornadoes.FormatCountsFn) TableRow(com.google.api.services.bigquery.model.TableRow) KV(org.apache.beam.sdk.values.KV) Test(org.junit.Test)

Example 22 with TableRow

use of com.google.api.services.bigquery.model.TableRow in project beam by apache.

the class CombinePerKeyExamplesTest method testFormatShakespeareOutputFn.

@Test
public void testFormatShakespeareOutputFn() throws Exception {
    DoFnTester<KV<String, String>, TableRow> formatShakespeareOutputFn = DoFnTester.of(new FormatShakespeareOutputFn());
    List<TableRow> results = formatShakespeareOutputFn.processBundle(COMBINED_TUPLES_ARRAY);
    Assert.assertThat(results, CoreMatchers.hasItem(resultRow1));
    Assert.assertThat(results, CoreMatchers.hasItem(resultRow2));
}
Also used : TableRow(com.google.api.services.bigquery.model.TableRow) KV(org.apache.beam.sdk.values.KV) FormatShakespeareOutputFn(org.apache.beam.examples.cookbook.CombinePerKeyExamples.FormatShakespeareOutputFn) Test(org.junit.Test)

Example 23 with TableRow

use of com.google.api.services.bigquery.model.TableRow in project beam by apache.

the class FilterExamplesTest method testProjectionFn.

@Test
public void testProjectionFn() throws Exception {
    DoFnTester<TableRow, TableRow> projectionFn = DoFnTester.of(new ProjectionFn());
    List<TableRow> results = projectionFn.processBundle(ROWS_ARRAY);
    Assert.assertThat(results, CoreMatchers.hasItem(outRow1));
    Assert.assertThat(results, CoreMatchers.hasItem(outRow2));
    Assert.assertThat(results, CoreMatchers.hasItem(outRow3));
}
Also used : ProjectionFn(org.apache.beam.examples.cookbook.FilterExamples.ProjectionFn) TableRow(com.google.api.services.bigquery.model.TableRow) Test(org.junit.Test)

Example 24 with TableRow

use of com.google.api.services.bigquery.model.TableRow in project beam by apache.

the class FilterExamplesTest method testFilterSingleMonthDataFn.

@Test
public void testFilterSingleMonthDataFn() throws Exception {
    DoFnTester<TableRow, TableRow> filterSingleMonthDataFn = DoFnTester.of(new FilterSingleMonthDataFn(7));
    List<TableRow> results = filterSingleMonthDataFn.processBundle(PROJROWS_ARRAY);
    Assert.assertThat(results, CoreMatchers.hasItem(outRow2));
}
Also used : TableRow(com.google.api.services.bigquery.model.TableRow) FilterSingleMonthDataFn(org.apache.beam.examples.cookbook.FilterExamples.FilterSingleMonthDataFn) Test(org.junit.Test)

Example 25 with TableRow

use of com.google.api.services.bigquery.model.TableRow in project beam by apache.

the class JoinExamplesTest method testExtractEventDataFn.

@Test
public void testExtractEventDataFn() throws Exception {
    DoFnTester<TableRow, KV<String, String>> extractEventDataFn = DoFnTester.of(new ExtractEventDataFn());
    List<KV<String, String>> results = extractEventDataFn.processBundle(EVENTS);
    Assert.assertThat(results, CoreMatchers.hasItem(kv1));
    Assert.assertThat(results, CoreMatchers.hasItem(kv2));
}
Also used : ExtractEventDataFn(org.apache.beam.examples.cookbook.JoinExamples.ExtractEventDataFn) TableRow(com.google.api.services.bigquery.model.TableRow) KV(org.apache.beam.sdk.values.KV) Test(org.junit.Test)

Aggregations

TableRow (com.google.api.services.bigquery.model.TableRow)73 Test (org.junit.Test)43 TableReference (com.google.api.services.bigquery.model.TableReference)24 TableSchema (com.google.api.services.bigquery.model.TableSchema)18 Pipeline (org.apache.beam.sdk.Pipeline)16 KV (org.apache.beam.sdk.values.KV)15 TableFieldSchema (com.google.api.services.bigquery.model.TableFieldSchema)14 JsonSchemaToTableSchema (org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.JsonSchemaToTableSchema)14 BigQueryHelpers.toJsonString (org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.toJsonString)13 TestPipeline (org.apache.beam.sdk.testing.TestPipeline)12 BigQueryHelpers.createTempTableReference (org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.createTempTableReference)11 Table (com.google.api.services.bigquery.model.Table)10 HashBasedTable (com.google.common.collect.HashBasedTable)10 JobStatus (com.google.api.services.bigquery.model.JobStatus)9 TableDataInsertAllResponse (com.google.api.services.bigquery.model.TableDataInsertAllResponse)8 ArrayList (java.util.ArrayList)8 List (java.util.List)8 Map (java.util.Map)8 ValueInSingleWindow (org.apache.beam.sdk.values.ValueInSingleWindow)7 JobStatistics (com.google.api.services.bigquery.model.JobStatistics)6