Search in sources :

Example 61 with PipelineResult

use of org.apache.beam.sdk.PipelineResult in project beam by apache.

the class BigQueryIOIT method testWrite.

private void testWrite(BigQueryIO.Write<byte[]> writeIO, String metricName) {
    Pipeline pipeline = Pipeline.create(options);
    BigQueryIO.Write.Method method = BigQueryIO.Write.Method.valueOf(options.getWriteMethod());
    pipeline.apply("Read from source", Read.from(new SyntheticBoundedSource(sourceOptions))).apply("Gather time", ParDo.of(new TimeMonitor<>(NAMESPACE, metricName))).apply("Map records", ParDo.of(new MapKVToV())).apply("Write to BQ", writeIO.to(tableQualifier).withCustomGcsTempLocation(ValueProvider.StaticValueProvider.of(tempRoot)).withMethod(method).withSchema(new TableSchema().setFields(Collections.singletonList(new TableFieldSchema().setName("data").setType("BYTES")))));
    PipelineResult pipelineResult = pipeline.run();
    pipelineResult.waitUntilFinish();
    extractAndPublishTime(pipelineResult, metricName);
}
Also used : SyntheticBoundedSource(org.apache.beam.sdk.io.synthetic.SyntheticBoundedSource) TimeMonitor(org.apache.beam.sdk.testutils.metrics.TimeMonitor) TableSchema(com.google.api.services.bigquery.model.TableSchema) PipelineResult(org.apache.beam.sdk.PipelineResult) TableFieldSchema(com.google.api.services.bigquery.model.TableFieldSchema) Pipeline(org.apache.beam.sdk.Pipeline)

Example 62 with PipelineResult

use of org.apache.beam.sdk.PipelineResult in project beam by apache.

the class BigQueryIOIT method testRead.

private void testRead() {
    Pipeline pipeline = Pipeline.create(options);
    pipeline.apply("Read from BQ", BigQueryIO.readTableRows().from(tableQualifier)).apply("Gather time", ParDo.of(new TimeMonitor<>(NAMESPACE, READ_TIME_METRIC_NAME)));
    PipelineResult result = pipeline.run();
    result.waitUntilFinish();
    extractAndPublishTime(result, READ_TIME_METRIC_NAME);
}
Also used : TimeMonitor(org.apache.beam.sdk.testutils.metrics.TimeMonitor) PipelineResult(org.apache.beam.sdk.PipelineResult) Pipeline(org.apache.beam.sdk.Pipeline)

Example 63 with PipelineResult

use of org.apache.beam.sdk.PipelineResult in project beam by apache.

the class MetricsReaderTest method doesntThrowIllegalStateExceptionWhenThereIsNoMetricFound.

@Test
public void doesntThrowIllegalStateExceptionWhenThereIsNoMetricFound() {
    PipelineResult result = testPipeline.run();
    MetricsReader reader = new MetricsReader(result, NAMESPACE);
    reader.getCounterMetric("nonexistent");
}
Also used : PipelineResult(org.apache.beam.sdk.PipelineResult) Test(org.junit.Test)

Example 64 with PipelineResult

use of org.apache.beam.sdk.PipelineResult in project beam by apache.

the class MetricsReaderTest method testCounterMetricReceivedFromPipelineResult.

@Test
public void testCounterMetricReceivedFromPipelineResult() {
    List<Integer> sampleInputData = Arrays.asList(1, 1, 1, 1, 1);
    createTestPipeline(sampleInputData, new MonitorWithCounter());
    PipelineResult result = testPipeline.run();
    MetricsReader reader = new MetricsReader(result, NAMESPACE);
    assertEquals(5, reader.getCounterMetric("counter"));
}
Also used : PipelineResult(org.apache.beam.sdk.PipelineResult) Test(org.junit.Test)

Example 65 with PipelineResult

use of org.apache.beam.sdk.PipelineResult in project beam by apache.

the class TpcdsRun method call.

@Override
public TpcdsRunResult call() {
    TpcdsRunResult tpcdsRunResult;
    try {
        PipelineResult pipelineResult = pipeline.run();
        long startTimeStamp = System.currentTimeMillis();
        State state = pipelineResult.waitUntilFinish();
        long endTimeStamp = System.currentTimeMillis();
        // Make sure to set the job status to be successful only when pipelineResult's final state is
        // DONE.
        boolean isSuccessful = state == State.DONE;
        tpcdsRunResult = new TpcdsRunResult(isSuccessful, startTimeStamp, endTimeStamp, pipeline.getOptions(), pipelineResult);
    } catch (Exception e) {
        // If the pipeline execution failed, return a result with failed status but don't interrupt
        // other threads.
        e.printStackTrace();
        tpcdsRunResult = new TpcdsRunResult(false, 0, 0, pipeline.getOptions(), null);
    }
    return tpcdsRunResult;
}
Also used : State(org.apache.beam.sdk.PipelineResult.State) PipelineResult(org.apache.beam.sdk.PipelineResult)

Aggregations

PipelineResult (org.apache.beam.sdk.PipelineResult)105 Test (org.junit.Test)66 Pipeline (org.apache.beam.sdk.Pipeline)29 TestPipeline (org.apache.beam.sdk.testing.TestPipeline)18 PCollection (org.apache.beam.sdk.values.PCollection)18 TimeMonitor (org.apache.beam.sdk.testutils.metrics.TimeMonitor)14 ArrayList (java.util.ArrayList)12 Category (org.junit.experimental.categories.Category)12 KV (org.apache.beam.sdk.values.KV)11 Rule (org.junit.Rule)11 IOException (java.io.IOException)10 ExampleUtils (org.apache.beam.examples.common.ExampleUtils)10 DoFn (org.apache.beam.sdk.transforms.DoFn)10 HashingFn (org.apache.beam.sdk.io.common.HashingFn)9 RunWith (org.junit.runner.RunWith)9 MetricQueryResults (org.apache.beam.sdk.metrics.MetricQueryResults)8 ParDo (org.apache.beam.sdk.transforms.ParDo)8 Duration (org.joda.time.Duration)8 Map (java.util.Map)7 TableReference (com.google.api.services.bigquery.model.TableReference)6