Search in sources :

Example 1 with BigQueryToElasticsearchOptions

use of com.google.cloud.teleport.v2.elasticsearch.options.BigQueryToElasticsearchOptions in project DataflowTemplates by GoogleCloudPlatform.

the class BigQueryToElasticsearch method main.

/**
 * Main entry point for pipeline execution.
 *
 * @param args Command line arguments to the pipeline.
 */
public static void main(String[] args) {
    BigQueryToElasticsearchOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(BigQueryToElasticsearchOptions.class);
    run(options);
}
Also used : BigQueryToElasticsearchOptions(com.google.cloud.teleport.v2.elasticsearch.options.BigQueryToElasticsearchOptions)

Example 2 with BigQueryToElasticsearchOptions

use of com.google.cloud.teleport.v2.elasticsearch.options.BigQueryToElasticsearchOptions in project DataflowTemplates by GoogleCloudPlatform.

the class BigQueryToElasticsearch method run.

/**
 * Runs the pipeline with the supplied options.
 *
 * @param options The execution parameters to the pipeline.
 * @return The result of the pipeline execution.
 */
private static PipelineResult run(BigQueryToElasticsearchOptions options) {
    // Create the pipeline.
    Pipeline pipeline = Pipeline.create(options);
    /*
     * Steps: 1) Read records from BigQuery via BigQueryIO.
     *        2) Create json string from Table Row.
     *        3) Write records to Elasticsearch.
     *
     *
     * Step #1: Read from BigQuery. If a query is provided then it is used to get the TableRows.
     */
    pipeline.apply("ReadFromBigQuery", ReadBigQuery.newBuilder().setOptions(options.as(BigQueryToElasticsearchOptions.class)).build()).apply("TableRowsToJsonDocument", ParDo.of(new TableRowToJsonFn())).apply("WriteToElasticsearch", WriteToElasticsearch.newBuilder().setOptions(options.as(BigQueryToElasticsearchOptions.class)).build());
    return pipeline.run();
}
Also used : BigQueryToElasticsearchOptions(com.google.cloud.teleport.v2.elasticsearch.options.BigQueryToElasticsearchOptions) TableRowToJsonFn(com.google.cloud.teleport.v2.transforms.BigQueryConverters.TableRowToJsonFn) Pipeline(org.apache.beam.sdk.Pipeline)

Aggregations

BigQueryToElasticsearchOptions (com.google.cloud.teleport.v2.elasticsearch.options.BigQueryToElasticsearchOptions)2 TableRowToJsonFn (com.google.cloud.teleport.v2.transforms.BigQueryConverters.TableRowToJsonFn)1 Pipeline (org.apache.beam.sdk.Pipeline)1