Search in sources :

Example 16 with JobReference

use of com.google.api.services.bigquery.model.JobReference in project beam by apache.

the class WriteTables method load.

private void load(JobService jobService, DatasetService datasetService, String jobIdPrefix, TableReference ref, @Nullable TableSchema schema, List<String> gcsUris, WriteDisposition writeDisposition, CreateDisposition createDisposition, @Nullable String tableDescription) throws InterruptedException, IOException {
    JobConfigurationLoad loadConfig = new JobConfigurationLoad().setDestinationTable(ref).setSchema(schema).setSourceUris(gcsUris).setWriteDisposition(writeDisposition.name()).setCreateDisposition(createDisposition.name()).setSourceFormat("NEWLINE_DELIMITED_JSON");
    String projectId = ref.getProjectId();
    Job lastFailedLoadJob = null;
    for (int i = 0; i < BatchLoads.MAX_RETRY_JOBS; ++i) {
        String jobId = jobIdPrefix + "-" + i;
        JobReference jobRef = new JobReference().setProjectId(projectId).setJobId(jobId);
        jobService.startLoadJob(jobRef, loadConfig);
        Job loadJob = jobService.pollJob(jobRef, BatchLoads.LOAD_JOB_POLL_MAX_RETRIES);
        Status jobStatus = BigQueryHelpers.parseStatus(loadJob);
        switch(jobStatus) {
            case SUCCEEDED:
                if (tableDescription != null) {
                    datasetService.patchTableDescription(ref, tableDescription);
                }
                return;
            case UNKNOWN:
                throw new RuntimeException(String.format("UNKNOWN status of load job [%s]: %s.", jobId, BigQueryHelpers.jobToPrettyString(loadJob)));
            case FAILED:
                lastFailedLoadJob = loadJob;
                continue;
            default:
                throw new IllegalStateException(String.format("Unexpected status [%s] of load job: %s.", jobStatus, BigQueryHelpers.jobToPrettyString(loadJob)));
        }
    }
    throw new RuntimeException(String.format("Failed to create load job with id prefix %s, " + "reached max retries: %d, last failed load job: %s.", jobIdPrefix, BatchLoads.MAX_RETRY_JOBS, BigQueryHelpers.jobToPrettyString(lastFailedLoadJob)));
}
Also used : Status(org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.Status) JobConfigurationLoad(com.google.api.services.bigquery.model.JobConfigurationLoad) JobReference(com.google.api.services.bigquery.model.JobReference) Job(com.google.api.services.bigquery.model.Job)

Aggregations

JobReference (com.google.api.services.bigquery.model.JobReference)16 Job (com.google.api.services.bigquery.model.Job)15 Test (org.junit.Test)11 JobStatus (com.google.api.services.bigquery.model.JobStatus)7 JobServiceImpl (org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.JobServiceImpl)6 MockSleeper (com.google.api.client.testing.util.MockSleeper)3 Sleeper (com.google.api.client.util.Sleeper)3 JobConfiguration (com.google.api.services.bigquery.model.JobConfiguration)3 JobConfigurationQuery (com.google.api.services.bigquery.model.JobConfigurationQuery)3 JobStatistics (com.google.api.services.bigquery.model.JobStatistics)3 TableReference (com.google.api.services.bigquery.model.TableReference)3 ApiErrorExtractor (com.google.cloud.hadoop.util.ApiErrorExtractor)3 IOException (java.io.IOException)3 FastNanoClockAndSleeper (org.apache.beam.sdk.util.FastNanoClockAndSleeper)3 Dataset (com.google.api.services.bigquery.model.Dataset)2 ErrorProto (com.google.api.services.bigquery.model.ErrorProto)2 JobStatistics2 (com.google.api.services.bigquery.model.JobStatistics2)2 TableRow (com.google.api.services.bigquery.model.TableRow)2 Status (org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.Status)2 Matchers.containsString (org.hamcrest.Matchers.containsString)2