Search in sources :

Example 1 with IngestRequestModel

use of bio.terra.model.IngestRequestModel in project jade-data-repo by DataBiosphere.

the class IngestSetupStep method doStep.

@Override
public StepResult doStep(FlightContext context) throws InterruptedException {
    if (configService.testInsertFault(ConfigEnum.TABLE_INGEST_LOCK_CONFLICT_STOP_FAULT)) {
        logger.info("TABLE_INGEST_LOCK_CONFLICT_STOP_FAULT");
        while (!configService.testInsertFault(ConfigEnum.TABLE_INGEST_LOCK_CONFLICT_CONTINUE_FAULT)) {
            logger.info("Sleeping for CONTINUE FAULT");
            TimeUnit.SECONDS.sleep(5);
        }
        logger.info("TABLE_INGEST_LOCK_CONFLICT_CONTINUE_FAULT");
    }
    IngestRequestModel ingestRequestModel = IngestUtils.getIngestRequestModel(context);
    // We don't actually care about the output here since BQ takes the raw "gs://" string as input.
    // As long as parsing succeeds, we're good to move forward.
    IngestUtils.parseBlobUri(ingestRequestModel.getPath());
    Dataset dataset = IngestUtils.getDataset(context, datasetService);
    IngestUtils.putDatasetName(context, dataset.getName());
    DatasetTable targetTable = IngestUtils.getDatasetTable(context, dataset);
    String sgName = DatasetUtils.generateAuxTableName(targetTable, "st");
    IngestUtils.putStagingTableName(context, sgName);
    return StepResult.getStepResultSuccess();
}
Also used : Dataset(bio.terra.service.dataset.Dataset) IngestRequestModel(bio.terra.model.IngestRequestModel) DatasetTable(bio.terra.service.dataset.DatasetTable)

Example 2 with IngestRequestModel

use of bio.terra.model.IngestRequestModel in project jade-data-repo by DataBiosphere.

the class IngestUtils method getDatasetTable.

public static DatasetTable getDatasetTable(FlightContext context, Dataset dataset) {
    IngestRequestModel ingestRequest = getIngestRequestModel(context);
    Optional<DatasetTable> optTable = dataset.getTableByName(ingestRequest.getTable());
    if (!optTable.isPresent()) {
        throw new TableNotFoundException("Table not found: " + ingestRequest.getTable());
    }
    return optTable.get();
}
Also used : TableNotFoundException(bio.terra.service.dataset.exception.TableNotFoundException) IngestRequestModel(bio.terra.model.IngestRequestModel) DatasetTable(bio.terra.service.dataset.DatasetTable)

Example 3 with IngestRequestModel

use of bio.terra.model.IngestRequestModel in project jade-data-repo by DataBiosphere.

the class IngestRequestValidator method validate.

@Override
@SuppressFBWarnings(value = "UC_USELESS_VOID_METHOD", justification = "FB mistake - this clearly validates and returns data in errors")
public void validate(@NotNull Object target, Errors errors) {
    if (target instanceof IngestRequestModel) {
        IngestRequestModel ingestRequest = (IngestRequestModel) target;
        validateTableName(ingestRequest.getTable(), errors);
    } else if (target instanceof FileLoadModel) {
        FileLoadModel fileLoadModel = (FileLoadModel) target;
        if (fileLoadModel.getProfileId() == null) {
            errors.rejectValue("profileId", "ProfileIdMissing", "File ingest requires a profile id.");
        }
    }
}
Also used : IngestRequestModel(bio.terra.model.IngestRequestModel) FileLoadModel(bio.terra.model.FileLoadModel) SuppressFBWarnings(edu.umd.cs.findbugs.annotations.SuppressFBWarnings)

Example 4 with IngestRequestModel

use of bio.terra.model.IngestRequestModel in project jade-data-repo by DataBiosphere.

the class IngestInsertIntoDatasetTableStep method doStep.

@Override
public StepResult doStep(FlightContext context) throws InterruptedException {
    Dataset dataset = IngestUtils.getDataset(context, datasetService);
    DatasetTable targetTable = IngestUtils.getDatasetTable(context, dataset);
    String stagingTableName = IngestUtils.getStagingTableName(context);
    IngestRequestModel ingestRequest = IngestUtils.getIngestRequestModel(context);
    PdaoLoadStatistics loadStatistics = IngestUtils.getIngestStatistics(context);
    IngestResponseModel ingestResponse = new IngestResponseModel().dataset(dataset.getName()).datasetId(dataset.getId().toString()).table(ingestRequest.getTable()).path(ingestRequest.getPath()).loadTag(ingestRequest.getLoadTag()).badRowCount(loadStatistics.getBadRecords()).rowCount(loadStatistics.getRowCount());
    context.getWorkingMap().put(JobMapKeys.RESPONSE.getKeyName(), ingestResponse);
    bigQueryPdao.insertIntoDatasetTable(dataset, targetTable, stagingTableName);
    return StepResult.getStepResultSuccess();
}
Also used : Dataset(bio.terra.service.dataset.Dataset) IngestRequestModel(bio.terra.model.IngestRequestModel) DatasetTable(bio.terra.service.dataset.DatasetTable) PdaoLoadStatistics(bio.terra.common.PdaoLoadStatistics) IngestResponseModel(bio.terra.model.IngestResponseModel)

Example 5 with IngestRequestModel

use of bio.terra.model.IngestRequestModel in project jade-data-repo by DataBiosphere.

the class IngestLoadTableStep method doStep.

@Override
public StepResult doStep(FlightContext context) throws InterruptedException {
    Dataset dataset = IngestUtils.getDataset(context, datasetService);
    DatasetTable targetTable = IngestUtils.getDatasetTable(context, dataset);
    String stagingTableName = IngestUtils.getStagingTableName(context);
    IngestRequestModel ingestRequest = IngestUtils.getIngestRequestModel(context);
    PdaoLoadStatistics ingestStatistics = bigQueryPdao.loadToStagingTable(dataset, targetTable, stagingTableName, ingestRequest);
    // Save away the stats in the working map. We will use some of them later
    // when we make the annotations. Others are returned on the ingest response.
    IngestUtils.putIngestStatistics(context, ingestStatistics);
    return StepResult.getStepResultSuccess();
}
Also used : Dataset(bio.terra.service.dataset.Dataset) IngestRequestModel(bio.terra.model.IngestRequestModel) DatasetTable(bio.terra.service.dataset.DatasetTable) PdaoLoadStatistics(bio.terra.common.PdaoLoadStatistics)

Aggregations

IngestRequestModel (bio.terra.model.IngestRequestModel)37 Test (org.junit.Test)27 SpringBootTest (org.springframework.boot.test.context.SpringBootTest)27 IngestResponseModel (bio.terra.model.IngestResponseModel)16 BlobInfo (com.google.cloud.storage.BlobInfo)11 DatasetSummaryModel (bio.terra.model.DatasetSummaryModel)10 SnapshotSummaryModel (bio.terra.model.SnapshotSummaryModel)10 CoreMatchers.containsString (org.hamcrest.CoreMatchers.containsString)9 JobModel (bio.terra.model.JobModel)8 Storage (com.google.cloud.storage.Storage)8 MockHttpServletResponse (org.springframework.mock.web.MockHttpServletResponse)7 MvcResult (org.springframework.test.web.servlet.MvcResult)7 SnapshotModel (bio.terra.model.SnapshotModel)6 Dataset (bio.terra.service.dataset.Dataset)6 ErrorModel (bio.terra.model.ErrorModel)4 FileModel (bio.terra.model.FileModel)4 DatasetTable (bio.terra.service.dataset.DatasetTable)4 BigQuery (com.google.cloud.bigquery.BigQuery)4 Blob (com.google.cloud.storage.Blob)4 DRSObject (bio.terra.model.DRSObject)3