Search in sources :

Example 6 with SnapshotRequestRowIdModel

use of bio.terra.model.SnapshotRequestRowIdModel in project jade-data-repo by DataBiosphere.

the class BigQueryPdao method createSnapshotWithProvidedIds.

public void createSnapshotWithProvidedIds(Snapshot snapshot, SnapshotRequestContentsModel contentsModel) throws InterruptedException {
    BigQueryProject bigQueryProject = bigQueryProjectForSnapshot(snapshot);
    String projectId = bigQueryProject.getProjectId();
    String snapshotName = snapshot.getName();
    BigQuery bigQuery = bigQueryProject.getBigQuery();
    SnapshotRequestRowIdModel rowIdModel = contentsModel.getRowIdSpec();
    // create snapshot BQ dataset
    snapshotCreateBQDataset(bigQueryProject, snapshot);
    // create the row id table
    bigQueryProject.createTable(snapshotName, PDAO_ROW_ID_TABLE, rowIdTableSchema());
    // populate root row ids. Must happen before the relationship walk.
    // NOTE: when we have multiple sources, we can put this into a loop
    SnapshotSource source = snapshot.getSnapshotSources().get(0);
    String datasetBqDatasetName = prefixName(source.getDataset().getName());
    for (SnapshotRequestRowIdTableModel table : rowIdModel.getTables()) {
        String tableName = table.getTableName();
        Table sourceTable = source.reverseTableLookup(tableName).orElseThrow(() -> new CorruptMetadataException("cannot find destination table: " + tableName));
        List<String> rowIds = table.getRowIds();
        if (rowIds.size() > 0) {
            ST sqlTemplate = new ST(loadRootRowIdsTemplate);
            sqlTemplate.add("project", projectId);
            sqlTemplate.add("snapshot", snapshotName);
            sqlTemplate.add("dataset", datasetBqDatasetName);
            sqlTemplate.add("tableId", sourceTable.getId().toString());
            sqlTemplate.add("rowIds", rowIds);
            bigQueryProject.query(sqlTemplate.render());
        }
        ST sqlTemplate = new ST(validateRowIdsForRootTemplate);
        sqlTemplate.add("project", projectId);
        sqlTemplate.add("snapshot", snapshotName);
        sqlTemplate.add("dataset", datasetBqDatasetName);
        sqlTemplate.add("table", sourceTable.getName());
        TableResult result = bigQueryProject.query(sqlTemplate.render());
        FieldValueList row = result.iterateAll().iterator().next();
        FieldValue countValue = row.get(0);
        if (countValue.getLongValue() != rowIds.size()) {
            logger.error("Invalid row ids supplied: rowIds=" + rowIds.size() + " count=" + countValue.getLongValue());
            for (String rowId : rowIds) {
                logger.error(" rowIdIn: " + rowId);
            }
            throw new PdaoException("Invalid row ids supplied");
        }
    }
    snapshotViewCreation(datasetBqDatasetName, snapshotName, snapshot, projectId, bigQuery, bigQueryProject);
}
Also used : ST(org.stringtemplate.v4.ST) BigQuery(com.google.cloud.bigquery.BigQuery) DatasetTable(bio.terra.service.dataset.DatasetTable) Table(bio.terra.common.Table) AssetTable(bio.terra.service.dataset.AssetTable) SnapshotTable(bio.terra.service.snapshot.SnapshotTable) SnapshotMapTable(bio.terra.service.snapshot.SnapshotMapTable) SnapshotRequestRowIdModel(bio.terra.model.SnapshotRequestRowIdModel) CorruptMetadataException(bio.terra.service.snapshot.exception.CorruptMetadataException) TableResult(com.google.cloud.bigquery.TableResult) SnapshotRequestRowIdTableModel(bio.terra.model.SnapshotRequestRowIdTableModel) PdaoException(bio.terra.common.exception.PdaoException) SnapshotSource(bio.terra.service.snapshot.SnapshotSource) FieldValueList(com.google.cloud.bigquery.FieldValueList) FieldValue(com.google.cloud.bigquery.FieldValue)

Example 7 with SnapshotRequestRowIdModel

use of bio.terra.model.SnapshotRequestRowIdModel in project jade-data-repo by DataBiosphere.

the class SnapshotValidationTest method testSnapshotRowIdsEmptyRowIds.

@Test
public void testSnapshotRowIdsEmptyRowIds() throws Exception {
    SnapshotRequestRowIdModel rowIdSpec = snapshotByRowIdsRequestModel.getContents().get(0).getRowIdSpec();
    rowIdSpec.getTables().get(0).setRowIds(Collections.emptyList());
    expectBadSnapshotCreateRequest(snapshotByRowIdsRequestModel);
}
Also used : SnapshotRequestRowIdModel(bio.terra.model.SnapshotRequestRowIdModel) Test(org.junit.Test) SpringBootTest(org.springframework.boot.test.context.SpringBootTest)

Aggregations

SnapshotRequestRowIdModel (bio.terra.model.SnapshotRequestRowIdModel)7 SnapshotRequestContentsModel (bio.terra.model.SnapshotRequestContentsModel)4 SnapshotRequestRowIdTableModel (bio.terra.model.SnapshotRequestRowIdTableModel)4 ValidationException (bio.terra.app.controller.exception.ValidationException)2 Table (bio.terra.common.Table)2 Query (bio.terra.grammar.Query)2 SnapshotRequestModel (bio.terra.model.SnapshotRequestModel)2 SnapshotRequestQueryModel (bio.terra.model.SnapshotRequestQueryModel)2 AssetSpecification (bio.terra.service.dataset.AssetSpecification)2 AssetTable (bio.terra.service.dataset.AssetTable)2 Dataset (bio.terra.service.dataset.Dataset)2 DatasetTable (bio.terra.service.dataset.DatasetTable)2 SnapshotSource (bio.terra.service.snapshot.SnapshotSource)2 AssetNotFoundException (bio.terra.service.snapshot.exception.AssetNotFoundException)2 InvalidSnapshotException (bio.terra.service.snapshot.exception.InvalidSnapshotException)2 Column (bio.terra.common.Column)1 MetadataEnumeration (bio.terra.common.MetadataEnumeration)1 Relationship (bio.terra.common.Relationship)1 PdaoException (bio.terra.common.exception.PdaoException)1 ColumnModel (bio.terra.model.ColumnModel)1