Search in sources :

Example 1 with CloningInstructions

use of bio.terra.workspace.service.resource.model.CloningInstructions in project terra-workspace-manager by DataBiosphere.

the class ReferencedGcpResourceController method cloneGcpGcsObjectReference.

@Override
public ResponseEntity<ApiCloneReferencedGcpGcsObjectResourceResult> cloneGcpGcsObjectReference(UUID workspaceId, UUID resourceId, @Valid ApiCloneReferencedResourceRequestBody body) {
    AuthenticatedUserRequest userRequest = getAuthenticatedInfo();
    final ReferencedResource sourceReferencedResource = referenceResourceService.getReferenceResource(workspaceId, resourceId, userRequest);
    final CloningInstructions effectiveCloningInstructions = Optional.ofNullable(body.getCloningInstructions()).map(CloningInstructions::fromApiModel).orElse(sourceReferencedResource.getCloningInstructions());
    if (CloningInstructions.COPY_REFERENCE != effectiveCloningInstructions) {
        // Nothing to clone here
        final var emptyResult = new ApiCloneReferencedGcpGcsObjectResourceResult().effectiveCloningInstructions(effectiveCloningInstructions.toApiModel()).sourceResourceId(sourceReferencedResource.getResourceId()).sourceWorkspaceId(sourceReferencedResource.getWorkspaceId()).resource(null);
        return new ResponseEntity<>(emptyResult, HttpStatus.OK);
    }
    // Clone the reference
    final ReferencedGcsObjectResource clonedReferencedResource = referenceResourceService.cloneReferencedResource(sourceReferencedResource, body.getDestinationWorkspaceId(), body.getName(), body.getDescription(), userRequest).castByEnum(WsmResourceType.REFERENCED_GCP_GCS_OBJECT);
    // Build the correct response type
    final var result = new ApiCloneReferencedGcpGcsObjectResourceResult().resource(clonedReferencedResource.toApiResource()).sourceWorkspaceId(sourceReferencedResource.getWorkspaceId()).sourceResourceId(sourceReferencedResource.getResourceId()).effectiveCloningInstructions(effectiveCloningInstructions.toApiModel());
    return new ResponseEntity<>(result, HttpStatus.OK);
}
Also used : ReferencedResource(bio.terra.workspace.service.resource.referenced.cloud.gcp.ReferencedResource) ResponseEntity(org.springframework.http.ResponseEntity) CloningInstructions(bio.terra.workspace.service.resource.model.CloningInstructions) AuthenticatedUserRequest(bio.terra.workspace.service.iam.AuthenticatedUserRequest) ApiCloneReferencedGcpGcsObjectResourceResult(bio.terra.workspace.generated.model.ApiCloneReferencedGcpGcsObjectResourceResult) ReferencedGcsObjectResource(bio.terra.workspace.service.resource.referenced.cloud.gcp.gcsobject.ReferencedGcsObjectResource)

Example 2 with CloningInstructions

use of bio.terra.workspace.service.resource.model.CloningInstructions in project terra-workspace-manager by DataBiosphere.

the class ReferencedGcpResourceController method cloneGcpGcsBucketReference.

@Override
public ResponseEntity<ApiCloneReferencedGcpGcsBucketResourceResult> cloneGcpGcsBucketReference(UUID workspaceId, UUID resourceId, @Valid ApiCloneReferencedResourceRequestBody body) {
    final AuthenticatedUserRequest petRequest = getCloningCredentials(workspaceId);
    final ReferencedResource sourceReferencedResource = referenceResourceService.getReferenceResource(workspaceId, resourceId, petRequest);
    final CloningInstructions effectiveCloningInstructions = Optional.ofNullable(body.getCloningInstructions()).map(CloningInstructions::fromApiModel).orElse(sourceReferencedResource.getCloningInstructions());
    if (CloningInstructions.COPY_REFERENCE != effectiveCloningInstructions) {
        // Nothing to clone here
        final var emptyResult = new ApiCloneReferencedGcpGcsBucketResourceResult().effectiveCloningInstructions(effectiveCloningInstructions.toApiModel()).sourceResourceId(sourceReferencedResource.getResourceId()).sourceWorkspaceId(sourceReferencedResource.getWorkspaceId()).resource(null);
        return new ResponseEntity<>(emptyResult, HttpStatus.OK);
    }
    // Clone the reference
    final ReferencedGcsBucketResource clonedReferencedResource = referenceResourceService.cloneReferencedResource(sourceReferencedResource, body.getDestinationWorkspaceId(), body.getName(), body.getDescription(), petRequest).castByEnum(WsmResourceType.REFERENCED_GCP_GCS_BUCKET);
    // Build the correct response type
    final var result = new ApiCloneReferencedGcpGcsBucketResourceResult().resource(clonedReferencedResource.toApiResource()).sourceWorkspaceId(sourceReferencedResource.getWorkspaceId()).sourceResourceId(sourceReferencedResource.getResourceId()).effectiveCloningInstructions(effectiveCloningInstructions.toApiModel());
    return new ResponseEntity<>(result, HttpStatus.OK);
}
Also used : ReferencedResource(bio.terra.workspace.service.resource.referenced.cloud.gcp.ReferencedResource) ResponseEntity(org.springframework.http.ResponseEntity) CloningInstructions(bio.terra.workspace.service.resource.model.CloningInstructions) ReferencedGcsBucketResource(bio.terra.workspace.service.resource.referenced.cloud.gcp.gcsbucket.ReferencedGcsBucketResource) ApiCloneReferencedGcpGcsBucketResourceResult(bio.terra.workspace.generated.model.ApiCloneReferencedGcpGcsBucketResourceResult) AuthenticatedUserRequest(bio.terra.workspace.service.iam.AuthenticatedUserRequest)

Example 3 with CloningInstructions

use of bio.terra.workspace.service.resource.model.CloningInstructions in project terra-workspace-manager by DataBiosphere.

the class ReferencedGcpResourceController method cloneGcpDataRepoSnapshotReference.

@Override
public ResponseEntity<ApiCloneReferencedGcpDataRepoSnapshotResourceResult> cloneGcpDataRepoSnapshotReference(UUID workspaceId, UUID resourceId, @Valid ApiCloneReferencedResourceRequestBody body) {
    final AuthenticatedUserRequest userRequest = getAuthenticatedInfo();
    final ReferencedResource sourceReferencedResource = referenceResourceService.getReferenceResource(workspaceId, resourceId, userRequest);
    final CloningInstructions effectiveCloningInstructions = Optional.ofNullable(body.getCloningInstructions()).map(CloningInstructions::fromApiModel).orElse(sourceReferencedResource.getCloningInstructions());
    if (CloningInstructions.COPY_REFERENCE != effectiveCloningInstructions) {
        // Nothing to clone here
        final var emptyResult = new ApiCloneReferencedGcpDataRepoSnapshotResourceResult().effectiveCloningInstructions(effectiveCloningInstructions.toApiModel()).sourceResourceId(sourceReferencedResource.getResourceId()).sourceWorkspaceId(sourceReferencedResource.getWorkspaceId()).resource(null);
        return new ResponseEntity<>(emptyResult, HttpStatus.OK);
    }
    // Clone the reference
    final ReferencedDataRepoSnapshotResource clonedReferencedResource = referenceResourceService.cloneReferencedResource(sourceReferencedResource, body.getDestinationWorkspaceId(), body.getName(), body.getDescription(), userRequest).castByEnum(WsmResourceType.REFERENCED_ANY_DATA_REPO_SNAPSHOT);
    // Build the correct response type
    final var result = new ApiCloneReferencedGcpDataRepoSnapshotResourceResult().resource(clonedReferencedResource.toApiResource()).sourceWorkspaceId(sourceReferencedResource.getWorkspaceId()).sourceResourceId(sourceReferencedResource.getResourceId()).effectiveCloningInstructions(effectiveCloningInstructions.toApiModel());
    return new ResponseEntity<>(result, HttpStatus.OK);
}
Also used : ReferencedResource(bio.terra.workspace.service.resource.referenced.cloud.gcp.ReferencedResource) ResponseEntity(org.springframework.http.ResponseEntity) CloningInstructions(bio.terra.workspace.service.resource.model.CloningInstructions) ApiCloneReferencedGcpDataRepoSnapshotResourceResult(bio.terra.workspace.generated.model.ApiCloneReferencedGcpDataRepoSnapshotResourceResult) AuthenticatedUserRequest(bio.terra.workspace.service.iam.AuthenticatedUserRequest) ReferencedDataRepoSnapshotResource(bio.terra.workspace.service.resource.referenced.cloud.gcp.datareposnapshot.ReferencedDataRepoSnapshotResource)

Example 4 with CloningInstructions

use of bio.terra.workspace.service.resource.model.CloningInstructions in project terra-workspace-manager by DataBiosphere.

the class ReferencedGcpResourceController method cloneGcpBigQueryDatasetReference.

@Override
public ResponseEntity<ApiCloneReferencedGcpBigQueryDatasetResourceResult> cloneGcpBigQueryDatasetReference(UUID workspaceId, UUID resourceId, @Valid ApiCloneReferencedResourceRequestBody body) {
    final AuthenticatedUserRequest petRequest = getCloningCredentials(workspaceId);
    final ReferencedResource sourceReferencedResource = referenceResourceService.getReferenceResource(workspaceId, resourceId, petRequest);
    final CloningInstructions effectiveCloningInstructions = Optional.ofNullable(body.getCloningInstructions()).map(CloningInstructions::fromApiModel).orElse(sourceReferencedResource.getCloningInstructions());
    if (CloningInstructions.COPY_REFERENCE != effectiveCloningInstructions) {
        // Nothing to clone here
        final var emptyResult = new ApiCloneReferencedGcpBigQueryDatasetResourceResult().effectiveCloningInstructions(effectiveCloningInstructions.toApiModel()).sourceResourceId(sourceReferencedResource.getResourceId()).sourceWorkspaceId(sourceReferencedResource.getWorkspaceId()).resource(null);
        return new ResponseEntity<>(emptyResult, HttpStatus.OK);
    }
    // Clone the reference
    final ReferencedBigQueryDatasetResource clonedReferencedResource = referenceResourceService.cloneReferencedResource(sourceReferencedResource, body.getDestinationWorkspaceId(), body.getName(), body.getDescription(), petRequest).castByEnum(WsmResourceType.REFERENCED_GCP_BIG_QUERY_DATASET);
    // Build the correct response type
    final var result = new ApiCloneReferencedGcpBigQueryDatasetResourceResult().resource(clonedReferencedResource.toApiResource()).sourceWorkspaceId(sourceReferencedResource.getWorkspaceId()).sourceResourceId(sourceReferencedResource.getResourceId()).effectiveCloningInstructions(effectiveCloningInstructions.toApiModel());
    return new ResponseEntity<>(result, HttpStatus.OK);
}
Also used : ReferencedResource(bio.terra.workspace.service.resource.referenced.cloud.gcp.ReferencedResource) ResponseEntity(org.springframework.http.ResponseEntity) CloningInstructions(bio.terra.workspace.service.resource.model.CloningInstructions) AuthenticatedUserRequest(bio.terra.workspace.service.iam.AuthenticatedUserRequest) ReferencedBigQueryDatasetResource(bio.terra.workspace.service.resource.referenced.cloud.gcp.bqdataset.ReferencedBigQueryDatasetResource) ApiCloneReferencedGcpBigQueryDatasetResourceResult(bio.terra.workspace.generated.model.ApiCloneReferencedGcpBigQueryDatasetResourceResult)

Example 5 with CloningInstructions

use of bio.terra.workspace.service.resource.model.CloningInstructions in project terra-workspace-manager by DataBiosphere.

the class CreateTableCopyJobsStep method doStep.

/**
 * Create one BigQuery copy job for each table in the source dataset. Keep a running map from
 * table ID to job ID as new jobs are created, and only create jobs for tables that aren't in the
 * map already. Rerun the step after every table is processed so that the map may be persisted
 * incrementally.
 *
 * <p>On retry, create the jobs for any tables that don't have them. Use WRITE_TRUNCATE to avoid
 * the possibility of duplicate data.
 */
@Override
public StepResult doStep(FlightContext flightContext) throws InterruptedException, RetryException {
    final FlightMap workingMap = flightContext.getWorkingMap();
    final CloningInstructions effectiveCloningInstructions = flightContext.getInputParameters().get(ControlledResourceKeys.CLONING_INSTRUCTIONS, CloningInstructions.class);
    if (CloningInstructions.COPY_RESOURCE != effectiveCloningInstructions) {
        return StepResult.getStepResultSuccess();
    }
    // Gather inputs
    final DatasetCloneInputs sourceInputs = getSourceInputs();
    workingMap.put(ControlledResourceKeys.SOURCE_CLONE_INPUTS, sourceInputs);
    final DatasetCloneInputs destinationInputs = getDestinationInputs(flightContext);
    workingMap.put(ControlledResourceKeys.DESTINATION_CLONE_INPUTS, destinationInputs);
    final BigQueryCow bigQueryCow = crlService.createWsmSaBigQueryCow();
    // TODO(jaycarlton):  remove usage of this client when it's all in CRL PF-942
    final Bigquery bigQueryClient = crlService.createWsmSaNakedBigQueryClient();
    try {
        // Get a list of all tables in the source dataset
        final TableList sourceTables = bigQueryCow.tables().list(sourceInputs.getProjectId(), sourceInputs.getDatasetName()).execute();
        // Start a copy job for each source table
        final Map<String, String> tableToJobId = Optional.ofNullable(workingMap.get(ControlledResourceKeys.TABLE_TO_JOB_ID_MAP, new TypeReference<Map<String, String>>() {
        })).orElseGet(HashMap::new);
        final List<Tables> tables = Optional.ofNullable(sourceTables.getTables()).orElse(Collections.emptyList());
        // Find the first table whose ID isn't a key in the map.
        final Optional<Tables> tableMaybe = tables.stream().filter(t -> null != t.getId() && !tableToJobId.containsKey(t.getId())).findFirst();
        if (tableMaybe.isPresent()) {
            final Tables table = tableMaybe.get();
            checkStreamingBuffer(sourceInputs, bigQueryCow, table);
            final Job inputJob = buildTableCopyJob(sourceInputs, destinationInputs, table);
            // bill the job to the destination project
            final Job submittedJob = bigQueryClient.jobs().insert(destinationInputs.getProjectId(), inputJob).execute();
            // Update the map, which will be persisted
            tableToJobId.put(table.getId(), submittedJob.getId());
            workingMap.put(ControlledResourceKeys.TABLE_TO_JOB_ID_MAP, tableToJobId);
            return new StepResult(StepStatus.STEP_RESULT_RERUN);
        } else {
            // All tables have entries in the map, so all jobs are started.
            workingMap.put(ControlledResourceKeys.TABLE_TO_JOB_ID_MAP, // in case it's empty
            tableToJobId);
            return StepResult.getStepResultSuccess();
        }
    } catch (IOException e) {
        return new StepResult(StepStatus.STEP_RESULT_FAILURE_RETRY, e);
    }
}
Also used : TableList(com.google.api.services.bigquery.model.TableList) BigQueryCow(bio.terra.cloudres.google.bigquery.BigQueryCow) LoggerFactory(org.slf4j.LoggerFactory) HashMap(java.util.HashMap) Tables(com.google.api.services.bigquery.model.TableList.Tables) StepResult(bio.terra.stairway.StepResult) Step(bio.terra.stairway.Step) RetryException(bio.terra.stairway.exception.RetryException) Duration(java.time.Duration) Map(java.util.Map) TypeReference(com.fasterxml.jackson.core.type.TypeReference) Job(com.google.api.services.bigquery.model.Job) CrlService(bio.terra.workspace.service.crl.CrlService) TableReference(com.google.api.services.bigquery.model.TableReference) ControlledBigQueryDatasetResource(bio.terra.workspace.service.resource.controlled.cloud.gcp.bqdataset.ControlledBigQueryDatasetResource) Logger(org.slf4j.Logger) FlightMap(bio.terra.stairway.FlightMap) IOException(java.io.IOException) UUID(java.util.UUID) Instant(java.time.Instant) JobConfigurationTableCopy(com.google.api.services.bigquery.model.JobConfigurationTableCopy) Table(com.google.api.services.bigquery.model.Table) List(java.util.List) GcpCloudContextService(bio.terra.workspace.service.workspace.GcpCloudContextService) Bigquery(com.google.api.services.bigquery.Bigquery) CloningInstructions(bio.terra.workspace.service.resource.model.CloningInstructions) Optional(java.util.Optional) ControlledResourceKeys(bio.terra.workspace.service.workspace.flight.WorkspaceFlightMapKeys.ControlledResourceKeys) StepStatus(bio.terra.stairway.StepStatus) Collections(java.util.Collections) FlightContext(bio.terra.stairway.FlightContext) JobConfiguration(com.google.api.services.bigquery.model.JobConfiguration) HashMap(java.util.HashMap) Bigquery(com.google.api.services.bigquery.Bigquery) TableList(com.google.api.services.bigquery.model.TableList) IOException(java.io.IOException) BigQueryCow(bio.terra.cloudres.google.bigquery.BigQueryCow) CloningInstructions(bio.terra.workspace.service.resource.model.CloningInstructions) Tables(com.google.api.services.bigquery.model.TableList.Tables) FlightMap(bio.terra.stairway.FlightMap) Job(com.google.api.services.bigquery.model.Job) StepResult(bio.terra.stairway.StepResult) HashMap(java.util.HashMap) Map(java.util.Map) FlightMap(bio.terra.stairway.FlightMap)

Aggregations

CloningInstructions (bio.terra.workspace.service.resource.model.CloningInstructions)16 FlightMap (bio.terra.stairway.FlightMap)7 AuthenticatedUserRequest (bio.terra.workspace.service.iam.AuthenticatedUserRequest)6 ReferencedResource (bio.terra.workspace.service.resource.referenced.cloud.gcp.ReferencedResource)6 ResponseEntity (org.springframework.http.ResponseEntity)6 StepResult (bio.terra.stairway.StepResult)5 IOException (java.io.IOException)5 UUID (java.util.UUID)3 ApiClonedControlledGcpGcsBucket (bio.terra.workspace.generated.model.ApiClonedControlledGcpGcsBucket)2 ControlledResourceIamRole (bio.terra.workspace.service.iam.model.ControlledResourceIamRole)2 JobBuilder (bio.terra.workspace.service.job.JobBuilder)2 ControlledBigQueryDatasetResource (bio.terra.workspace.service.resource.controlled.cloud.gcp.bqdataset.ControlledBigQueryDatasetResource)2 ControlledResource (bio.terra.workspace.service.resource.controlled.model.ControlledResource)2 ControlledResourceFields (bio.terra.workspace.service.resource.controlled.model.ControlledResourceFields)2 Bigquery (com.google.api.services.bigquery.Bigquery)2 Job (com.google.api.services.bigquery.model.Job)2 TableReference (com.google.api.services.bigquery.model.TableReference)2 Storagetransfer (com.google.api.services.storagetransfer.v1.Storagetransfer)2 Map (java.util.Map)2 BigQueryCow (bio.terra.cloudres.google.bigquery.BigQueryCow)1