Search in sources :

Example 6 with PipelineStatus

use of com.hartwig.pipeline.execution.PipelineStatus in project pipeline5 by hartwigmedical.

the class QuotaConstrainedComputeEngine method submit.

@Override
public PipelineStatus submit(final RuntimeBucket bucket, final VirtualMachineJobDefinition jobDefinition, final String discriminator) {
    VirtualMachineJobDefinition constrained = jobDefinition;
    MachineType machineType = jobDefinition.performanceProfile().machineType();
    try {
        QuotaBucket regionalQuota = serviceUsage.services().consumerQuotaMetrics().limits().get(cpuQuotaName(project)).execute().getQuotaBuckets().stream().filter(b -> b.getDimensions() != null).filter(b -> region(b) != null).filter(b -> region(b).equals(region)).findFirst().orElseThrow();
        int maxCPU = (int) (regionalQuota.getEffectiveLimit().intValue() * constrainByPercentage);
        if (machineType.cpus() > maxCPU) {
            double reductionRatio = (double) maxCPU / machineType.cpus();
            constrained = VirtualMachineJobDefinition.builder().from(constrained).performanceProfile(VirtualMachinePerformanceProfile.custom(maxCPU, (int) (machineType.memoryGB() * reductionRatio))).build();
        }
        return decorated.submit(bucket, constrained, discriminator);
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
}
Also used : ServiceUsage(com.google.api.services.serviceusage.v1beta1.ServiceUsage) MachineType(com.hartwig.pipeline.execution.MachineType) QuotaBucket(com.google.api.services.serviceusage.v1beta1.model.QuotaBucket) RuntimeBucket(com.hartwig.pipeline.storage.RuntimeBucket) IOException(java.io.IOException) PipelineStatus(com.hartwig.pipeline.execution.PipelineStatus) MachineType(com.hartwig.pipeline.execution.MachineType) IOException(java.io.IOException) QuotaBucket(com.google.api.services.serviceusage.v1beta1.model.QuotaBucket)

Aggregations

PipelineStatus (com.hartwig.pipeline.execution.PipelineStatus)6 RuntimeBucket (com.hartwig.pipeline.storage.RuntimeBucket)4 BashStartupScript (com.hartwig.pipeline.execution.vm.BashStartupScript)3 ServiceUsage (com.google.api.services.serviceusage.v1beta1.ServiceUsage)2 QuotaBucket (com.google.api.services.serviceusage.v1beta1.model.QuotaBucket)2 ResultsDirectory (com.hartwig.pipeline.ResultsDirectory)2 MachineType (com.hartwig.pipeline.execution.MachineType)2 BashCommand (com.hartwig.pipeline.execution.vm.BashCommand)2 OutputUpload (com.hartwig.pipeline.execution.vm.OutputUpload)2 VirtualMachineJobDefinition (com.hartwig.pipeline.execution.vm.VirtualMachineJobDefinition)2 AddDatatype (com.hartwig.pipeline.metadata.AddDatatype)2 ArchivePath (com.hartwig.pipeline.metadata.ArchivePath)2 Folder (com.hartwig.pipeline.report.Folder)2 ReportComponent (com.hartwig.pipeline.report.ReportComponent)2 RunLogComponent (com.hartwig.pipeline.report.RunLogComponent)2 SingleFileComponent (com.hartwig.pipeline.report.SingleFileComponent)2 SubStageInputOutput (com.hartwig.pipeline.stages.SubStageInputOutput)2 GoogleStorageLocation (com.hartwig.pipeline.storage.GoogleStorageLocation)2 StageTrace (com.hartwig.pipeline.trace.StageTrace)2 IOException (java.io.IOException)2