Search in sources :

Example 96 with Digest

use of com.google.cloud.kms.v1.Digest in project bazel-buildfarm by bazelbuild.

the class ReportResultStageTest method operationErrorOnStatusException.

@Test
public void operationErrorOnStatusException() throws Exception {
    WorkerContext context = mock(WorkerContext.class);
    SingleOutputSink output = new SingleOutputSink();
    Operation erroringOperation = Operation.newBuilder().setName("erroring").setMetadata(Any.pack(ExecutingOperationMetadata.getDefaultInstance())).build();
    Action action = Action.getDefaultInstance();
    Digest actionDigest = DIGEST_UTIL.compute(action);
    QueueEntry erroringEntry = QueueEntry.newBuilder().setExecuteEntry(ExecuteEntry.newBuilder().setOperationName(erroringOperation.getName()).setActionDigest(actionDigest)).build();
    OperationContext erroringContext = OperationContext.newBuilder().setCommand(Command.getDefaultInstance()).setAction(action).setOperation(erroringOperation).setQueueEntry(erroringEntry).setExecDir(Paths.get("erroring-operation-path")).setPoller(mock(Poller.class)).build();
    when(context.putOperation(any(Operation.class))).thenReturn(true);
    Status erroredStatus = Status.newBuilder().setCode(Code.FAILED_PRECONDITION.getNumber()).build();
    doThrow(StatusProto.toStatusException(erroredStatus)).when(context).uploadOutputs(eq(actionDigest), any(ActionResult.Builder.class), eq(erroringContext.execDir), eq(ImmutableList.of()), eq(ImmutableList.of()));
    PipelineStage reportResultStage = new ReportResultStage(context, output, /* error=*/
    null);
    reportResultStage.put(erroringContext);
    reportResultStage.run();
    verify(context, times(1)).destroyExecDir(erroringContext.execDir);
    ArgumentCaptor<Operation> operationCaptor = ArgumentCaptor.forClass(Operation.class);
    verify(context, times(1)).putOperation(operationCaptor.capture());
    Operation erroredOperation = operationCaptor.getValue();
    assertThat(output.get().operation).isEqualTo(erroredOperation);
    verify(context, times(1)).uploadOutputs(eq(DIGEST_UTIL.compute(erroringContext.action)), any(ActionResult.Builder.class), eq(erroringContext.execDir), eq(ImmutableList.of()), eq(ImmutableList.of()));
    assertThat(erroredOperation.getResponse().unpack(ExecuteResponse.class).getStatus()).isEqualTo(erroredStatus);
    verify(context, never()).putActionResult(any(ActionKey.class), any(ActionResult.class));
}
Also used : Status(com.google.rpc.Status) Action(build.bazel.remote.execution.v2.Action) Digest(build.bazel.remote.execution.v2.Digest) ExecuteResponse(build.bazel.remote.execution.v2.ExecuteResponse) ActionKey(build.buildfarm.common.DigestUtil.ActionKey) Operation(com.google.longrunning.Operation) QueueEntry(build.buildfarm.v1test.QueueEntry) ActionResult(build.bazel.remote.execution.v2.ActionResult) Test(org.junit.Test)

Example 97 with Digest

use of com.google.cloud.kms.v1.Digest in project bazel-buildfarm by bazelbuild.

the class FailoverInputStreamFactoryTest method DigestInPrimaryIsNotDelegated.

@Test
public void DigestInPrimaryIsNotDelegated() throws IOException, InterruptedException {
    ByteString content = ByteString.copyFromUtf8("Hello, World");
    Digest contentDigest = DIGEST_UTIL.compute(content);
    FailoverInputStreamFactory failoverFactory = new FailoverInputStreamFactory(/* primary=*/
    (digest, offset) -> {
        if (digest.equals(contentDigest)) {
            return content.newInput();
        }
        throw new NoSuchFileException(DigestUtil.toString(digest));
    }, /* failover=*/
    (digest, offset) -> {
        throw new IOException("invalid");
    });
    InputStream in = failoverFactory.newInput(contentDigest, /* offset=*/
    0);
    assertThat(ByteString.readFrom(in)).isEqualTo(content);
}
Also used : Digest(build.bazel.remote.execution.v2.Digest) ByteString(com.google.protobuf.ByteString) InputStream(java.io.InputStream) NoSuchFileException(java.nio.file.NoSuchFileException) IOException(java.io.IOException) Test(org.junit.Test)

Example 98 with Digest

use of com.google.cloud.kms.v1.Digest in project bazel-buildfarm by bazelbuild.

the class ContentAddressableStorageService method putAllBlobs.

private static Iterable<ListenableFuture<Response>> putAllBlobs(Instance instance, Iterable<Request> requests, long writeDeadlineAfter, TimeUnit writeDeadlineAfterUnits) {
    ImmutableList.Builder<ListenableFuture<Response>> responses = new ImmutableList.Builder<>();
    for (Request request : requests) {
        Digest digest = request.getDigest();
        ListenableFuture<Digest> future = putBlobFuture(instance, digest, request.getData(), writeDeadlineAfter, writeDeadlineAfterUnits, TracingMetadataUtils.fromCurrentContext());
        responses.add(toResponseFuture(catching(transform(future, (d) -> Code.OK, directExecutor()), Throwable.class, (e) -> Status.fromThrowable(e).getCode(), directExecutor()), digest));
    }
    return responses.build();
}
Also used : GetTreeRequest(build.bazel.remote.execution.v2.GetTreeRequest) ListenableFuture(com.google.common.util.concurrent.ListenableFuture) FindMissingBlobsRequest(build.bazel.remote.execution.v2.FindMissingBlobsRequest) Stopwatch(com.google.common.base.Stopwatch) DigestUtil(build.buildfarm.common.DigestUtil) TracingMetadataUtils(build.buildfarm.common.grpc.TracingMetadataUtils) Level(java.util.logging.Level) BatchReadBlobsResponse(build.bazel.remote.execution.v2.BatchReadBlobsResponse) Code(io.grpc.Status.Code) StreamObserver(io.grpc.stub.StreamObserver) Futures.addCallback(com.google.common.util.concurrent.Futures.addCallback) ContentAddressableStorageGrpc(build.bazel.remote.execution.v2.ContentAddressableStorageGrpc) ImmutableList(com.google.common.collect.ImmutableList) Digest(build.bazel.remote.execution.v2.Digest) Tree(build.buildfarm.v1test.Tree) Histogram(io.prometheus.client.Histogram) StreamSupport(java.util.stream.StreamSupport) FindMissingBlobsResponse(build.bazel.remote.execution.v2.FindMissingBlobsResponse) Status(io.grpc.Status) GetTreeResponse(build.bazel.remote.execution.v2.GetTreeResponse) Futures.allAsList(com.google.common.util.concurrent.Futures.allAsList) BatchUpdateBlobsRequest(build.bazel.remote.execution.v2.BatchUpdateBlobsRequest) Futures.catching(com.google.common.util.concurrent.Futures.catching) Logger(java.util.logging.Logger) BatchUpdateBlobsResponse(build.bazel.remote.execution.v2.BatchUpdateBlobsResponse) Collectors(java.util.stream.Collectors) String.format(java.lang.String.format) Instance(build.buildfarm.instance.Instance) FutureCallback(com.google.common.util.concurrent.FutureCallback) MoreExecutors.directExecutor(com.google.common.util.concurrent.MoreExecutors.directExecutor) BatchReadBlobsRequest(build.bazel.remote.execution.v2.BatchReadBlobsRequest) TimeUnit(java.util.concurrent.TimeUnit) Response(build.bazel.remote.execution.v2.BatchUpdateBlobsResponse.Response) Request(build.bazel.remote.execution.v2.BatchUpdateBlobsRequest.Request) Futures.transform(com.google.common.util.concurrent.Futures.transform) MICROSECONDS(java.util.concurrent.TimeUnit.MICROSECONDS) Utils.putBlobFuture(build.buildfarm.instance.Utils.putBlobFuture) Digest(build.bazel.remote.execution.v2.Digest) ImmutableList(com.google.common.collect.ImmutableList) GetTreeRequest(build.bazel.remote.execution.v2.GetTreeRequest) FindMissingBlobsRequest(build.bazel.remote.execution.v2.FindMissingBlobsRequest) BatchUpdateBlobsRequest(build.bazel.remote.execution.v2.BatchUpdateBlobsRequest) BatchReadBlobsRequest(build.bazel.remote.execution.v2.BatchReadBlobsRequest) Request(build.bazel.remote.execution.v2.BatchUpdateBlobsRequest.Request) ListenableFuture(com.google.common.util.concurrent.ListenableFuture)

Example 99 with Digest

use of com.google.cloud.kms.v1.Digest in project bazel-buildfarm by bazelbuild.

the class WriteStreamObserver method getWrite.

private Write getWrite(String resourceName) throws EntryLimitException, InvalidResourceNameException {
    switch(detectResourceOperation(resourceName)) {
        case UploadBlob:
            Digest uploadBlobDigest = parseUploadBlobDigest(resourceName);
            expectedCommittedSize = uploadBlobDigest.getSizeBytes();
            return ByteStreamService.getUploadBlobWrite(instance, uploadBlobDigest, parseUploadBlobUUID(resourceName));
        case OperationStream:
            return ByteStreamService.getOperationStreamWrite(instance, resourceName);
        case Blob:
        default:
            throw INVALID_ARGUMENT.withDescription("unknown resource operation for " + resourceName).asRuntimeException();
    }
}
Also used : UrlPath.parseUploadBlobDigest(build.buildfarm.common.UrlPath.parseUploadBlobDigest) Digest(build.bazel.remote.execution.v2.Digest)

Example 100 with Digest

use of com.google.cloud.kms.v1.Digest in project bazel-buildfarm by bazelbuild.

the class Cat method reTreeToTree.

private static Tree reTreeToTree(DigestUtil digestUtil, build.bazel.remote.execution.v2.Tree reTree) {
    Digest rootDigest = digestUtil.compute(reTree.getRoot());
    Tree.Builder tree = Tree.newBuilder().setRootDigest(rootDigest);
    tree.putDirectories(rootDigest.getHash(), reTree.getRoot());
    for (Directory directory : reTree.getChildrenList()) {
        tree.putDirectories(digestUtil.compute(directory).getHash(), directory);
    }
    return tree.build();
}
Also used : Digest(build.bazel.remote.execution.v2.Digest) Tree(build.buildfarm.v1test.Tree) OutputDirectory(build.bazel.remote.execution.v2.OutputDirectory) Directory(build.bazel.remote.execution.v2.Directory)

Aggregations

Digest (build.bazel.remote.execution.v2.Digest)191 ByteString (com.google.protobuf.ByteString)110 Test (org.junit.Test)99 IOException (java.io.IOException)55 Directory (build.bazel.remote.execution.v2.Directory)42 ImmutableList (com.google.common.collect.ImmutableList)35 Path (java.nio.file.Path)33 Status (io.grpc.Status)30 ExecutionException (java.util.concurrent.ExecutionException)29 RequestMetadata (build.bazel.remote.execution.v2.RequestMetadata)27 InputStream (java.io.InputStream)25 Instance (build.buildfarm.instance.Instance)24 Action (build.bazel.remote.execution.v2.Action)22 DigestUtil (build.buildfarm.common.DigestUtil)22 OutputStream (java.io.OutputStream)22 Write (build.buildfarm.common.Write)21 ListenableFuture (com.google.common.util.concurrent.ListenableFuture)21 Operation (com.google.longrunning.Operation)21 UUID (java.util.UUID)20 ExecuteResponse (build.bazel.remote.execution.v2.ExecuteResponse)19