Search in sources :

Example 6 with WriteObjectResponse

use of com.google.storage.v2.WriteObjectResponse in project hadoop-connectors by GoogleCloudDataproc.

the class WatchdogTest method watchOnClientStreamingRPCWithoutTimeout.

@Test
public void watchOnClientStreamingRPCWithoutTimeout() {
    NoopClientCallStub<WriteObjectRequest, WriteObjectResponse> clientCall = new NoopClientCallStub<>();
    StreamObserver<WriteObjectRequest> timeoutStreamObserver = new StreamObserver<>() {

        @Override
        public void onNext(WriteObjectRequest value) {
            logger.atInfo().log("Sleeping for 10 seconds");
            sleepUninterruptibly(Duration.ofSeconds(10));
        }

        @Override
        public void onError(Throwable t) {
        }

        @Override
        public void onCompleted() {
        }
    };
    StreamObserver<WriteObjectRequest> watch = watchdog.watch(clientCall, timeoutStreamObserver, zeroWaitTime);
    WriteObjectRequest value = WriteObjectRequest.newBuilder().build();
    watch.onNext(value);
    assertThat(clientCall.cancelled).isFalse();
    assertThat(clientCall.cause).isNull();
}
Also used : StreamObserver(io.grpc.stub.StreamObserver) WriteObjectRequest(com.google.storage.v2.WriteObjectRequest) WriteObjectResponse(com.google.storage.v2.WriteObjectResponse) Test(org.junit.Test)

Example 7 with WriteObjectResponse

use of com.google.storage.v2.WriteObjectResponse in project hadoop-connectors by GoogleCloudDataproc.

the class GoogleCloudStorageGrpcWriteChannel method handleResponse.

@Override
public void handleResponse(WriteObjectResponse response) {
    Object resource = response.getResource();
    Map<String, byte[]> metadata = resource.getMetadataMap().entrySet().stream().collect(toMap(Map.Entry::getKey, entry -> BaseEncoding.base64().decode(entry.getValue())));
    byte[] md5Hash = null;
    byte[] crc32c = null;
    if (resource.hasChecksums()) {
        md5Hash = !resource.getChecksums().getMd5Hash().isEmpty() ? resource.getChecksums().getMd5Hash().toByteArray() : null;
        crc32c = resource.getChecksums().hasCrc32C() ? ByteBuffer.allocate(4).putInt(resource.getChecksums().getCrc32C()).array() : null;
    }
    completedItemInfo = GoogleCloudStorageItemInfo.createObject(resourceId, Timestamps.toMillis(resource.getCreateTime()), Timestamps.toMillis(resource.getUpdateTime()), resource.getSize(), resource.getContentType(), resource.getContentEncoding(), metadata, resource.getGeneration(), resource.getMetageneration(), new VerificationAttributes(md5Hash, crc32c));
}
Also used : Object(com.google.storage.v2.Object) BufferedInputStream(java.io.BufferedInputStream) CommonRequestParams(com.google.storage.v2.CommonRequestParams) Hashing(com.google.common.hash.Hashing) ObjectChecksums(com.google.storage.v2.ObjectChecksums) Callable(java.util.concurrent.Callable) QueryWriteStatusResponse(com.google.storage.v2.QueryWriteStatusResponse) ByteBuffer(java.nio.ByteBuffer) GoogleCloudStorageImpl.encodeMetadata(com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.encodeMetadata) StorageStub(com.google.storage.v2.StorageGrpc.StorageStub) StreamObserver(io.grpc.stub.StreamObserver) Collectors.toMap(java.util.stream.Collectors.toMap) ClientResponseObserver(io.grpc.stub.ClientResponseObserver) ResilientOperation(com.google.cloud.hadoop.util.ResilientOperation) ClientCalls(io.grpc.stub.ClientCalls) Duration(java.time.Duration) Map(java.util.Map) WriteObjectRequest(com.google.storage.v2.WriteObjectRequest) Status(io.grpc.Status) Hasher(com.google.common.hash.Hasher) QueryWriteStatusRequest(com.google.storage.v2.QueryWriteStatusRequest) ExecutorService(java.util.concurrent.ExecutorService) BackOffFactory(com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.BackOffFactory) ImmutableSet(com.google.common.collect.ImmutableSet) BaseEncoding(com.google.common.io.BaseEncoding) Preconditions.checkNotNull(com.google.common.base.Preconditions.checkNotNull) Timestamps(com.google.protobuf.util.Timestamps) IOException(java.io.IOException) MILLISECONDS(java.util.concurrent.TimeUnit.MILLISECONDS) ClientCallStreamObserver(io.grpc.stub.ClientCallStreamObserver) GoogleLogger(com.google.common.flogger.GoogleLogger) ClientCall(io.grpc.ClientCall) StartResumableWriteResponse(com.google.storage.v2.StartResumableWriteResponse) AbstractGoogleAsyncWriteChannel(com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel) RetryDeterminer(com.google.cloud.hadoop.util.RetryDeterminer) ByteString(com.google.protobuf.ByteString) WriteObjectResponse(com.google.storage.v2.WriteObjectResponse) MAX_WRITE_CHUNK_BYTES(com.google.storage.v2.ServiceConstants.Values.MAX_WRITE_CHUNK_BYTES) CountDownLatch(java.util.concurrent.CountDownLatch) WriteObjectSpec(com.google.storage.v2.WriteObjectSpec) TreeMap(java.util.TreeMap) StorageGrpc(com.google.storage.v2.StorageGrpc) ChecksummedData(com.google.storage.v2.ChecksummedData) ByteStreams(com.google.common.io.ByteStreams) StartResumableWriteRequest(com.google.storage.v2.StartResumableWriteRequest) AsyncWriteChannelOptions(com.google.cloud.hadoop.util.AsyncWriteChannelOptions) InputStream(java.io.InputStream) Object(com.google.storage.v2.Object) ByteString(com.google.protobuf.ByteString) Collectors.toMap(java.util.stream.Collectors.toMap) Map(java.util.Map) TreeMap(java.util.TreeMap)

Aggregations

WriteObjectRequest (com.google.storage.v2.WriteObjectRequest)7 WriteObjectResponse (com.google.storage.v2.WriteObjectResponse)7 StreamObserver (io.grpc.stub.StreamObserver)5 Test (org.junit.Test)4 ByteString (com.google.protobuf.ByteString)2 StorageGrpc (com.google.storage.v2.StorageGrpc)2 ApiStreamObserver (com.google.api.gax.rpc.ApiStreamObserver)1 BackOffFactory (com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.BackOffFactory)1 GoogleCloudStorageImpl.encodeMetadata (com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.encodeMetadata)1 AbstractGoogleAsyncWriteChannel (com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel)1 AsyncWriteChannelOptions (com.google.cloud.hadoop.util.AsyncWriteChannelOptions)1 ResilientOperation (com.google.cloud.hadoop.util.ResilientOperation)1 RetryDeterminer (com.google.cloud.hadoop.util.RetryDeterminer)1 Preconditions.checkNotNull (com.google.common.base.Preconditions.checkNotNull)1 ImmutableSet (com.google.common.collect.ImmutableSet)1 GoogleLogger (com.google.common.flogger.GoogleLogger)1 Hasher (com.google.common.hash.Hasher)1 Hashing (com.google.common.hash.Hashing)1 BaseEncoding (com.google.common.io.BaseEncoding)1 ByteStreams (com.google.common.io.ByteStreams)1