Search in sources :

Example 11 with Storage

use of com.google.api.services.storage.Storage in project beam by apache.

the class GcsUtilTest method testBucketDoesNotExistBecauseOfAccessError.

@Test
public void testBucketDoesNotExistBecauseOfAccessError() throws IOException {
    GcsOptions pipelineOptions = gcsOptionsWithTestCredential();
    GcsUtil gcsUtil = pipelineOptions.getGcsUtil();
    Storage mockStorage = Mockito.mock(Storage.class);
    gcsUtil.setStorageClient(mockStorage);
    Storage.Buckets mockStorageObjects = Mockito.mock(Storage.Buckets.class);
    Storage.Buckets.Get mockStorageGet = Mockito.mock(Storage.Buckets.Get.class);
    BackOff mockBackOff = BackOffAdapter.toGcpBackOff(FluentBackoff.DEFAULT.backoff());
    GoogleJsonResponseException expectedException = googleJsonResponseException(HttpStatusCodes.STATUS_CODE_FORBIDDEN, "Waves hand mysteriously", "These aren't the buckets you're looking for");
    when(mockStorage.buckets()).thenReturn(mockStorageObjects);
    when(mockStorageObjects.get("testbucket")).thenReturn(mockStorageGet);
    when(mockStorageGet.execute()).thenThrow(expectedException);
    assertFalse(gcsUtil.bucketAccessible(GcsPath.fromComponents("testbucket", "testobject"), mockBackOff, new FastNanoClockAndSleeper()));
}
Also used : GoogleJsonResponseException(com.google.api.client.googleapis.json.GoogleJsonResponseException) Storage(com.google.api.services.storage.Storage) GcsOptions(org.apache.beam.sdk.extensions.gcp.options.GcsOptions) BackOff(com.google.api.client.util.BackOff) Test(org.junit.Test)

Example 12 with Storage

use of com.google.api.services.storage.Storage in project beam by apache.

the class GcsUtilTest method testGetBucket.

@Test
public void testGetBucket() throws IOException {
    GcsOptions pipelineOptions = gcsOptionsWithTestCredential();
    GcsUtil gcsUtil = pipelineOptions.getGcsUtil();
    Storage mockStorage = Mockito.mock(Storage.class);
    gcsUtil.setStorageClient(mockStorage);
    Storage.Buckets mockStorageObjects = Mockito.mock(Storage.Buckets.class);
    Storage.Buckets.Get mockStorageGet = Mockito.mock(Storage.Buckets.Get.class);
    BackOff mockBackOff = BackOffAdapter.toGcpBackOff(FluentBackoff.DEFAULT.backoff());
    when(mockStorage.buckets()).thenReturn(mockStorageObjects);
    when(mockStorageObjects.get("testbucket")).thenReturn(mockStorageGet);
    when(mockStorageGet.execute()).thenThrow(new SocketTimeoutException("SocketException")).thenReturn(new Bucket());
    assertNotNull(gcsUtil.getBucket(GcsPath.fromComponents("testbucket", "testobject"), mockBackOff, new FastNanoClockAndSleeper()));
}
Also used : Storage(com.google.api.services.storage.Storage) SocketTimeoutException(java.net.SocketTimeoutException) Bucket(com.google.api.services.storage.model.Bucket) GcsOptions(org.apache.beam.sdk.extensions.gcp.options.GcsOptions) BackOff(com.google.api.client.util.BackOff) Test(org.junit.Test)

Example 13 with Storage

use of com.google.api.services.storage.Storage in project beam by apache.

the class GcsUtilTest method testFileSizeWhenFileNotFoundNonBatch.

@Test
public void testFileSizeWhenFileNotFoundNonBatch() throws Exception {
    MockLowLevelHttpResponse notFoundResponse = new MockLowLevelHttpResponse();
    notFoundResponse.setContent("");
    notFoundResponse.setStatusCode(HttpStatusCodes.STATUS_CODE_NOT_FOUND);
    MockHttpTransport mockTransport = new MockHttpTransport.Builder().setLowLevelHttpResponse(notFoundResponse).build();
    GcsOptions pipelineOptions = gcsOptionsWithTestCredential();
    GcsUtil gcsUtil = pipelineOptions.getGcsUtil();
    gcsUtil.setStorageClient(new Storage(mockTransport, Transport.getJsonFactory(), null));
    thrown.expect(FileNotFoundException.class);
    gcsUtil.fileSize(GcsPath.fromComponents("testbucket", "testobject"));
}
Also used : MockLowLevelHttpResponse(com.google.api.client.testing.http.MockLowLevelHttpResponse) MockHttpTransport(com.google.api.client.testing.http.MockHttpTransport) Storage(com.google.api.services.storage.Storage) GcsOptions(org.apache.beam.sdk.extensions.gcp.options.GcsOptions) Test(org.junit.Test)

Example 14 with Storage

use of com.google.api.services.storage.Storage in project beam by apache.

the class GcsUtilTest method testGetBucketNotExists.

@Test
public void testGetBucketNotExists() throws IOException {
    GcsOptions pipelineOptions = gcsOptionsWithTestCredential();
    GcsUtil gcsUtil = pipelineOptions.getGcsUtil();
    Storage mockStorage = Mockito.mock(Storage.class);
    gcsUtil.setStorageClient(mockStorage);
    Storage.Buckets mockStorageObjects = Mockito.mock(Storage.Buckets.class);
    Storage.Buckets.Get mockStorageGet = Mockito.mock(Storage.Buckets.Get.class);
    BackOff mockBackOff = BackOffAdapter.toGcpBackOff(FluentBackoff.DEFAULT.backoff());
    when(mockStorage.buckets()).thenReturn(mockStorageObjects);
    when(mockStorageObjects.get("testbucket")).thenReturn(mockStorageGet);
    when(mockStorageGet.execute()).thenThrow(googleJsonResponseException(HttpStatusCodes.STATUS_CODE_NOT_FOUND, "It don't exist", "Nothing here to see"));
    thrown.expect(FileNotFoundException.class);
    thrown.expectMessage("It don't exist");
    gcsUtil.getBucket(GcsPath.fromComponents("testbucket", "testobject"), mockBackOff, new FastNanoClockAndSleeper());
}
Also used : Storage(com.google.api.services.storage.Storage) GcsOptions(org.apache.beam.sdk.extensions.gcp.options.GcsOptions) BackOff(com.google.api.client.util.BackOff) Test(org.junit.Test)

Example 15 with Storage

use of com.google.api.services.storage.Storage in project beam by apache.

the class GcsUtilTest method testCreateBucket.

@Test
public void testCreateBucket() throws IOException {
    GcsOptions pipelineOptions = gcsOptionsWithTestCredential();
    GcsUtil gcsUtil = pipelineOptions.getGcsUtil();
    Storage.Buckets mockStorageObjects = Mockito.mock(Storage.Buckets.class);
    Storage mockStorage = Mockito.mock(Storage.class);
    gcsUtil.setStorageClient(mockStorage);
    Storage.Buckets.Insert mockStorageInsert = Mockito.mock(Storage.Buckets.Insert.class);
    BackOff mockBackOff = BackOffAdapter.toGcpBackOff(FluentBackoff.DEFAULT.backoff());
    when(mockStorage.buckets()).thenReturn(mockStorageObjects);
    when(mockStorageObjects.insert(any(String.class), any(Bucket.class))).thenReturn(mockStorageInsert);
    when(mockStorageInsert.execute()).thenThrow(new SocketTimeoutException("SocketException")).thenReturn(new Bucket());
    gcsUtil.createBucket("a", new Bucket(), mockBackOff, new FastNanoClockAndSleeper());
}
Also used : Storage(com.google.api.services.storage.Storage) SocketTimeoutException(java.net.SocketTimeoutException) Bucket(com.google.api.services.storage.model.Bucket) GcsOptions(org.apache.beam.sdk.extensions.gcp.options.GcsOptions) BackOff(com.google.api.client.util.BackOff) Test(org.junit.Test)

Aggregations

Storage (com.google.api.services.storage.Storage)20 Test (org.junit.Test)17 GcsOptions (org.apache.beam.sdk.extensions.gcp.options.GcsOptions)14 BackOff (com.google.api.client.util.BackOff)8 Objects (com.google.api.services.storage.model.Objects)6 GoogleJsonResponseException (com.google.api.client.googleapis.json.GoogleJsonResponseException)5 MockHttpTransport (com.google.api.client.testing.http.MockHttpTransport)5 Bucket (com.google.api.services.storage.model.Bucket)5 SocketTimeoutException (java.net.SocketTimeoutException)5 StorageObject (com.google.api.services.storage.model.StorageObject)4 GcsPath (org.apache.beam.sdk.util.gcsfs.GcsPath)4 JsonFactory (com.google.api.client.json.JsonFactory)3 MockLowLevelHttpResponse (com.google.api.client.testing.http.MockLowLevelHttpResponse)3 GoogleCredential (com.google.api.client.googleapis.auth.oauth2.GoogleCredential)2 GoogleNetHttpTransport (com.google.api.client.googleapis.javanet.GoogleNetHttpTransport)2 HttpTransport (com.google.api.client.http.HttpTransport)2 GenericJson (com.google.api.client.json.GenericJson)2 JacksonFactory (com.google.api.client.json.jackson2.JacksonFactory)2 MockLowLevelHttpRequest (com.google.api.client.testing.http.MockLowLevelHttpRequest)2 ImmutableList (com.google.common.collect.ImmutableList)2