Search in sources :

Example 6 with Storage

use of com.google.api.services.storage.Storage in project beam by apache.

the class GcsUtilTest method testBucketDoesNotExist.

@Test
public void testBucketDoesNotExist() throws IOException {
    GcsOptions pipelineOptions = gcsOptionsWithTestCredential();
    GcsUtil gcsUtil = pipelineOptions.getGcsUtil();
    Storage mockStorage = Mockito.mock(Storage.class);
    gcsUtil.setStorageClient(mockStorage);
    Storage.Buckets mockStorageObjects = Mockito.mock(Storage.Buckets.class);
    Storage.Buckets.Get mockStorageGet = Mockito.mock(Storage.Buckets.Get.class);
    BackOff mockBackOff = BackOffAdapter.toGcpBackOff(FluentBackoff.DEFAULT.backoff());
    when(mockStorage.buckets()).thenReturn(mockStorageObjects);
    when(mockStorageObjects.get("testbucket")).thenReturn(mockStorageGet);
    when(mockStorageGet.execute()).thenThrow(googleJsonResponseException(HttpStatusCodes.STATUS_CODE_NOT_FOUND, "It don't exist", "Nothing here to see"));
    assertFalse(gcsUtil.bucketAccessible(GcsPath.fromComponents("testbucket", "testobject"), mockBackOff, new FastNanoClockAndSleeper()));
}
Also used : Storage(com.google.api.services.storage.Storage) GcsOptions(org.apache.beam.sdk.extensions.gcp.options.GcsOptions) BackOff(com.google.api.client.util.BackOff) Test(org.junit.Test)

Example 7 with Storage

use of com.google.api.services.storage.Storage in project druid by druid-io.

the class GoogleStorageDruidModule method getGoogleStorage.

@Provides
@LazySingleton
public GoogleStorage getGoogleStorage(final GoogleAccountConfig config) throws IOException, GeneralSecurityException {
    LOG.info("Building Cloud Storage Client...");
    HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
    JsonFactory jsonFactory = JacksonFactory.getDefaultInstance();
    GoogleCredential credential = GoogleCredential.getApplicationDefault(httpTransport, jsonFactory);
    if (credential.createScopedRequired()) {
        credential = credential.createScoped(StorageScopes.all());
    }
    Storage storage = new Storage.Builder(httpTransport, jsonFactory, credential).setApplicationName(APPLICATION_NAME).build();
    return new GoogleStorage(storage);
}
Also used : HttpTransport(com.google.api.client.http.HttpTransport) GoogleNetHttpTransport(com.google.api.client.googleapis.javanet.GoogleNetHttpTransport) Storage(com.google.api.services.storage.Storage) JsonFactory(com.google.api.client.json.JsonFactory) GoogleCredential(com.google.api.client.googleapis.auth.oauth2.GoogleCredential) LazySingleton(io.druid.guice.LazySingleton) Provides(com.google.inject.Provides)

Example 8 with Storage

use of com.google.api.services.storage.Storage in project beam by apache.

the class GcsUtilTest method testRecursiveGlobExpansion.

@Test
public void testRecursiveGlobExpansion() throws IOException {
    GcsOptions pipelineOptions = gcsOptionsWithTestCredential();
    GcsUtil gcsUtil = pipelineOptions.getGcsUtil();
    Storage mockStorage = Mockito.mock(Storage.class);
    gcsUtil.setStorageClient(mockStorage);
    Storage.Objects mockStorageObjects = Mockito.mock(Storage.Objects.class);
    Storage.Objects.Get mockStorageGet = Mockito.mock(Storage.Objects.Get.class);
    Storage.Objects.List mockStorageList = Mockito.mock(Storage.Objects.List.class);
    Objects modelObjects = new Objects();
    List<StorageObject> items = new ArrayList<>();
    // A directory
    items.add(new StorageObject().setBucket("testbucket").setName("testdirectory/"));
    // Files within the directory
    items.add(new StorageObject().setBucket("testbucket").setName("test/directory/file1.txt"));
    items.add(new StorageObject().setBucket("testbucket").setName("test/directory/file2.txt"));
    items.add(new StorageObject().setBucket("testbucket").setName("test/directory/file3.txt"));
    items.add(new StorageObject().setBucket("testbucket").setName("test/directory/otherfile"));
    items.add(new StorageObject().setBucket("testbucket").setName("test/directory/anotherfile"));
    items.add(new StorageObject().setBucket("testbucket").setName("test/file4.txt"));
    modelObjects.setItems(items);
    when(mockStorage.objects()).thenReturn(mockStorageObjects);
    when(mockStorageObjects.get("testbucket", "test/directory/otherfile")).thenReturn(mockStorageGet);
    when(mockStorageObjects.list("testbucket")).thenReturn(mockStorageList);
    when(mockStorageGet.execute()).thenReturn(new StorageObject().setBucket("testbucket").setName("test/directory/otherfile"));
    when(mockStorageList.execute()).thenReturn(modelObjects);
    {
        GcsPath pattern = GcsPath.fromUri("gs://testbucket/test/**/*.txt");
        List<GcsPath> expectedFiles = ImmutableList.of(GcsPath.fromUri("gs://testbucket/test/directory/file1.txt"), GcsPath.fromUri("gs://testbucket/test/directory/file2.txt"), GcsPath.fromUri("gs://testbucket/test/directory/file3.txt"), GcsPath.fromUri("gs://testbucket/test/file4.txt"));
        assertThat(expectedFiles, contains(gcsUtil.expand(pattern).toArray()));
    }
}
Also used : Storage(com.google.api.services.storage.Storage) StorageObject(com.google.api.services.storage.model.StorageObject) Objects(com.google.api.services.storage.model.Objects) ArrayList(java.util.ArrayList) GcsPath(org.apache.beam.sdk.util.gcsfs.GcsPath) List(java.util.List) ArrayList(java.util.ArrayList) ImmutableList(com.google.common.collect.ImmutableList) GcsOptions(org.apache.beam.sdk.extensions.gcp.options.GcsOptions) Test(org.junit.Test)

Example 9 with Storage

use of com.google.api.services.storage.Storage in project beam by apache.

the class GcsUtilTest method testGlobExpansion.

@Test
public void testGlobExpansion() throws IOException {
    GcsOptions pipelineOptions = gcsOptionsWithTestCredential();
    GcsUtil gcsUtil = pipelineOptions.getGcsUtil();
    Storage mockStorage = Mockito.mock(Storage.class);
    gcsUtil.setStorageClient(mockStorage);
    Storage.Objects mockStorageObjects = Mockito.mock(Storage.Objects.class);
    Storage.Objects.Get mockStorageGet = Mockito.mock(Storage.Objects.Get.class);
    Storage.Objects.List mockStorageList = Mockito.mock(Storage.Objects.List.class);
    Objects modelObjects = new Objects();
    List<StorageObject> items = new ArrayList<>();
    // A directory
    items.add(new StorageObject().setBucket("testbucket").setName("testdirectory/"));
    // Files within the directory
    items.add(new StorageObject().setBucket("testbucket").setName("testdirectory/file1name"));
    items.add(new StorageObject().setBucket("testbucket").setName("testdirectory/file2name"));
    items.add(new StorageObject().setBucket("testbucket").setName("testdirectory/file3name"));
    items.add(new StorageObject().setBucket("testbucket").setName("testdirectory/otherfile"));
    items.add(new StorageObject().setBucket("testbucket").setName("testdirectory/anotherfile"));
    modelObjects.setItems(items);
    when(mockStorage.objects()).thenReturn(mockStorageObjects);
    when(mockStorageObjects.get("testbucket", "testdirectory/otherfile")).thenReturn(mockStorageGet);
    when(mockStorageObjects.list("testbucket")).thenReturn(mockStorageList);
    when(mockStorageGet.execute()).thenReturn(new StorageObject().setBucket("testbucket").setName("testdirectory/otherfile"));
    when(mockStorageList.execute()).thenReturn(modelObjects);
    // Test a single file.
    {
        GcsPath pattern = GcsPath.fromUri("gs://testbucket/testdirectory/otherfile");
        List<GcsPath> expectedFiles = ImmutableList.of(GcsPath.fromUri("gs://testbucket/testdirectory/otherfile"));
        assertThat(expectedFiles, contains(gcsUtil.expand(pattern).toArray()));
    }
    // Test patterns.
    {
        GcsPath pattern = GcsPath.fromUri("gs://testbucket/testdirectory/file*");
        List<GcsPath> expectedFiles = ImmutableList.of(GcsPath.fromUri("gs://testbucket/testdirectory/file1name"), GcsPath.fromUri("gs://testbucket/testdirectory/file2name"), GcsPath.fromUri("gs://testbucket/testdirectory/file3name"));
        assertThat(expectedFiles, contains(gcsUtil.expand(pattern).toArray()));
    }
    {
        GcsPath pattern = GcsPath.fromUri("gs://testbucket/testdirectory/file[1-3]*");
        List<GcsPath> expectedFiles = ImmutableList.of(GcsPath.fromUri("gs://testbucket/testdirectory/file1name"), GcsPath.fromUri("gs://testbucket/testdirectory/file2name"), GcsPath.fromUri("gs://testbucket/testdirectory/file3name"));
        assertThat(expectedFiles, contains(gcsUtil.expand(pattern).toArray()));
    }
    {
        GcsPath pattern = GcsPath.fromUri("gs://testbucket/testdirectory/file?name");
        List<GcsPath> expectedFiles = ImmutableList.of(GcsPath.fromUri("gs://testbucket/testdirectory/file1name"), GcsPath.fromUri("gs://testbucket/testdirectory/file2name"), GcsPath.fromUri("gs://testbucket/testdirectory/file3name"));
        assertThat(expectedFiles, contains(gcsUtil.expand(pattern).toArray()));
    }
    {
        GcsPath pattern = GcsPath.fromUri("gs://testbucket/test*ectory/fi*name");
        List<GcsPath> expectedFiles = ImmutableList.of(GcsPath.fromUri("gs://testbucket/testdirectory/file1name"), GcsPath.fromUri("gs://testbucket/testdirectory/file2name"), GcsPath.fromUri("gs://testbucket/testdirectory/file3name"));
        assertThat(expectedFiles, contains(gcsUtil.expand(pattern).toArray()));
    }
}
Also used : Storage(com.google.api.services.storage.Storage) StorageObject(com.google.api.services.storage.model.StorageObject) Objects(com.google.api.services.storage.model.Objects) ArrayList(java.util.ArrayList) GcsPath(org.apache.beam.sdk.util.gcsfs.GcsPath) List(java.util.List) ArrayList(java.util.ArrayList) ImmutableList(com.google.common.collect.ImmutableList) GcsOptions(org.apache.beam.sdk.extensions.gcp.options.GcsOptions) Test(org.junit.Test)

Example 10 with Storage

use of com.google.api.services.storage.Storage in project beam by apache.

the class GcsUtilTest method testGetSizeBytesWhenFileNotFoundBatchRetry.

@Test
public void testGetSizeBytesWhenFileNotFoundBatchRetry() throws Exception {
    JsonFactory jsonFactory = new JacksonFactory();
    String contentBoundary = "batch_foobarbaz";
    String contentBoundaryLine = "--" + contentBoundary;
    String endOfContentBoundaryLine = "--" + contentBoundary + "--";
    GenericJson error = new GenericJson().set("error", new GenericJson().set("code", 404));
    error.setFactory(jsonFactory);
    String content = contentBoundaryLine + "\n" + "Content-Type: application/http\n" + "\n" + "HTTP/1.1 404 Not Found\n" + "Content-Length: -1\n" + "\n" + error.toString() + "\n" + "\n" + endOfContentBoundaryLine + "\n";
    thrown.expect(FileNotFoundException.class);
    final LowLevelHttpResponse mockResponse = Mockito.mock(LowLevelHttpResponse.class);
    when(mockResponse.getContentType()).thenReturn("multipart/mixed; boundary=" + contentBoundary);
    // 429: Too many requests, then 200: OK.
    when(mockResponse.getStatusCode()).thenReturn(429, 200);
    when(mockResponse.getContent()).thenReturn(toStream("error"), toStream(content));
    // A mock transport that lets us mock the API responses.
    MockHttpTransport mockTransport = new MockHttpTransport.Builder().setLowLevelHttpRequest(new MockLowLevelHttpRequest() {

        @Override
        public LowLevelHttpResponse execute() throws IOException {
            return mockResponse;
        }
    }).build();
    GcsUtil gcsUtil = gcsOptionsWithTestCredential().getGcsUtil();
    gcsUtil.setStorageClient(new Storage(mockTransport, Transport.getJsonFactory(), new RetryHttpRequestInitializer()));
    gcsUtil.fileSizes(ImmutableList.of(GcsPath.fromComponents("testbucket", "testobject")));
}
Also used : GenericJson(com.google.api.client.json.GenericJson) MockHttpTransport(com.google.api.client.testing.http.MockHttpTransport) Storage(com.google.api.services.storage.Storage) JsonFactory(com.google.api.client.json.JsonFactory) MockLowLevelHttpResponse(com.google.api.client.testing.http.MockLowLevelHttpResponse) LowLevelHttpResponse(com.google.api.client.http.LowLevelHttpResponse) JacksonFactory(com.google.api.client.json.jackson2.JacksonFactory) MockLowLevelHttpRequest(com.google.api.client.testing.http.MockLowLevelHttpRequest) Test(org.junit.Test)

Aggregations

Storage (com.google.api.services.storage.Storage)20 Test (org.junit.Test)17 GcsOptions (org.apache.beam.sdk.extensions.gcp.options.GcsOptions)14 BackOff (com.google.api.client.util.BackOff)8 Objects (com.google.api.services.storage.model.Objects)6 GoogleJsonResponseException (com.google.api.client.googleapis.json.GoogleJsonResponseException)5 MockHttpTransport (com.google.api.client.testing.http.MockHttpTransport)5 Bucket (com.google.api.services.storage.model.Bucket)5 SocketTimeoutException (java.net.SocketTimeoutException)5 StorageObject (com.google.api.services.storage.model.StorageObject)4 GcsPath (org.apache.beam.sdk.util.gcsfs.GcsPath)4 JsonFactory (com.google.api.client.json.JsonFactory)3 MockLowLevelHttpResponse (com.google.api.client.testing.http.MockLowLevelHttpResponse)3 GoogleCredential (com.google.api.client.googleapis.auth.oauth2.GoogleCredential)2 GoogleNetHttpTransport (com.google.api.client.googleapis.javanet.GoogleNetHttpTransport)2 HttpTransport (com.google.api.client.http.HttpTransport)2 GenericJson (com.google.api.client.json.GenericJson)2 JacksonFactory (com.google.api.client.json.jackson2.JacksonFactory)2 MockLowLevelHttpRequest (com.google.api.client.testing.http.MockLowLevelHttpRequest)2 ImmutableList (com.google.common.collect.ImmutableList)2