Search in sources :

Example 1 with ListEntriesRequest

use of com.google.cloud.datacatalog.v1beta1.ListEntriesRequest in project DataflowTemplates by GoogleCloudPlatform.

the class DataCatalogSchemaUtils method getSchemasForEntryGroup.

/**
 * Retrieve all of the {@link Schema}s associated to {@link Entry}s in an {@link EntryGroup}.
 */
public static Map<String, Schema> getSchemasForEntryGroup(String gcpProject, String entryGroupId) {
    DataCatalogClient client = null;
    try {
        client = DataCatalogClient.create();
    } catch (IOException e) {
        throw new RuntimeException("Unable to create a DataCatalogClient", e);
    }
    if (client == null) {
        return null;
    }
    String formattedParent = DataCatalogClient.formatEntryGroupName(gcpProject, DEFAULT_LOCATION, entryGroupId);
    List<Entry> entries = new ArrayList<>();
    ListEntriesRequest request = ListEntriesRequest.newBuilder().setParent(formattedParent).build();
    while (true) {
        ListEntriesResponse response = client.listEntriesCallable().call(request);
        entries.addAll(response.getEntriesList());
        String nextPageToken = response.getNextPageToken();
        if (!Strings.isNullOrEmpty(nextPageToken)) {
            request = request.toBuilder().setPageToken(nextPageToken).build();
        } else {
            break;
        }
    }
    LOG.debug("Fetched entries: {}", entries);
    return entries.stream().collect(Collectors.toMap(Entry::getDescription, e -> SchemaUtils.toBeamSchema(e.getSchema())));
}
Also used : ListEntriesResponse(com.google.cloud.datacatalog.v1beta1.ListEntriesResponse) Logger(org.slf4j.Logger) DataCatalogClient(com.google.cloud.datacatalog.v1beta1.DataCatalogClient) CreateEntryRequest(com.google.cloud.datacatalog.v1beta1.CreateEntryRequest) LocationName(com.google.cloud.datacatalog.v1beta1.LocationName) LoggerFactory(org.slf4j.LoggerFactory) IOException(java.io.IOException) UpdateEntryRequest(com.google.cloud.datacatalog.v1beta1.UpdateEntryRequest) EntryGroupName(com.google.cloud.datacatalog.v1.EntryGroupName) Collectors(java.util.stream.Collectors) Schema(org.apache.beam.sdk.schemas.Schema) AlreadyExistsException(com.google.api.gax.rpc.AlreadyExistsException) ApiException(com.google.api.gax.rpc.ApiException) LookupEntryRequest(com.google.cloud.datacatalog.v1beta1.LookupEntryRequest) ArrayList(java.util.ArrayList) EntryGroup(com.google.cloud.datacatalog.v1beta1.EntryGroup) ListEntriesResponse(com.google.cloud.datacatalog.v1beta1.ListEntriesResponse) Strings(com.google.common.base.Strings) List(java.util.List) Entry(com.google.cloud.datacatalog.v1beta1.Entry) Map(java.util.Map) ListEntriesRequest(com.google.cloud.datacatalog.v1beta1.ListEntriesRequest) CreateEntryGroupRequest(com.google.cloud.datacatalog.v1beta1.CreateEntryGroupRequest) Entry(com.google.cloud.datacatalog.v1beta1.Entry) ArrayList(java.util.ArrayList) DataCatalogClient(com.google.cloud.datacatalog.v1beta1.DataCatalogClient) IOException(java.io.IOException) ListEntriesRequest(com.google.cloud.datacatalog.v1beta1.ListEntriesRequest)

Aggregations

AlreadyExistsException (com.google.api.gax.rpc.AlreadyExistsException)1 ApiException (com.google.api.gax.rpc.ApiException)1 EntryGroupName (com.google.cloud.datacatalog.v1.EntryGroupName)1 CreateEntryGroupRequest (com.google.cloud.datacatalog.v1beta1.CreateEntryGroupRequest)1 CreateEntryRequest (com.google.cloud.datacatalog.v1beta1.CreateEntryRequest)1 DataCatalogClient (com.google.cloud.datacatalog.v1beta1.DataCatalogClient)1 Entry (com.google.cloud.datacatalog.v1beta1.Entry)1 EntryGroup (com.google.cloud.datacatalog.v1beta1.EntryGroup)1 ListEntriesRequest (com.google.cloud.datacatalog.v1beta1.ListEntriesRequest)1 ListEntriesResponse (com.google.cloud.datacatalog.v1beta1.ListEntriesResponse)1 LocationName (com.google.cloud.datacatalog.v1beta1.LocationName)1 LookupEntryRequest (com.google.cloud.datacatalog.v1beta1.LookupEntryRequest)1 UpdateEntryRequest (com.google.cloud.datacatalog.v1beta1.UpdateEntryRequest)1 Strings (com.google.common.base.Strings)1 IOException (java.io.IOException)1 ArrayList (java.util.ArrayList)1 List (java.util.List)1 Map (java.util.Map)1 Collectors (java.util.stream.Collectors)1 Schema (org.apache.beam.sdk.schemas.Schema)1