Search in sources :

Example 6 with InputConfig

use of com.google.cloud.translate.v3.InputConfig in project java-automl by googleapis.

the class ImportDataset method importDataset.

// Import a dataset
static void importDataset(String projectId, String datasetId, String path) throws IOException, ExecutionException, InterruptedException, TimeoutException {
    Duration totalTimeout = Duration.ofMinutes(45);
    RetrySettings retrySettings = RetrySettings.newBuilder().setTotalTimeout(totalTimeout).build();
    AutoMlSettings.Builder builder = AutoMlSettings.newBuilder();
    builder.importDataSettings().setRetrySettings(retrySettings).build();
    AutoMlSettings settings = builder.build();
    // the "close" method on the client to safely clean up any remaining background resources.
    try (AutoMlClient client = AutoMlClient.create(settings)) {
        // Get the complete path of the dataset.
        DatasetName datasetFullId = DatasetName.of(projectId, "us-central1", datasetId);
        // Get multiple Google Cloud Storage URIs to import data from
        GcsSource gcsSource = GcsSource.newBuilder().addAllInputUris(Arrays.asList(path.split(","))).build();
        // Import data from the input URI
        InputConfig inputConfig = InputConfig.newBuilder().setGcsSource(gcsSource).build();
        System.out.println("Processing import...");
        // Start the import job
        OperationFuture<Empty, OperationMetadata> operation = client.importDataAsync(datasetFullId, inputConfig);
        System.out.format("Operation name: %s%n", operation.getName());
        // If you want to wait for the operation to finish, adjust the timeout appropriately. The
        // operation will still run if you choose not to wait for it to complete. You can check the
        // status of your operation using the operation's name.
        Empty response = operation.get(45, TimeUnit.MINUTES);
        System.out.format("Dataset imported. %s%n", response);
    } catch (TimeoutException e) {
        System.out.println("The operation's polling period was not long enough.");
        System.out.println("You can use the Operation's name to get the current status.");
        System.out.println("The import job is still running and will complete as expected.");
        throw e;
    }
}
Also used : RetrySettings(com.google.api.gax.retrying.RetrySettings) Empty(com.google.protobuf.Empty) GcsSource(com.google.cloud.automl.v1beta1.GcsSource) DatasetName(com.google.cloud.automl.v1beta1.DatasetName) Duration(org.threeten.bp.Duration) InputConfig(com.google.cloud.automl.v1beta1.InputConfig) AutoMlSettings(com.google.cloud.automl.v1beta1.AutoMlSettings) OperationMetadata(com.google.cloud.automl.v1beta1.OperationMetadata) AutoMlClient(com.google.cloud.automl.v1beta1.AutoMlClient) TimeoutException(java.util.concurrent.TimeoutException)

Example 7 with InputConfig

use of com.google.cloud.translate.v3.InputConfig in project java-automl by googleapis.

the class ImportDataset method importDataset.

// Import a dataset
static void importDataset(String projectId, String datasetId, String path) throws IOException, ExecutionException, InterruptedException, TimeoutException {
    // the "close" method on the client to safely clean up any remaining background resources.
    try (AutoMlClient client = AutoMlClient.create()) {
        // Get the complete path of the dataset.
        DatasetName datasetFullId = DatasetName.of(projectId, "us-central1", datasetId);
        // Get multiple Google Cloud Storage URIs to import data from
        GcsSource gcsSource = GcsSource.newBuilder().addAllInputUris(Arrays.asList(path.split(","))).build();
        // Import data from the input URI
        InputConfig inputConfig = InputConfig.newBuilder().setGcsSource(gcsSource).build();
        System.out.println("Processing import...");
        // Start the import job
        OperationFuture<Empty, OperationMetadata> operation = client.importDataAsync(datasetFullId, inputConfig);
        System.out.format("Operation name: %s%n", operation.getName());
        // If you want to wait for the operation to finish, adjust the timeout appropriately. The
        // operation will still run if you choose not to wait for it to complete. You can check the
        // status of your operation using the operation's name.
        Empty response = operation.get(45, TimeUnit.MINUTES);
        System.out.format("Dataset imported. %s%n", response);
    } catch (TimeoutException e) {
        System.out.println("The operation's polling period was not long enough.");
        System.out.println("You can use the Operation's name to get the current status.");
        System.out.println("The import job is still running and will complete as expected.");
        throw e;
    }
}
Also used : Empty(com.google.protobuf.Empty) GcsSource(com.google.cloud.automl.v1.GcsSource) DatasetName(com.google.cloud.automl.v1.DatasetName) InputConfig(com.google.cloud.automl.v1.InputConfig) OperationMetadata(com.google.cloud.automl.v1.OperationMetadata) AutoMlClient(com.google.cloud.automl.v1.AutoMlClient) TimeoutException(java.util.concurrent.TimeoutException)

Example 8 with InputConfig

use of com.google.cloud.translate.v3.InputConfig in project java-document-ai by googleapis.

the class BatchParseFormBeta method batchParseFormGcs.

public static void batchParseFormGcs(String projectId, String location, String outputGcsBucketName, String outputGcsPrefix, String inputGcsUri) throws IOException, InterruptedException, ExecutionException, TimeoutException {
    // the "close" method on the client to safely clean up any remaining background resources.
    try (DocumentUnderstandingServiceClient client = DocumentUnderstandingServiceClient.create()) {
        // Configure the request for processing the PDF
        String parent = String.format("projects/%s/locations/%s", projectId, location);
        // Improve form parsing results by providing key-value pair hints.
        // For each key hint, key is text that is likely to appear in the
        // document as a form field name (i.e. "DOB").
        // Value types are optional, but can be one or more of:
        // ADDRESS, LOCATION, ORGANIZATION, PERSON, PHONE_NUMBER, ID,
        // NUMBER, EMAIL, PRICE, TERMS, DATE, NAME
        KeyValuePairHint keyValuePairHint = KeyValuePairHint.newBuilder().setKey("Phone").addValueTypes("PHONE_NUMBER").build();
        KeyValuePairHint keyValuePairHint2 = KeyValuePairHint.newBuilder().setKey("Contact").addValueTypes("EMAIL").addValueTypes("NAME").build();
        // Setting enabled=True enables form extraction
        FormExtractionParams params = FormExtractionParams.newBuilder().setEnabled(true).addKeyValuePairHints(keyValuePairHint).addKeyValuePairHints(keyValuePairHint2).build();
        GcsSource inputUri = GcsSource.newBuilder().setUri(inputGcsUri).build();
        // mime_type can be application/pdf, image/tiff,
        // and image/gif, or application/json
        InputConfig config = InputConfig.newBuilder().setGcsSource(inputUri).setMimeType("application/pdf").build();
        GcsDestination gcsDestination = GcsDestination.newBuilder().setUri(String.format("gs://%s/%s", outputGcsBucketName, outputGcsPrefix)).build();
        OutputConfig outputConfig = OutputConfig.newBuilder().setGcsDestination(gcsDestination).setPagesPerShard(1).build();
        ProcessDocumentRequest request = ProcessDocumentRequest.newBuilder().setFormExtractionParams(params).setInputConfig(config).setOutputConfig(outputConfig).build();
        BatchProcessDocumentsRequest requests = BatchProcessDocumentsRequest.newBuilder().addRequests(request).setParent(parent).build();
        // Batch process document using a long-running operation.
        OperationFuture<BatchProcessDocumentsResponse, OperationMetadata> future = client.batchProcessDocumentsAsync(requests);
        // Wait for operation to complete.
        System.out.println("Waiting for operation to complete...");
        future.get(360, TimeUnit.SECONDS);
        System.out.println("Document processing complete.");
        Storage storage = StorageOptions.newBuilder().setProjectId(projectId).build().getService();
        Bucket bucket = storage.get(outputGcsBucketName);
        // List all of the files in the Storage bucket.
        Page<Blob> blobs = bucket.list(Storage.BlobListOption.currentDirectory(), Storage.BlobListOption.prefix(outputGcsPrefix));
        int idx = 0;
        for (Blob blob : blobs.iterateAll()) {
            if (!blob.isDirectory()) {
                System.out.printf("Fetched file #%d\n", ++idx);
                // Read the results
                // Download and store json data in a temp file.
                File tempFile = File.createTempFile("file", ".json");
                Blob fileInfo = storage.get(BlobId.of(outputGcsBucketName, blob.getName()));
                fileInfo.downloadTo(tempFile.toPath());
                // Parse json file into Document.
                FileReader reader = new FileReader(tempFile);
                Document.Builder builder = Document.newBuilder();
                JsonFormat.parser().merge(reader, builder);
                Document document = builder.build();
                // Get all of the document text as one big string.
                String text = document.getText();
                // Process the output.
                if (document.getPagesCount() > 0) {
                    Document.Page page1 = document.getPages(0);
                    for (Document.Page.FormField field : page1.getFormFieldsList()) {
                        String fieldName = getText(field.getFieldName(), text);
                        String fieldValue = getText(field.getFieldValue(), text);
                        System.out.println("Extracted form fields pair:");
                        System.out.printf("\t(%s, %s))", fieldName, fieldValue);
                    }
                }
                // Clean up temp file.
                tempFile.deleteOnExit();
            }
        }
    }
}
Also used : BatchProcessDocumentsResponse(com.google.cloud.documentai.v1beta2.BatchProcessDocumentsResponse) DocumentUnderstandingServiceClient(com.google.cloud.documentai.v1beta2.DocumentUnderstandingServiceClient) GcsSource(com.google.cloud.documentai.v1beta2.GcsSource) Page(com.google.api.gax.paging.Page) Document(com.google.cloud.documentai.v1beta2.Document) InputConfig(com.google.cloud.documentai.v1beta2.InputConfig) FileReader(java.io.FileReader) BatchProcessDocumentsRequest(com.google.cloud.documentai.v1beta2.BatchProcessDocumentsRequest) OperationMetadata(com.google.cloud.documentai.v1beta2.OperationMetadata) ProcessDocumentRequest(com.google.cloud.documentai.v1beta2.ProcessDocumentRequest) KeyValuePairHint(com.google.cloud.documentai.v1beta2.KeyValuePairHint) Blob(com.google.cloud.storage.Blob) KeyValuePairHint(com.google.cloud.documentai.v1beta2.KeyValuePairHint) OutputConfig(com.google.cloud.documentai.v1beta2.OutputConfig) Storage(com.google.cloud.storage.Storage) Bucket(com.google.cloud.storage.Bucket) FormExtractionParams(com.google.cloud.documentai.v1beta2.FormExtractionParams) GcsDestination(com.google.cloud.documentai.v1beta2.GcsDestination) File(java.io.File)

Example 9 with InputConfig

use of com.google.cloud.translate.v3.InputConfig in project java-document-ai by googleapis.

the class ParseFormBeta method parseForm.

public static void parseForm(String projectId, String location, String inputGcsUri) throws IOException, ExecutionException, InterruptedException {
    // the "close" method on the client to safely clean up any remaining background resources.
    try (DocumentUnderstandingServiceClient client = DocumentUnderstandingServiceClient.create()) {
        // Configure the request for processing the PDF
        String parent = String.format("projects/%s/locations/%s", projectId, location);
        // Improve form parsing results by providing key-value pair hints.
        // For each key hint, key is text that is likely to appear in the
        // document as a form field name (i.e. "DOB").
        // Value types are optional, but can be one or more of:
        // ADDRESS, LOCATION, ORGANIZATION, PERSON, PHONE_NUMBER, ID,
        // NUMBER, EMAIL, PRICE, TERMS, DATE, NAME
        KeyValuePairHint keyValuePairHint = KeyValuePairHint.newBuilder().setKey("Phone").addValueTypes("PHONE_NUMBER").build();
        KeyValuePairHint keyValuePairHint2 = KeyValuePairHint.newBuilder().setKey("Contact").addValueTypes("EMAIL").addValueTypes("NAME").build();
        // Setting enabled=True enables form extraction
        FormExtractionParams params = FormExtractionParams.newBuilder().setEnabled(true).addKeyValuePairHints(keyValuePairHint).addKeyValuePairHints(keyValuePairHint2).build();
        GcsSource uri = GcsSource.newBuilder().setUri(inputGcsUri).build();
        // mime_type can be application/pdf, image/tiff,
        // and image/gif, or application/json
        InputConfig config = InputConfig.newBuilder().setGcsSource(uri).setMimeType("application/pdf").build();
        ProcessDocumentRequest request = ProcessDocumentRequest.newBuilder().setParent(parent).setFormExtractionParams(params).setInputConfig(config).build();
        // Recognizes text entities in the PDF document
        Document response = client.processDocument(request);
        // Get all of the document text as one big string
        String text = response.getText();
        // Process the output
        if (response.getPagesCount() > 0) {
            Document.Page page1 = response.getPages(0);
            for (Document.Page.FormField field : page1.getFormFieldsList()) {
                String fieldName = getText(field.getFieldName(), text);
                String fieldValue = getText(field.getFieldValue(), text);
                System.out.println("Extracted form fields pair:");
                System.out.printf("\t(%s, %s))", fieldName, fieldValue);
            }
        }
    }
}
Also used : KeyValuePairHint(com.google.cloud.documentai.v1beta2.KeyValuePairHint) DocumentUnderstandingServiceClient(com.google.cloud.documentai.v1beta2.DocumentUnderstandingServiceClient) GcsSource(com.google.cloud.documentai.v1beta2.GcsSource) FormExtractionParams(com.google.cloud.documentai.v1beta2.FormExtractionParams) InputConfig(com.google.cloud.documentai.v1beta2.InputConfig) Document(com.google.cloud.documentai.v1beta2.Document) ProcessDocumentRequest(com.google.cloud.documentai.v1beta2.ProcessDocumentRequest)

Example 10 with InputConfig

use of com.google.cloud.translate.v3.InputConfig in project spring-cloud-gcp by GoogleCloudPlatform.

the class DocumentOcrTemplate method runOcrForDocument.

/**
 * Runs OCR processing for a specified {@code document} and generates OCR output files under the
 * path specified by {@code outputFilePathPrefix}.
 *
 * <p>For example, if you specify an {@code outputFilePathPrefix} of
 * "gs://bucket_name/ocr_results/myDoc_", all the output files of OCR processing will be saved
 * under prefix, such as:
 *
 * <ul>
 *   <li>gs://bucket_name/ocr_results/myDoc_output-1-to-5.json
 *   <li>gs://bucket_name/ocr_results/myDoc_output-6-to-10.json
 *   <li>gs://bucket_name/ocr_results/myDoc_output-11-to-15.json
 * </ul>
 *
 * <p>Note: OCR processing operations may take several minutes to complete, so it may not be
 * advisable to block on the completion of the operation. One may use the returned {@link
 * ListenableFuture} to register callbacks or track the status of the operation.
 *
 * @param document The {@link GoogleStorageLocation} of the document to run OCR processing
 * @param outputFilePathPrefix The {@link GoogleStorageLocation} of a file, folder, or a bucket
 *     describing the path for which all output files shall be saved under
 * @return A {@link ListenableFuture} allowing you to register callbacks or wait for the
 *     completion of the operation.
 */
public ListenableFuture<DocumentOcrResultSet> runOcrForDocument(GoogleStorageLocation document, GoogleStorageLocation outputFilePathPrefix) {
    Assert.isTrue(document.isFile(), "Provided document location is not a valid file location: " + document);
    GcsSource gcsSource = GcsSource.newBuilder().setUri(document.uriString()).build();
    String contentType = extractContentType(document);
    InputConfig inputConfig = InputConfig.newBuilder().setMimeType(contentType).setGcsSource(gcsSource).build();
    GcsDestination gcsDestination = GcsDestination.newBuilder().setUri(outputFilePathPrefix.uriString()).build();
    OutputConfig outputConfig = OutputConfig.newBuilder().setGcsDestination(gcsDestination).setBatchSize(this.jsonOutputBatchSize).build();
    AsyncAnnotateFileRequest request = AsyncAnnotateFileRequest.newBuilder().addFeatures(DOCUMENT_OCR_FEATURE).setInputConfig(inputConfig).setOutputConfig(outputConfig).build();
    OperationFuture<AsyncBatchAnnotateFilesResponse, OperationMetadata> result = imageAnnotatorClient.asyncBatchAnnotateFilesAsync(Collections.singletonList(request));
    return extractOcrResultFuture(result);
}
Also used : GcsSource(com.google.cloud.vision.v1.GcsSource) OutputConfig(com.google.cloud.vision.v1.OutputConfig) AsyncBatchAnnotateFilesResponse(com.google.cloud.vision.v1.AsyncBatchAnnotateFilesResponse) InputConfig(com.google.cloud.vision.v1.InputConfig) AsyncAnnotateFileRequest(com.google.cloud.vision.v1.AsyncAnnotateFileRequest) GcsDestination(com.google.cloud.vision.v1.GcsDestination) OperationMetadata(com.google.cloud.vision.v1.OperationMetadata)

Aggregations

Document (com.google.cloud.documentai.v1beta2.Document)7 DocumentUnderstandingServiceClient (com.google.cloud.documentai.v1beta2.DocumentUnderstandingServiceClient)7 GcsSource (com.google.cloud.documentai.v1beta2.GcsSource)7 InputConfig (com.google.cloud.documentai.v1beta2.InputConfig)7 ProcessDocumentRequest (com.google.cloud.documentai.v1beta2.ProcessDocumentRequest)7 InputConfig (com.google.cloud.vision.v1.InputConfig)6 GcsSource (com.google.cloud.translate.v3.GcsSource)5 LocationName (com.google.cloud.translate.v3.LocationName)5 TranslationServiceClient (com.google.cloud.translate.v3.TranslationServiceClient)5 BatchTranslateMetadata (com.google.cloud.translate.v3.BatchTranslateMetadata)4 BatchTranslateResponse (com.google.cloud.translate.v3.BatchTranslateResponse)4 BatchTranslateTextRequest (com.google.cloud.translate.v3.BatchTranslateTextRequest)4 GcsDestination (com.google.cloud.translate.v3.GcsDestination)4 InputConfig (com.google.cloud.translate.v3.InputConfig)4 OutputConfig (com.google.cloud.translate.v3.OutputConfig)4 ByteString (com.google.protobuf.ByteString)4 GlossaryName (com.google.cloud.translate.v3.GlossaryName)3 AnnotateImageResponse (com.google.cloud.vision.v1.AnnotateImageResponse)3 Feature (com.google.cloud.vision.v1.Feature)3 GcsSource (com.google.cloud.vision.v1.GcsSource)3