Search in sources :

Example 1 with DescribeJournalS3ExportResult

use of com.amazonaws.services.qldb.model.DescribeJournalS3ExportResult in project amazon-qldb-dmv-sample-java by aws-samples.

the class ExportJournal method waitForExportToComplete.

/**
 * Wait for the JournalS3Export to complete.
 *
 * @param ledgerName
 *              Name of the ledger.
 * @param exportId
 *              Optional KMS ARN used for S3-KMS encryption.
 * @param awaitTimeoutMs
 *              Milliseconds to wait for export to complete.
 * @return {@link DescribeJournalS3ExportResult} from QLDB.
 * @throws InterruptedException if thread is interrupted while busy waiting for JournalS3Export to complete.
 */
public static DescribeJournalS3ExportResult waitForExportToComplete(String ledgerName, String exportId, long awaitTimeoutMs) throws InterruptedException {
    log.info("Waiting for JournalS3Export for " + exportId + "to complete.");
    int count = 0;
    long maxRetryCount = (awaitTimeoutMs / EXPORT_COMPLETION_POLL_PERIOD_MS) + 1;
    while (count < maxRetryCount) {
        DescribeJournalS3ExportResult result = DescribeJournalExport.describeExport(ledgerName, exportId);
        if (result.getExportDescription().getStatus().equalsIgnoreCase(ExportStatus.COMPLETED.name())) {
            log.info("JournalS3Export completed.");
            return result;
        }
        log.info("JournalS3Export is still in progress. Please wait. ");
        Thread.sleep(EXPORT_COMPLETION_POLL_PERIOD_MS);
        count++;
    }
    throw new IllegalStateException("Journal Export did not complete for " + exportId);
}
Also used : DescribeJournalS3ExportResult(com.amazonaws.services.qldb.model.DescribeJournalS3ExportResult)

Example 2 with DescribeJournalS3ExportResult

use of com.amazonaws.services.qldb.model.DescribeJournalS3ExportResult in project amazon-qldb-dmv-sample-java by aws-samples.

the class JournalS3ExportReader method readExport.

/**
 * Read the S3 export within a {@link JournalBlock}.
 *
 * @param describeJournalS3ExportResult
 *              The result from the QLDB database describing a journal export.
 * @param amazonS3
 *              The low level S3 client.
 * @return a list of {@link JournalBlock}.
 */
public static List<JournalBlock> readExport(final DescribeJournalS3ExportResult describeJournalS3ExportResult, final AmazonS3 amazonS3) {
    S3ExportConfiguration exportConfiguration = describeJournalS3ExportResult.getExportDescription().getS3ExportConfiguration();
    ListObjectsV2Request listObjectsRequest = new ListObjectsV2Request().withBucketName(exportConfiguration.getBucket()).withPrefix(exportConfiguration.getPrefix());
    ListObjectsV2Result listObjectsV2Result = amazonS3.listObjectsV2(listObjectsRequest);
    log.info("Found the following objects for list from s3: ");
    listObjectsV2Result.getObjectSummaries().forEach(s3ObjectSummary -> log.info(s3ObjectSummary.getKey()));
    // Validate initial manifest file was written.
    String expectedManifestKey = exportConfiguration.getPrefix() + describeJournalS3ExportResult.getExportDescription().getExportId() + ".started" + ".manifest";
    String initialManifestKey = listObjectsV2Result.getObjectSummaries().stream().filter(s3ObjectSummary -> s3ObjectSummary.getKey().equalsIgnoreCase(expectedManifestKey)).map(S3ObjectSummary::getKey).findFirst().orElseThrow(() -> new IllegalStateException("Initial manifest not found."));
    log.info("Found the initial manifest with key " + initialManifestKey);
    // Find the final manifest file, it should contain the exportId in it.
    String completedManifestFileKey = listObjectsV2Result.getObjectSummaries().stream().filter(s3ObjectSummary -> s3ObjectSummary.getKey().endsWith("completed.manifest") && (s3ObjectSummary.getKey().contains(describeJournalS3ExportResult.getExportDescription().getExportId()))).map(S3ObjectSummary::getKey).findFirst().orElseThrow(() -> new IllegalStateException("Completed manifest not found."));
    log.info("Found the completed manifest with key " + completedManifestFileKey);
    // Read manifest file to find data file keys.
    S3Object completedManifestObject = amazonS3.getObject(exportConfiguration.getBucket(), completedManifestFileKey);
    List<String> dataFileKeys = getDataFileKeysFromManifest(completedManifestObject);
    log.info("Found the following keys in the manifest files: " + dataFileKeys);
    List<JournalBlock> journalBlocks = new ArrayList<>();
    for (String key : dataFileKeys) {
        log.info("Reading file with S3 key " + key + " from bucket: " + exportConfiguration.getBucket());
        S3Object s3Object = amazonS3.getObject(exportConfiguration.getBucket(), key);
        List<JournalBlock> blocks = getJournalBlocks(s3Object);
        compareKeyWithContentRange(key, blocks.get(0), blocks.get(blocks.size() - 1));
        journalBlocks.addAll(blocks);
    }
    return journalBlocks;
}
Also used : JournalBlock(software.amazon.qldb.tutorial.qldb.JournalBlock) Logger(org.slf4j.Logger) IonSystem(com.amazon.ion.IonSystem) IonReaderBuilder(com.amazon.ion.system.IonReaderBuilder) LoggerFactory(org.slf4j.LoggerFactory) IOException(java.io.IOException) IonList(com.amazon.ion.IonList) DescribeJournalS3ExportResult(com.amazonaws.services.qldb.model.DescribeJournalS3ExportResult) ListObjectsV2Result(com.amazonaws.services.s3.model.ListObjectsV2Result) ArrayList(java.util.ArrayList) IonStruct(com.amazon.ion.IonStruct) ListObjectsV2Request(com.amazonaws.services.s3.model.ListObjectsV2Request) List(java.util.List) IonReader(com.amazon.ion.IonReader) IonType(com.amazon.ion.IonType) IonSystemBuilder(com.amazon.ion.system.IonSystemBuilder) S3Object(com.amazonaws.services.s3.model.S3Object) S3ExportConfiguration(com.amazonaws.services.qldb.model.S3ExportConfiguration) AmazonS3(com.amazonaws.services.s3.AmazonS3) S3ObjectSummary(com.amazonaws.services.s3.model.S3ObjectSummary) IonString(com.amazon.ion.IonString) ListObjectsV2Request(com.amazonaws.services.s3.model.ListObjectsV2Request) ListObjectsV2Result(com.amazonaws.services.s3.model.ListObjectsV2Result) ArrayList(java.util.ArrayList) S3ExportConfiguration(com.amazonaws.services.qldb.model.S3ExportConfiguration) S3ObjectSummary(com.amazonaws.services.s3.model.S3ObjectSummary) IonString(com.amazon.ion.IonString) S3Object(com.amazonaws.services.s3.model.S3Object) JournalBlock(software.amazon.qldb.tutorial.qldb.JournalBlock)

Example 3 with DescribeJournalS3ExportResult

use of com.amazonaws.services.qldb.model.DescribeJournalS3ExportResult in project amazon-qldb-dmv-sample-java by aws-samples.

the class DescribeJournalExport method describeExport.

/**
 * Describe a journal export.
 *
 * @param ledgerName
 *              The ledger from which the journal is being exported.
 * @param exportId
 *              Unique ExportID of the journal export.
 * @return {@link DescribeJournalS3ExportResult} from QLDB.
 */
public static DescribeJournalS3ExportResult describeExport(String ledgerName, String exportId) {
    log.info("Let's describe a journal export for ledger with name: {}, ExportId: {}...", ledgerName, exportId);
    DescribeJournalS3ExportRequest request = new DescribeJournalS3ExportRequest().withName(ledgerName).withExportId(exportId);
    DescribeJournalS3ExportResult result = client.describeJournalS3Export(request);
    log.info("Export described. result = " + result);
    return result;
}
Also used : DescribeJournalS3ExportResult(com.amazonaws.services.qldb.model.DescribeJournalS3ExportResult) DescribeJournalS3ExportRequest(com.amazonaws.services.qldb.model.DescribeJournalS3ExportRequest)

Aggregations

DescribeJournalS3ExportResult (com.amazonaws.services.qldb.model.DescribeJournalS3ExportResult)3 IonList (com.amazon.ion.IonList)1 IonReader (com.amazon.ion.IonReader)1 IonString (com.amazon.ion.IonString)1 IonStruct (com.amazon.ion.IonStruct)1 IonSystem (com.amazon.ion.IonSystem)1 IonType (com.amazon.ion.IonType)1 IonReaderBuilder (com.amazon.ion.system.IonReaderBuilder)1 IonSystemBuilder (com.amazon.ion.system.IonSystemBuilder)1 DescribeJournalS3ExportRequest (com.amazonaws.services.qldb.model.DescribeJournalS3ExportRequest)1 S3ExportConfiguration (com.amazonaws.services.qldb.model.S3ExportConfiguration)1 AmazonS3 (com.amazonaws.services.s3.AmazonS3)1 ListObjectsV2Request (com.amazonaws.services.s3.model.ListObjectsV2Request)1 ListObjectsV2Result (com.amazonaws.services.s3.model.ListObjectsV2Result)1 S3Object (com.amazonaws.services.s3.model.S3Object)1 S3ObjectSummary (com.amazonaws.services.s3.model.S3ObjectSummary)1 IOException (java.io.IOException)1 ArrayList (java.util.ArrayList)1 List (java.util.List)1 Logger (org.slf4j.Logger)1