Examples with GetPartitionsResponsePage com.palantir.foundry.athena.api.GetPartitionsResponsePage used on opensource projects

Example 1 with GetPartitionsResponsePage

use of com.palantir.foundry.athena.api.GetPartitionsResponsePage in project foundry-athena-query-federation-connector by palantir.

the class PartitionFetcher method getAndWritePartitions.

private void getAndWritePartitions(BlockWriter blockWriter, GetTableLayoutRequest request, QueryStatusChecker queryStatusChecker) {
    CatalogLocator locator = FoundryAthenaObjectMapper.objectMapper().convertValue(request.getSchema().getCustomMetadata(), CatalogLocator.class);
    Optional<String> pageToken = Optional.empty();
    while (queryStatusChecker.isQueryRunning()) {
        GetPartitionsResponsePage page = metadataService.getPartitions(authProvider.getAuthHeader(), GetPartitionsRequest.builder().locator(locator).limit(PAGE_SIZE).pageToken(pageToken).build());
        page.getPartitions().forEach(partition -> blockWriter.writeRows((block, rowNum) -> {
            boolean matched = partition.get().entrySet().stream().map(fieldName -> fieldName.getValue().accept(new PartitionValueWriter(block, fieldName.getKey(), rowNum))).reduce(true, Boolean::logicalAnd);
            // if all fields passed then we wrote 1 row
            return matched ? 1 : 0;
        }));
        if (page.getNextPageToken().isPresent()) {
            pageToken = page.getNextPageToken();
        } else {
            return;
        }
    }
}