Search in sources :

Example 1 with PagedIterable

use of com.azure.core.http.rest.PagedIterable in project bulk-scan-processor by hmcts.

the class BlobContainerInfoContributorTest method should_return_info_about_application.

@Test
@SuppressWarnings("unchecked")
public void should_return_info_about_application() throws Exception {
    PagedIterable pagedIterable = mock(PagedIterable.class);
    given(blobServiceClient.listBlobContainers()).willReturn(pagedIterable);
    given(pagedIterable.stream()).willReturn(Stream.of(new BlobContainerItem().setName("sscs"), new BlobContainerItem().setName("bulkscan"), new BlobContainerItem().setName("sscs-rejected")));
    this.mockMvc.perform(get("/info")).andDo(print()).andExpect(status().isOk()).andExpect(jsonPath("$.containers[0]").value("sscs")).andExpect(jsonPath("$.containers[1]").value("bulkscan")).andExpect(jsonPath("$.containers[2]").value("sscs-rejected"));
}
Also used : BlobContainerItem(com.azure.storage.blob.models.BlobContainerItem) PagedIterable(com.azure.core.http.rest.PagedIterable) IntegrationTest(uk.gov.hmcts.reform.bulkscanprocessor.config.IntegrationTest) Test(org.junit.jupiter.api.Test)

Example 2 with PagedIterable

use of com.azure.core.http.rest.PagedIterable in project bulk-scan-processor by hmcts.

the class BlobManagerTest method listInputContainers_retrieves_input_containers_from_client.

@Test
void listInputContainers_retrieves_input_containers_from_client() {
    List<BlobContainerItem> allContainers = Arrays.asList(mockBlobContainerItem("test1"), mockBlobContainerItem("test1-rejected"), mockBlobContainerItem("test2"));
    PagedIterable pagedIterable = mock(PagedIterable.class);
    given(blobServiceClient.listBlobContainers()).willReturn(pagedIterable);
    given(pagedIterable.stream()).willReturn(allContainers.stream());
    given(blobManagementProperties.getBlobSelectedContainer()).willReturn("all");
    BlobContainerClient test1ContainerClient = mock(BlobContainerClient.class);
    BlobContainerClient test2ContainerClient = mock(BlobContainerClient.class);
    given(blobServiceClient.getBlobContainerClient("test1")).willReturn(test1ContainerClient);
    given(blobServiceClient.getBlobContainerClient("test2")).willReturn(test2ContainerClient);
    List<BlobContainerClient> containers = blobManager.listInputContainerClients();
    assertThat(containers).hasSameElementsAs(Arrays.asList(test1ContainerClient, test2ContainerClient));
    verify(blobServiceClient, times(2)).getBlobContainerClient(anyString());
    verifyNoMoreInteractions(blobServiceClient);
}
Also used : BlobContainerItem(com.azure.storage.blob.models.BlobContainerItem) BlobContainerClient(com.azure.storage.blob.BlobContainerClient) PagedIterable(com.azure.core.http.rest.PagedIterable) Test(org.junit.jupiter.api.Test)

Example 3 with PagedIterable

use of com.azure.core.http.rest.PagedIterable in project bulk-scan-processor by hmcts.

the class BlobManagerTest method listRejectedContainers_retrieves_rejected_containers_only.

@Test
void listRejectedContainers_retrieves_rejected_containers_only() {
    // given
    List<BlobContainerItem> allContainers = Arrays.asList(mockBlobContainerItem("test1"), mockBlobContainerItem("test1-rejected"), mockBlobContainerItem("test2"), mockBlobContainerItem("test2-rejected"));
    PagedIterable pagedIterable = mock(PagedIterable.class);
    given(blobServiceClient.listBlobContainers()).willReturn(pagedIterable);
    given(pagedIterable.stream()).willReturn(allContainers.stream());
    given(blobServiceClient.getBlobContainerClient(anyString())).willReturn(mock(BlobContainerClient.class));
    // when
    List<BlobContainerClient> rejectedContainers = blobManager.listRejectedContainers();
    // then
    assertThat(rejectedContainers.size()).isEqualTo(2);
    var conditionCapturer = ArgumentCaptor.forClass(String.class);
    verify(blobServiceClient, times(2)).getBlobContainerClient(conditionCapturer.capture());
    assertThat(conditionCapturer.getAllValues()).hasSameElementsAs(Arrays.asList("test1-rejected", "test2-rejected"));
}
Also used : BlobContainerItem(com.azure.storage.blob.models.BlobContainerItem) BlobContainerClient(com.azure.storage.blob.BlobContainerClient) PagedIterable(com.azure.core.http.rest.PagedIterable) Test(org.junit.jupiter.api.Test)

Example 4 with PagedIterable

use of com.azure.core.http.rest.PagedIterable in project bulk-scan-processor by hmcts.

the class RejectedFilesReportServiceTest method setUpContainer.

@SuppressWarnings("unchecked")
private void setUpContainer(BlobContainerClient container, List<BlobItem> listBlobItems) {
    PagedIterable pagedIterable = mock(PagedIterable.class);
    given(pagedIterable.stream()).willReturn(listBlobItems.stream());
    given(container.listBlobs(any(), eq(null))).willReturn(pagedIterable);
}
Also used : PagedIterable(com.azure.core.http.rest.PagedIterable)

Example 5 with PagedIterable

use of com.azure.core.http.rest.PagedIterable in project vividus by vividus-framework.

the class BlobStorageSteps method findBlobs.

/**
 * Finds blobs with names filtered by the specified rules in the container.
 *
 * @param filter            Filter to apply to blob names.
 *                          <div>Example:</div>
 *                          <code>
 *                          <br>When I filter blobs by:
 *                          <br>|blobNamePrefix|blobNameFilterRule|blobNameFilterValue|resultsLimit|
 *                          <br>|data/         |contains          |file-key.txt       |10          |
 *                          <br>in container `global` of storage account `storage`
 *                          <br> and save result to story variable `blobs`
 *                          </code>
 *                          <br>
 *                          <br>where all filters are optional, but at least one rule is required.
 *                          <ul>
 *                          <li><code>blobNameFilterRule</code> The blob name comparison rule: "is equal to",
 *                           "contains", "does not contain" or "matches".
 *                           Should be specified along with <i>blobNameFilterValue</i>.</li>
 *                          <li><code>blobNameFilterValue</code> The full or partial blob name to be matched.
 *                           Should be specified along with <i>blobNameFilterRule</i>.</li>
 *                          <li><code>blobNamePrefix</code> The prefix which blob names should start with.</li>
 *                          <li><code>resultsLimit</code> Maximum number of blob names to return.</li>
 *                          </ul>
 * @param containerName     The name of the container to point to.
 * @param storageAccountKey The key to Storage Account endpoint.
 * @param scopes            The set (comma separated list of scopes e.g.: STORY, NEXT_BATCHES) of the variable
 *                          scopes.<br>
 *                          <i>Available scopes:</i>
 *                          <ul>
 *                          <li><b>STEP</b> - the variable will be available only within the step,
 *                          <li><b>SCENARIO</b> - the variable will be available only within the scenario,
 *                          <li><b>STORY</b> - the variable will be available within the whole story,
 *                          <li><b>NEXT_BATCHES</b> - the variable will be available starting from next batch
 *                          </ul>
 * @param variableName      The variable name to store the list of found blob names.
 */
@When("I filter blobs by:$filter in container `$containerName` of storage account `$storageAccountKey`" + " and save result to $scopes variable `$variableName`")
public void findBlobs(BlobFilter filter, String containerName, String storageAccountKey, Set<VariableScope> scopes, String variableName) {
    BlobContainerClient blobContainerClient = createBlobContainerClient(containerName, storageAccountKey);
    ListBlobsOptions options = new ListBlobsOptions();
    filter.getBlobNamePrefix().ifPresent(options::setPrefix);
    options.setMaxResultsPerPage(filter.getResultsLimit().map(limit -> Math.min(limit, DEFAULT_MAX_RESULTS_PER_PAGE)).orElse(DEFAULT_MAX_RESULTS_PER_PAGE));
    PagedIterable<BlobItem> blobItems = blobContainerClient.listBlobs(options, null);
    Stream<String> blobNames = StreamSupport.stream(blobItems.iterableByPage().spliterator(), false).map(PagedResponse::getValue).flatMap(List::stream).map(BlobItem::getName);
    Stream<String> filteredBlobNames = filter.getBlobNameMatcher().map(matcher -> blobNames.filter(matcher::matches)).orElse(blobNames);
    List<String> result = filter.getResultsLimit().map(filteredBlobNames::limit).orElse(filteredBlobNames).collect(Collectors.toList());
    variableContext.putVariable(scopes, variableName, result);
}
Also used : BlobItem(com.azure.storage.blob.models.BlobItem) LoadingCache(com.google.common.cache.LoadingCache) ByteArrayOutputStream(java.io.ByteArrayOutputStream) VariableContext(org.vividus.context.VariableContext) LoggerFactory(org.slf4j.LoggerFactory) PropertyMappedCollection(org.vividus.util.property.PropertyMappedCollection) JsonUtils(org.vividus.util.json.JsonUtils) When(org.jbehave.core.annotations.When) BlobClient(com.azure.storage.blob.BlobClient) VariableScope(org.vividus.variable.VariableScope) StreamSupport(java.util.stream.StreamSupport) BlobServiceClientBuilder(com.azure.storage.blob.BlobServiceClientBuilder) BlobItem(com.azure.storage.blob.models.BlobItem) ListBlobsOptions(com.azure.storage.blob.models.ListBlobsOptions) BlobProperties(com.azure.storage.blob.models.BlobProperties) Logger(org.slf4j.Logger) BlobContainerClient(com.azure.storage.blob.BlobContainerClient) StringComparisonRule(org.vividus.steps.StringComparisonRule) Set(java.util.Set) IOException(java.io.IOException) ResourceUtils(org.vividus.util.ResourceUtils) Collectors(java.util.stream.Collectors) StandardCharsets(java.nio.charset.StandardCharsets) CacheLoader(com.google.common.cache.CacheLoader) BlobServiceClient(com.azure.storage.blob.BlobServiceClient) List(java.util.List) Stream(java.util.stream.Stream) Matcher(org.hamcrest.Matcher) TokenCredential(com.azure.core.credential.TokenCredential) PagedIterable(com.azure.core.http.rest.PagedIterable) PagedResponse(com.azure.core.http.rest.PagedResponse) CacheBuilder(com.google.common.cache.CacheBuilder) DataWrapper(org.vividus.steps.DataWrapper) BlobServiceProperties(com.azure.storage.blob.models.BlobServiceProperties) BinaryData(com.azure.core.util.BinaryData) BlobContainerClient(com.azure.storage.blob.BlobContainerClient) ListBlobsOptions(com.azure.storage.blob.models.ListBlobsOptions) PagedResponse(com.azure.core.http.rest.PagedResponse) When(org.jbehave.core.annotations.When)

Aggregations

PagedIterable (com.azure.core.http.rest.PagedIterable)5 BlobContainerClient (com.azure.storage.blob.BlobContainerClient)3 BlobContainerItem (com.azure.storage.blob.models.BlobContainerItem)3 Test (org.junit.jupiter.api.Test)3 TokenCredential (com.azure.core.credential.TokenCredential)1 PagedResponse (com.azure.core.http.rest.PagedResponse)1 BinaryData (com.azure.core.util.BinaryData)1 BlobClient (com.azure.storage.blob.BlobClient)1 BlobServiceClient (com.azure.storage.blob.BlobServiceClient)1 BlobServiceClientBuilder (com.azure.storage.blob.BlobServiceClientBuilder)1 BlobItem (com.azure.storage.blob.models.BlobItem)1 BlobProperties (com.azure.storage.blob.models.BlobProperties)1 BlobServiceProperties (com.azure.storage.blob.models.BlobServiceProperties)1 ListBlobsOptions (com.azure.storage.blob.models.ListBlobsOptions)1 CacheBuilder (com.google.common.cache.CacheBuilder)1 CacheLoader (com.google.common.cache.CacheLoader)1 LoadingCache (com.google.common.cache.LoadingCache)1 ByteArrayOutputStream (java.io.ByteArrayOutputStream)1 IOException (java.io.IOException)1 StandardCharsets (java.nio.charset.StandardCharsets)1