Search in sources :

Example 51 with CloudBlobClient

use of com.microsoft.azure.storage.blob.CloudBlobClient in project nifi by apache.

the class DeleteAzureBlobStorage method onTrigger.

@Override
public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
    FlowFile flowFile = session.get();
    if (flowFile == null) {
        return;
    }
    final long startNanos = System.nanoTime();
    String containerName = context.getProperty(AzureStorageUtils.CONTAINER).evaluateAttributeExpressions(flowFile).getValue();
    String blobPath = context.getProperty(BLOB).evaluateAttributeExpressions(flowFile).getValue();
    try {
        CloudBlobClient blobClient = AzureStorageUtils.createCloudBlobClient(context, getLogger(), flowFile);
        CloudBlobContainer container = blobClient.getContainerReference(containerName);
        CloudBlob blob = container.getBlockBlobReference(blobPath);
        blob.deleteIfExists();
        session.transfer(flowFile, REL_SUCCESS);
        final long transferMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);
        session.getProvenanceReporter().send(flowFile, blob.getSnapshotQualifiedUri().toString(), transferMillis);
    } catch (StorageException | URISyntaxException e) {
        getLogger().error("Failed to delete the specified blob {} from Azure Storage. Routing to failure", new Object[] { blobPath }, e);
        flowFile = session.penalize(flowFile);
        session.transfer(flowFile, REL_FAILURE);
    }
}
Also used : FlowFile(org.apache.nifi.flowfile.FlowFile) CloudBlob(com.microsoft.azure.storage.blob.CloudBlob) CloudBlobClient(com.microsoft.azure.storage.blob.CloudBlobClient) CloudBlobContainer(com.microsoft.azure.storage.blob.CloudBlobContainer) URISyntaxException(java.net.URISyntaxException) StorageException(com.microsoft.azure.storage.StorageException)

Example 52 with CloudBlobClient

use of com.microsoft.azure.storage.blob.CloudBlobClient in project jackrabbit-oak by apache.

the class Utils method getBlobClient.

/**
 * Create CloudBlobClient from properties.
 *
 * @param connectionString connectionString to configure @link {@link CloudBlobClient}
 * @return {@link CloudBlobClient}
 */
public static CloudBlobClient getBlobClient(final String connectionString) throws URISyntaxException, InvalidKeyException {
    CloudStorageAccount account = CloudStorageAccount.parse(connectionString);
    CloudBlobClient client = account.createCloudBlobClient();
    return client;
}
Also used : CloudBlobClient(com.microsoft.azure.storage.blob.CloudBlobClient) CloudStorageAccount(com.microsoft.azure.storage.CloudStorageAccount)

Example 53 with CloudBlobClient

use of com.microsoft.azure.storage.blob.CloudBlobClient in project photon-model by vmware.

the class AzureInstanceService method deleteVHDBlobsInAzure.

private DeferredResult<Void> deleteVHDBlobsInAzure(AzureInstanceContext ctx) {
    if (isManagedDisk().test(ctx.bootDiskState)) {
        return DeferredResult.completed(null);
    }
    String defaultSAName = ctx.bootDiskState.storageDescription.name;
    String defaultSAKey = ctx.storageAccountKeysForDisks.get(ctx.bootDiskState.storageDescription.authCredentialsLink);
    List<DiskService.DiskStateExpanded> diskStatesToDelete = ctx.dataDiskStates.stream().filter(disk -> disk.persistent == false).collect(Collectors.toList());
    if (ctx.bootDiskState.persistent == false) {
        diskStatesToDelete.add(ctx.bootDiskState);
    }
    OperationContext opCtx = OperationContext.getOperationContext();
    // Create DR
    DeferredResult<Void> dr = new DeferredResult<>();
    CompletableFuture.runAsync(() -> {
        diskStatesToDelete.forEach(disk -> {
            try {
                CloudStorageAccount storageAccountOfVHDBlob;
                if (disk.storageDescription == null) {
                    storageAccountOfVHDBlob = CloudStorageAccount.parse(String.format(STORAGE_CONNECTION_STRING, defaultSAName, defaultSAKey));
                } else {
                    String key = ctx.storageAccountKeysForDisks.get(disk.storageDescription.authCredentialsLink);
                    storageAccountOfVHDBlob = CloudStorageAccount.parse(String.format(STORAGE_CONNECTION_STRING, disk.storageDescription.name, key));
                }
                CloudBlobClient client = storageAccountOfVHDBlob.createCloudBlobClient();
                CloudBlobContainer container = client.getContainerReference("vhds");
                String vhdBlobName = disk.id.substring(disk.id.lastIndexOf("/") + 1);
                CloudPageBlob blob = container.getPageBlobReference(vhdBlobName);
                blob.deleteIfExists();
            } catch (URISyntaxException | InvalidKeyException | StorageException e) {
                logSevere("Unable to delete VHD file for disk [%s] because of %s ", disk.name, e.toString());
            }
        });
    }, getHost().allocateExecutor(this)).whenComplete((obj, throwable) -> {
        OperationContext.restoreOperationContext(opCtx);
        dr.complete(obj);
    });
    return dr;
}
Also used : VirtualMachine(com.microsoft.azure.management.compute.VirtualMachine) NetworkManagementClientImpl(com.microsoft.azure.management.network.implementation.NetworkManagementClientImpl) PROVISIONING_STATE_SUCCEEDED(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.PROVISIONING_STATE_SUCCEEDED) ComputeManager(com.microsoft.azure.management.compute.implementation.ComputeManager) IpAssignment(com.vmware.photon.controller.model.resources.NetworkInterfaceDescriptionService.IpAssignment) VirtualNetworkInner(com.microsoft.azure.management.network.implementation.VirtualNetworkInner) LifecycleState(com.vmware.photon.controller.model.resources.ComputeService.LifecycleState) COMPUTE_NAMESPACE(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.COMPUTE_NAMESPACE) DISK_CONTROLLER_NUMBER(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.DISK_CONTROLLER_NUMBER) StorageManagementClientImpl(com.microsoft.azure.management.storage.implementation.StorageManagementClientImpl) AzureDiagnosticSettings(com.vmware.photon.controller.model.adapters.azure.model.diagnostics.AzureDiagnosticSettings) INVALID_RESOURCE_GROUP(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.INVALID_RESOURCE_GROUP) Utils(com.vmware.xenon.common.Utils) Map(java.util.Map) StorageDescription(com.vmware.photon.controller.model.resources.StorageDescriptionService.StorageDescription) ProvisioningState(com.microsoft.azure.management.storage.ProvisioningState) OSDisk(com.microsoft.azure.management.compute.OSDisk) AZURE_STORAGE_ACCOUNT_KEY1(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.AZURE_STORAGE_ACCOUNT_KEY1) ProviderInner(com.microsoft.azure.management.resources.implementation.ProviderInner) CachingTypes(com.microsoft.azure.management.compute.CachingTypes) Indexable(com.microsoft.azure.management.resources.fluentcore.model.Indexable) StorageAccountListKeysResultInner(com.microsoft.azure.management.storage.implementation.StorageAccountListKeysResultInner) NetworkInterfaceState(com.vmware.photon.controller.model.resources.NetworkInterfaceService.NetworkInterfaceState) StatelessService(com.vmware.xenon.common.StatelessService) AZURE_STORAGE_ACCOUNT_DEFAULT_RG_NAME(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.AZURE_STORAGE_ACCOUNT_DEFAULT_RG_NAME) Disk(com.microsoft.azure.management.compute.Disk) AzureProvisioningCallback(com.vmware.photon.controller.model.adapters.azure.utils.AzureProvisioningCallback) CompletionStage(java.util.concurrent.CompletionStage) AzureSecurityGroupUtils(com.vmware.photon.controller.model.adapters.azure.utils.AzureSecurityGroupUtils) StorageAccountTypes(com.microsoft.azure.management.compute.StorageAccountTypes) SkuInner(com.microsoft.azure.management.storage.implementation.SkuInner) StorageDescriptionService(com.vmware.photon.controller.model.resources.StorageDescriptionService) DeferredResult(com.vmware.xenon.common.DeferredResult) UriUtils(com.vmware.xenon.common.UriUtils) STATUS_CODE_UNAUTHORIZED(com.vmware.xenon.common.Operation.STATUS_CODE_UNAUTHORIZED) InvalidKeyException(java.security.InvalidKeyException) StorageAccountCreateParametersInner(com.microsoft.azure.management.storage.implementation.StorageAccountCreateParametersInner) DiskCreateOptionTypes(com.microsoft.azure.management.compute.DiskCreateOptionTypes) ResourceGroupsInner(com.microsoft.azure.management.resources.implementation.ResourceGroupsInner) AzureProvisioningCallbackWithRetry(com.vmware.photon.controller.model.adapters.azure.utils.AzureProvisioningCallbackWithRetry) AZURE_DATA_DISK_CACHING(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.AZURE_DATA_DISK_CACHING) INVALID_PARAMETER(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.INVALID_PARAMETER) STORAGE_ACCOUNT_ALREADY_EXIST(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.STORAGE_ACCOUNT_ALREADY_EXIST) VirtualNetworksInner(com.microsoft.azure.management.network.implementation.VirtualNetworksInner) ImageReferenceInner(com.microsoft.azure.management.compute.implementation.ImageReferenceInner) OperationContext(com.vmware.xenon.common.OperationContext) PROVISIONING_STATE_FAILED_NO_SUBNET(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.PROVISIONING_STATE_FAILED_NO_SUBNET) ComputeDescriptionService(com.vmware.photon.controller.model.resources.ComputeDescriptionService) ArrayList(java.util.ArrayList) OSProfile(com.microsoft.azure.management.compute.OSProfile) ComputeInstanceRequest(com.vmware.photon.controller.model.adapterapi.ComputeInstanceRequest) SecurityGroupState(com.vmware.photon.controller.model.resources.SecurityGroupService.SecurityGroupState) StorageException(com.microsoft.azure.storage.StorageException) AzureAsyncCallback(com.vmware.photon.controller.model.adapters.azure.AzureAsyncCallback) AzureImageSource(com.vmware.photon.controller.model.adapters.azure.instance.AzureInstanceContext.AzureImageSource) MISSING_SUBSCRIPTION_CODE(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.MISSING_SUBSCRIPTION_CODE) StorageProfile(com.microsoft.azure.management.compute.StorageProfile) CloudBlobContainer(com.microsoft.azure.storage.blob.CloudBlobContainer) BiConsumer(java.util.function.BiConsumer) CloudPageBlob(com.microsoft.azure.storage.blob.CloudPageBlob) OperationStatusResponseInner(com.microsoft.azure.management.compute.implementation.OperationStatusResponseInner) VirtualMachineInner(com.microsoft.azure.management.compute.implementation.VirtualMachineInner) PublicIPAddressInner(com.microsoft.azure.management.network.implementation.PublicIPAddressInner) AdapterUtils(com.vmware.photon.controller.model.adapters.util.AdapterUtils) ServiceErrorResponse(com.vmware.xenon.common.ServiceErrorResponse) DataDisk(com.microsoft.azure.management.compute.DataDisk) ServiceCallback(com.microsoft.rest.ServiceCallback) STORAGE_NAMESPACE(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.STORAGE_NAMESPACE) SubnetInner(com.microsoft.azure.management.network.implementation.SubnetInner) ImageSource(com.vmware.photon.controller.model.adapters.util.instance.BaseComputeInstanceContext.ImageSource) File(java.io.File) ManagedDiskParametersInner(com.microsoft.azure.management.compute.implementation.ManagedDiskParametersInner) NETWORK_NAMESPACE(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.NETWORK_NAMESPACE) AzureUtils.getStorageAccountKeyName(com.vmware.photon.controller.model.adapters.azure.utils.AzureUtils.getStorageAccountKeyName) ApplicationTokenCredentials(com.microsoft.azure.credentials.ApplicationTokenCredentials) PhotonModelUriUtils.createInventoryUri(com.vmware.photon.controller.model.util.PhotonModelUriUtils.createInventoryUri) AuthCredentialsServiceState(com.vmware.xenon.services.common.AuthCredentialsService.AuthCredentialsServiceState) IPAllocationMethod(com.microsoft.azure.management.network.IPAllocationMethod) CloudError(com.microsoft.azure.CloudError) HardwareProfile(com.microsoft.azure.management.compute.HardwareProfile) AzureNicContext(com.vmware.photon.controller.model.adapters.azure.instance.AzureInstanceContext.AzureNicContext) URISyntaxException(java.net.URISyntaxException) NetworkProfile(com.microsoft.azure.management.compute.NetworkProfile) VirtualMachineImageResourceInner(com.microsoft.azure.management.compute.implementation.VirtualMachineImageResourceInner) AzureUriPaths(com.vmware.photon.controller.model.adapters.azure.AzureUriPaths) SubscriptionInner(com.microsoft.azure.management.resources.implementation.SubscriptionInner) AZURE_MANAGED_DISK_TYPE(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.AZURE_MANAGED_DISK_TYPE) ComputeType(com.vmware.photon.controller.model.resources.ComputeDescriptionService.ComputeDescription.ComputeType) Creatable(com.microsoft.azure.management.resources.fluentcore.model.Creatable) AzureSdkClients(com.vmware.photon.controller.model.adapters.azure.utils.AzureSdkClients) ResourceManagementClientImpl(com.microsoft.azure.management.resources.implementation.ResourceManagementClientImpl) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) URI(java.net.URI) AzureConstants(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants) AvailabilitySet(com.microsoft.azure.management.compute.AvailabilitySet) ComputeDescription(com.vmware.photon.controller.model.resources.ComputeDescriptionService.ComputeDescription) SkuName(com.microsoft.azure.management.storage.SkuName) AddressSpace(com.microsoft.azure.management.network.AddressSpace) COMPUTER_NAME(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.COMPUTER_NAME) Predicate(java.util.function.Predicate) Collection(java.util.Collection) DiskConfiguration(com.vmware.photon.controller.model.resources.ImageService.ImageState.DiskConfiguration) AvailabilitySetInner(com.microsoft.azure.management.compute.implementation.AvailabilitySetInner) DiskState(com.vmware.photon.controller.model.resources.DiskService.DiskState) CloudStorageAccount(com.microsoft.azure.storage.CloudStorageAccount) InstanceRequestType(com.vmware.photon.controller.model.adapterapi.ComputeInstanceRequest.InstanceRequestType) Collectors(java.util.stream.Collectors) Base64(java.util.Base64) List(java.util.List) VirtualHardDisk(com.microsoft.azure.management.compute.VirtualHardDisk) AzureUtils(com.vmware.photon.controller.model.adapters.azure.utils.AzureUtils) NetworkSecurityGroupsInner(com.microsoft.azure.management.network.implementation.NetworkSecurityGroupsInner) NetworkInterfacesInner(com.microsoft.azure.management.network.implementation.NetworkInterfacesInner) Optional(java.util.Optional) NetworkSecurityGroupInner(com.microsoft.azure.management.network.implementation.NetworkSecurityGroupInner) DiskService(com.vmware.photon.controller.model.resources.DiskService) UnsupportedEncodingException(java.io.UnsupportedEncodingException) Default(com.vmware.photon.controller.model.adapters.azure.utils.AzureDeferredResultServiceCallback.Default) SubResource(com.microsoft.azure.SubResource) AZURE_OSDISK_CACHING(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.AZURE_OSDISK_CACHING) ResourceGroupInner(com.microsoft.azure.management.resources.implementation.ResourceGroupInner) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) Region(com.microsoft.azure.management.resources.fluentcore.arm.Region) HashMap(java.util.HashMap) CompletableFuture(java.util.concurrent.CompletableFuture) AzureDeferredResultServiceCallbackWithRetry(com.vmware.photon.controller.model.adapters.azure.utils.AzureDeferredResultServiceCallbackWithRetry) VirtualMachineSizeTypes(com.microsoft.azure.management.compute.VirtualMachineSizeTypes) SubscriptionClientImpl(com.microsoft.azure.management.resources.implementation.SubscriptionClientImpl) Kind(com.microsoft.azure.management.storage.Kind) Level(java.util.logging.Level) FileUtils(com.vmware.xenon.common.FileUtils) AZURE_STORAGE_ACCOUNT_RG_NAME(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.AZURE_STORAGE_ACCOUNT_RG_NAME) AuthCredentialsService(com.vmware.xenon.services.common.AuthCredentialsService) ComputeState(com.vmware.photon.controller.model.resources.ComputeService.ComputeState) PublicIPAddressesInner(com.microsoft.azure.management.network.implementation.PublicIPAddressesInner) STORAGE_CONNECTION_STRING(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.STORAGE_CONNECTION_STRING) AzureDecommissionCallback(com.vmware.photon.controller.model.adapters.azure.utils.AzureDecommissionCallback) ComputeManagementClientImpl(com.microsoft.azure.management.compute.implementation.ComputeManagementClientImpl) NetworkInterfaceIPConfigurationInner(com.microsoft.azure.management.network.implementation.NetworkInterfaceIPConfigurationInner) EncryptionUtils(com.vmware.photon.controller.model.security.util.EncryptionUtils) CloudBlobClient(com.microsoft.azure.storage.blob.CloudBlobClient) Operation(com.vmware.xenon.common.Operation) AvailabilitySetSkuTypes(com.microsoft.azure.management.compute.AvailabilitySetSkuTypes) StorageAccountKey(com.microsoft.azure.management.storage.StorageAccountKey) BaseAdapterStage(com.vmware.photon.controller.model.adapters.util.BaseAdapterContext.BaseAdapterStage) TimeUnit(java.util.concurrent.TimeUnit) Consumer(java.util.function.Consumer) NetworkInterfaceReferenceInner(com.microsoft.azure.management.compute.implementation.NetworkInterfaceReferenceInner) AzureDeferredResultServiceCallback(com.vmware.photon.controller.model.adapters.azure.utils.AzureDeferredResultServiceCallback) AZURE_STORAGE_ACCOUNT_NAME(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.AZURE_STORAGE_ACCOUNT_NAME) CloudException(com.microsoft.azure.CloudException) ImageState(com.vmware.photon.controller.model.resources.ImageService.ImageState) StorageAccountInner(com.microsoft.azure.management.storage.implementation.StorageAccountInner) NetworkInterfaceDescription(com.vmware.photon.controller.model.resources.NetworkInterfaceDescriptionService.NetworkInterfaceDescription) NetworkInterfaceInner(com.microsoft.azure.management.network.implementation.NetworkInterfaceInner) CLOUD_CONFIG_DEFAULT_FILE_INDEX(com.vmware.photon.controller.model.constants.PhotonModelConstants.CLOUD_CONFIG_DEFAULT_FILE_INDEX) Collections(java.util.Collections) OperationJoin(com.vmware.xenon.common.OperationJoin) RESOURCE_GROUP_NAME(com.vmware.photon.controller.model.ComputeProperties.RESOURCE_GROUP_NAME) PROVIDER_REGISTRED_STATE(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.PROVIDER_REGISTRED_STATE) OperationContext(com.vmware.xenon.common.OperationContext) CloudBlobClient(com.microsoft.azure.storage.blob.CloudBlobClient) CloudStorageAccount(com.microsoft.azure.storage.CloudStorageAccount) CloudBlobContainer(com.microsoft.azure.storage.blob.CloudBlobContainer) DeferredResult(com.vmware.xenon.common.DeferredResult) CloudPageBlob(com.microsoft.azure.storage.blob.CloudPageBlob)

Example 54 with CloudBlobClient

use of com.microsoft.azure.storage.blob.CloudBlobClient in project photon-model by vmware.

the class TestAzureDeleteTask method testProvisionWithPersistentUnmanagedDisk.

/**
 * Creates a Azure unmanaged instance with persist options for disks. Checks if VHD blob cleanup happens according to the persist flag.
 */
@Test
public void testProvisionWithPersistentUnmanagedDisk() throws Throwable {
    BaseComputeInstanceContext.ImageSource imageSource = createImageSource(getHost(), this.endpointState, IMAGE_REFERENCE);
    // Create a Azure VM compute resource with 2 additional disks.
    int numberOfAdditionalDisks = 3;
    AzureTestUtil.VMResourceSpec vmResourceSpec = new AzureTestUtil.VMResourceSpec(getHost(), this.computeHost, this.endpointState, azureVMName).withImageSource(imageSource).withNicSpecs(DEFAULT_NIC_SPEC).withNumberOfAdditionalDisks(numberOfAdditionalDisks).withPersistentDisks(AzureTestUtil.VMResourceSpec.PersistentDisks.SOME).withExistingStorageAccount(storageAccountName, resourceGroupName).withManagedDisk(false);
    // create Azure VM compute resource.
    this.vmState = createVMResourceFromSpec(vmResourceSpec);
    kickOffProvisionTask();
    List<DiskService.DiskState> diskStates = this.vmState.diskLinks.stream().map(diskLink -> getHost().getServiceState(null, DiskService.DiskState.class, UriUtils.buildUri(getHost(), diskLink))).collect(Collectors.toList());
    int computeStatesToRemain = 1;
    deleteVMs(getHost(), this.vmState.documentSelfLink, this.isMock, computeStatesToRemain);
    // assertions
    if (!this.isMock) {
        Azure az = getAzureSdkClients().getAzureClient();
        String key = az.storageAccounts().getByResourceGroup(resourceGroupName, storageAccountName).getKeys().get(0).value();
        CloudStorageAccount cloudStorageAccount = CloudStorageAccount.parse(String.format(STORAGE_CONNECTION_STRING, storageAccountName, key));
        CloudBlobClient client = cloudStorageAccount.createCloudBlobClient();
        CloudBlobContainer container = client.getContainerReference("vhds");
        for (DiskService.DiskState diskState : diskStates) {
            String vhdBlobName = diskState.id.substring(diskState.id.lastIndexOf("/") + 1);
            CloudPageBlob blob = container.getPageBlobReference(vhdBlobName);
            assertEquals("Disk VHD blob should persist", diskState.persistent, blob.exists());
        }
    }
}
Also used : ProvisionComputeTaskService(com.vmware.photon.controller.model.tasks.ProvisionComputeTaskService) AzureTestUtil.deleteVMs(com.vmware.photon.controller.model.adapters.azure.instance.AzureTestUtil.deleteVMs) PhotonModelMetricServices(com.vmware.photon.controller.model.PhotonModelMetricServices) AZURE_RESOURCE_GROUP_LOCATION(com.vmware.photon.controller.model.adapters.azure.instance.AzureTestUtil.AZURE_RESOURCE_GROUP_LOCATION) AzureBaseTest(com.vmware.photon.controller.model.adapters.azure.base.AzureBaseTest) Azure(com.microsoft.azure.management.Azure) CloudBlobContainer(com.microsoft.azure.storage.blob.CloudBlobContainer) After(org.junit.After) CloudPageBlob(com.microsoft.azure.storage.blob.CloudPageBlob) IMAGE_REFERENCE(com.vmware.photon.controller.model.adapters.azure.instance.AzureTestUtil.IMAGE_REFERENCE) STORAGE_CONNECTION_STRING(com.vmware.photon.controller.model.adapters.azure.constants.AzureConstants.STORAGE_CONNECTION_STRING) Before(org.junit.Before) DEFAULT_NIC_SPEC(com.vmware.photon.controller.model.adapters.azure.instance.AzureTestUtil.DEFAULT_NIC_SPEC) CloudBlobClient(com.microsoft.azure.storage.blob.CloudBlobClient) TestUtils(com.vmware.photon.controller.model.tasks.TestUtils) Test(org.junit.Test) AzureTestUtil.createVMResourceFromSpec(com.vmware.photon.controller.model.adapters.azure.instance.AzureTestUtil.createVMResourceFromSpec) CloudStorageAccount(com.microsoft.azure.storage.CloudStorageAccount) Collectors(java.util.stream.Collectors) List(java.util.List) BaseComputeInstanceContext(com.vmware.photon.controller.model.adapters.util.instance.BaseComputeInstanceContext) AzureTestUtil.createImageSource(com.vmware.photon.controller.model.adapters.azure.instance.AzureTestUtil.createImageSource) AzureTestUtil.generateName(com.vmware.photon.controller.model.adapters.azure.instance.AzureTestUtil.generateName) UriUtils(com.vmware.xenon.common.UriUtils) ComputeService(com.vmware.photon.controller.model.resources.ComputeService) DiskService(com.vmware.photon.controller.model.resources.DiskService) Assert.assertEquals(org.junit.Assert.assertEquals) SdkContext(com.microsoft.azure.management.resources.fluentcore.utils.SdkContext) Azure(com.microsoft.azure.management.Azure) CloudBlobClient(com.microsoft.azure.storage.blob.CloudBlobClient) BaseComputeInstanceContext(com.vmware.photon.controller.model.adapters.util.instance.BaseComputeInstanceContext) CloudStorageAccount(com.microsoft.azure.storage.CloudStorageAccount) DiskService(com.vmware.photon.controller.model.resources.DiskService) CloudBlobContainer(com.microsoft.azure.storage.blob.CloudBlobContainer) CloudPageBlob(com.microsoft.azure.storage.blob.CloudPageBlob) AzureBaseTest(com.vmware.photon.controller.model.adapters.azure.base.AzureBaseTest) Test(org.junit.Test)

Example 55 with CloudBlobClient

use of com.microsoft.azure.storage.blob.CloudBlobClient in project photon-model by vmware.

the class AzureComputeHostStorageStatsGatherer method getBlobUsedBytesAsync.

private void getBlobUsedBytesAsync(AzureStorageStatsDataHolder statsData, StorageMetricsStages next) {
    Runnable getBlobsAsync = () -> {
        String metricName = PhotonModelConstants.STORAGE_USED_BYTES;
        List<ServiceStats.ServiceStat> statDatapoints = new ArrayList<>();
        AtomicInteger accountsCount = new AtomicInteger(statsData.storageAccounts.size());
        final List<Throwable> exs = new ArrayList<>();
        for (Map.Entry<String, StorageAccount> account : statsData.storageAccounts.entrySet()) {
            String resourceGroupName = getResourceGroupName(account.getValue().id);
            statsData.azureClients.getAzureClient().storageAccounts().inner().listKeysAsync(resourceGroupName, account.getValue().name, new AzureAsyncCallback<StorageAccountListKeysResultInner>() {

                @Override
                public void onError(Throwable e) {
                    handleError(statsData, e);
                }

                @Override
                public void onSuccess(StorageAccountListKeysResultInner result) {
                    logFine(() -> String.format("Retrieved the storage account keys for" + " storage account [%s].", account.getValue().name));
                    String storageConnectionString = String.format(STORAGE_CONNECTION_STRING, account.getValue().name, result.keys().get(0).value());
                    try {
                        CloudStorageAccount storageAccount = getAzureStorageClient(storageConnectionString);
                        CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
                        ResultContinuation nextContainerResults = null;
                        do {
                            ResultSegment<CloudBlobContainer> contSegment = blobClient.listContainersSegmented(null, ContainerListingDetails.NONE, QUERY_RESULT_LIMIT, nextContainerResults, null, null);
                            nextContainerResults = contSegment.getContinuationToken();
                            for (CloudBlobContainer container : contSegment.getResults()) {
                                ResultContinuation nextBlobResults = null;
                                do {
                                    ResultSegment<ListBlobItem> blobsSegment = container.listBlobsSegmented(null, false, EnumSet.noneOf(BlobListingDetails.class), QUERY_RESULT_LIMIT, nextBlobResults, null, null);
                                    nextBlobResults = blobsSegment.getContinuationToken();
                                    for (ListBlobItem blobItem : blobsSegment.getResults()) {
                                        if (blobItem instanceof CloudPageBlob) {
                                            CloudPageBlob pageBlob = (CloudPageBlob) blobItem;
                                            // TODO https://jira-hzn.eng.vmware.com/browse/VSYM-3445
                                            try {
                                                CloudBlob blobSnapshot = pageBlob.createSnapshot();
                                                statsData.snapshots.add(blobSnapshot);
                                                CloudPageBlob pageBlobSnapshot = (CloudPageBlob) blobSnapshot;
                                                ArrayList<PageRange> pages = pageBlobSnapshot.downloadPageRanges();
                                                // https://jira-hzn.eng.vmware.com/browse/VSYM-3355
                                                for (PageRange pageRange : pages) {
                                                    statsData.utilizedBytes += pageRange.getEndOffset() - pageRange.getStartOffset();
                                                }
                                            } catch (StorageException e) {
                                                logWarning(() -> String.format("Error getting blob size: [%s]", e.getMessage()));
                                            }
                                        }
                                    }
                                } while (nextBlobResults != null);
                            }
                        } while (nextContainerResults != null);
                    } catch (Exception e) {
                        logWarning(() -> String.format("Exception while getting blob used bytes: %s", Utils.toString(e)));
                        exs.add(e);
                    } finally {
                        // in the Azure account
                        if (statsData.snapshots.size() > 0) {
                            synchronized (statsData.snapshots) {
                                Iterator<CloudBlob> snapshotIterator = statsData.snapshots.iterator();
                                while (snapshotIterator.hasNext()) {
                                    try {
                                        CloudBlob snapshot = snapshotIterator.next();
                                        snapshot.deleteIfExists();
                                        snapshotIterator.remove();
                                    } catch (StorageException e) {
                                        // Best effort to delete all the snapshots
                                        logWarning(() -> String.format("Exception while deleting snapshot: %s", Utils.toString(e)));
                                    }
                                }
                            }
                        }
                    }
                    // if all storage accounts were processed, create ServiceStat and finish
                    if (accountsCount.decrementAndGet() == 0) {
                        if (!exs.isEmpty()) {
                            handleError(statsData, exs.iterator().next());
                            return;
                        }
                        if (statsData.utilizedBytes != 0) {
                            ServiceStats.ServiceStat stat = new ServiceStats.ServiceStat();
                            stat.latestValue = statsData.utilizedBytes;
                            stat.sourceTimeMicrosUtc = TimeUnit.MILLISECONDS.toMicros(Utils.getNowMicrosUtc());
                            stat.unit = PhotonModelConstants.getUnitForMetric(metricName);
                            statDatapoints.add(stat);
                        }
                        statsData.statsResponse.statValues.put(metricName, statDatapoints);
                        if (statsData.statsResponse.statValues.size() == 1) {
                            statsData.statsResponse.computeLink = statsData.computeHostDesc.documentSelfLink;
                        }
                        statsData.stage = next;
                        handleStorageMetricDiscovery(statsData);
                    }
                }
            });
        }
    };
    PhotonModelUtils.runInExecutor(this.executorService, getBlobsAsync, throwable -> handleError(statsData, throwable));
}
Also used : BlobListingDetails(com.microsoft.azure.storage.blob.BlobListingDetails) ListBlobItem(com.microsoft.azure.storage.blob.ListBlobItem) StorageAccountListKeysResultInner(com.microsoft.azure.management.storage.implementation.StorageAccountListKeysResultInner) CloudBlob(com.microsoft.azure.storage.blob.CloudBlob) ServiceStats(com.vmware.xenon.common.ServiceStats) AzureAsyncCallback(com.vmware.photon.controller.model.adapters.azure.AzureAsyncCallback) List(java.util.List) ArrayList(java.util.ArrayList) StorageAccountResultList(com.vmware.photon.controller.model.adapters.azure.model.storage.StorageAccountResultList) CloudBlobClient(com.microsoft.azure.storage.blob.CloudBlobClient) ResultContinuation(com.microsoft.azure.storage.ResultContinuation) CloudStorageAccount(com.microsoft.azure.storage.CloudStorageAccount) StorageException(com.microsoft.azure.storage.StorageException) PageRange(com.microsoft.azure.storage.blob.PageRange) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) CloudBlobContainer(com.microsoft.azure.storage.blob.CloudBlobContainer) StorageException(com.microsoft.azure.storage.StorageException) CloudPageBlob(com.microsoft.azure.storage.blob.CloudPageBlob)

Aggregations

CloudBlobClient (com.microsoft.azure.storage.blob.CloudBlobClient)74 CloudBlobContainer (com.microsoft.azure.storage.blob.CloudBlobContainer)42 CloudStorageAccount (com.microsoft.azure.storage.CloudStorageAccount)19 StorageException (com.microsoft.azure.storage.StorageException)19 URISyntaxException (java.net.URISyntaxException)17 CloudBlockBlob (com.microsoft.azure.storage.blob.CloudBlockBlob)16 ListBlobItem (com.microsoft.azure.storage.blob.ListBlobItem)10 URI (java.net.URI)10 Supplier (java.util.function.Supplier)9 CloudBlob (com.microsoft.azure.storage.blob.CloudBlob)8 InvalidKeyException (java.security.InvalidKeyException)8 Test (org.junit.Test)8 ArrayList (java.util.ArrayList)6 ParameterizedMessage (org.apache.logging.log4j.message.ParameterizedMessage)6 TypeLiteral (com.google.inject.TypeLiteral)5 RetryExponentialRetry (com.microsoft.azure.storage.RetryExponentialRetry)5 StorageCredentials (com.microsoft.azure.storage.StorageCredentials)5 BlobListingDetails (com.microsoft.azure.storage.blob.BlobListingDetails)5 IOException (java.io.IOException)5 BlobProperties (com.microsoft.azure.storage.blob.BlobProperties)4