Search in sources :

Example 1 with AzureSparkServerlessAccount

use of com.microsoft.azure.hdinsight.sdk.common.azure.serverless.AzureSparkServerlessAccount in project azure-tools-for-java by Microsoft.

the class SynapseCosmosSparkPool method getConfigurationInfo.

@Override
public void getConfigurationInfo() throws IOException {
    if (!isConfigInfoAvailable()) {
        // Extract Subscription ID from ADLA resource ID
        // Sample adlaResouceId: /subscriptions/a00b00a0-00a0-0000-b000-a000b0a00000/resourceGroups/testRG/providers/Microsoft.DataLakeAnalytics/accounts/testAccount
        Matcher matcher = ADLA_RESOURCE_ID_PATTERN.matcher(getAdlaResourceId());
        if (!matcher.matches()) {
            String errorMsg = String.format("ADLA resource ID doesn't match with pattern. AdlaResourceId: %s. Pattern: %s", getAdlaResourceId(), ADLA_RESOURCE_ID_PATTERN);
            throw new IOException(errorMsg);
        }
        String subscriptionId = matcher.group("sid");
        // Get SubscriptionDetail from subscription ID
        AzureManager azureManager = AuthMethodManager.getInstance().getAzureManager();
        SubscriptionManager subscriptionManager = azureManager.getSubscriptionManager();
        SubscriptionDetail subscription = subscriptionManager.getSubscriptionIdToSubscriptionDetailsMap().getOrDefault(subscriptionId, null);
        if (subscription == null) {
            throw new IOException("User has no permission to access subscription " + subscriptionId + ".");
        }
        // Get ADLA account details through Azure REST API
        AzureHttpObservable managementHttp = new AzureManagementHttpObservable(subscription, ApiVersion.VERSION);
        String resourceManagerEndpoint = CommonSettings.getAdEnvironment().resourceManagerEndpoint();
        URI accountDetailUri = URI.create(resourceManagerEndpoint).resolve(getAdlaResourceId());
        AzureSparkServerlessAccount azureSparkServerlessAccount = managementHttp.withUuidUserAgent().get(accountDetailUri.toString(), null, null, DataLakeAnalyticsAccount.class).doOnError(err -> log().warn("Error getting ADLA account details with error: " + err.getMessage())).map(dataLakeAnalyticsAccount -> new AzureSparkServerlessAccount(subscription, URI.create("https://" + dataLakeAnalyticsAccount.endpoint()), dataLakeAnalyticsAccount.name()).setDetailResponse(dataLakeAnalyticsAccount)).subscribeOn(Schedulers.io()).toBlocking().singleOrDefault(null);
        // Get default storage account info from ADLA account
        String storageRootPath = azureSparkServerlessAccount.getStorageRootPath();
        IHDIStorageAccount adlsGen1StorageAccount = storageRootPath == null ? null : new AzureSparkCosmosCluster.StorageAccount(azureSparkServerlessAccount.getDetailResponse().defaultDataLakeStoreAccount(), storageRootPath, azureSparkServerlessAccount.getSubscription().getSubscriptionId());
        synchronized (this) {
            if (!isConfigInfoAvailable()) {
                this.http = managementHttp;
                this.storageAccount = adlsGen1StorageAccount;
                isConfigInfoAvailable = true;
            }
        }
    }
}
Also used : AuthMethodManager(com.microsoft.azuretools.authmanage.AuthMethodManager) AzureSparkServerlessAccount(com.microsoft.azure.hdinsight.sdk.common.azure.serverless.AzureSparkServerlessAccount) SparkSubmitStorageTypeOptionsForCluster(com.microsoft.azure.hdinsight.spark.common.SparkSubmitStorageTypeOptionsForCluster) ArcadiaWorkSpace(com.microsoft.azure.projectarcadia.common.ArcadiaWorkSpace) SubscriptionManager(com.microsoft.azuretools.authmanage.SubscriptionManager) Nullable(com.microsoft.azuretools.azurecommons.helpers.Nullable) SparkSubmitStorageType(com.microsoft.azure.hdinsight.spark.common.SparkSubmitStorageType) AzureManager(com.microsoft.azuretools.sdkmanage.AzureManager) IOException(java.io.IOException) ArcadiaSparkCompute(com.microsoft.azure.projectarcadia.common.ArcadiaSparkCompute) DataLakeAnalyticsAccount(com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.DataLakeAnalyticsAccount) BigDataPoolResourceInfo(com.microsoft.azure.hdinsight.sdk.rest.azure.synapse.models.BigDataPoolResourceInfo) Matcher(java.util.regex.Matcher) Schedulers(rx.schedulers.Schedulers) AzureSparkCosmosCluster(com.microsoft.azure.hdinsight.sdk.common.azure.serverless.AzureSparkCosmosCluster) SubscriptionDetail(com.microsoft.azuretools.authmanage.models.SubscriptionDetail) CommonSettings(com.microsoft.azuretools.authmanage.CommonSettings) URI(java.net.URI) Pattern(java.util.regex.Pattern) AzureManagementHttpObservable(com.microsoft.azure.hdinsight.sdk.common.AzureManagementHttpObservable) IHDIStorageAccount(com.microsoft.azure.hdinsight.sdk.storage.IHDIStorageAccount) AzureHttpObservable(com.microsoft.azure.hdinsight.sdk.common.AzureHttpObservable) ApiVersion(com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.ApiVersion) ExceptionUtils(org.apache.commons.lang3.exception.ExceptionUtils) IHDIStorageAccount(com.microsoft.azure.hdinsight.sdk.storage.IHDIStorageAccount) AzureManager(com.microsoft.azuretools.sdkmanage.AzureManager) Matcher(java.util.regex.Matcher) IOException(java.io.IOException) SubscriptionManager(com.microsoft.azuretools.authmanage.SubscriptionManager) AzureSparkCosmosCluster(com.microsoft.azure.hdinsight.sdk.common.azure.serverless.AzureSparkCosmosCluster) URI(java.net.URI) AzureHttpObservable(com.microsoft.azure.hdinsight.sdk.common.AzureHttpObservable) AzureSparkServerlessAccount(com.microsoft.azure.hdinsight.sdk.common.azure.serverless.AzureSparkServerlessAccount) SubscriptionDetail(com.microsoft.azuretools.authmanage.models.SubscriptionDetail) DataLakeAnalyticsAccount(com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.DataLakeAnalyticsAccount) AzureManagementHttpObservable(com.microsoft.azure.hdinsight.sdk.common.AzureManagementHttpObservable)

Aggregations

AzureHttpObservable (com.microsoft.azure.hdinsight.sdk.common.AzureHttpObservable)1 AzureManagementHttpObservable (com.microsoft.azure.hdinsight.sdk.common.AzureManagementHttpObservable)1 AzureSparkCosmosCluster (com.microsoft.azure.hdinsight.sdk.common.azure.serverless.AzureSparkCosmosCluster)1 AzureSparkServerlessAccount (com.microsoft.azure.hdinsight.sdk.common.azure.serverless.AzureSparkServerlessAccount)1 ApiVersion (com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.ApiVersion)1 DataLakeAnalyticsAccount (com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.DataLakeAnalyticsAccount)1 BigDataPoolResourceInfo (com.microsoft.azure.hdinsight.sdk.rest.azure.synapse.models.BigDataPoolResourceInfo)1 IHDIStorageAccount (com.microsoft.azure.hdinsight.sdk.storage.IHDIStorageAccount)1 SparkSubmitStorageType (com.microsoft.azure.hdinsight.spark.common.SparkSubmitStorageType)1 SparkSubmitStorageTypeOptionsForCluster (com.microsoft.azure.hdinsight.spark.common.SparkSubmitStorageTypeOptionsForCluster)1 ArcadiaSparkCompute (com.microsoft.azure.projectarcadia.common.ArcadiaSparkCompute)1 ArcadiaWorkSpace (com.microsoft.azure.projectarcadia.common.ArcadiaWorkSpace)1 AuthMethodManager (com.microsoft.azuretools.authmanage.AuthMethodManager)1 CommonSettings (com.microsoft.azuretools.authmanage.CommonSettings)1 SubscriptionManager (com.microsoft.azuretools.authmanage.SubscriptionManager)1 SubscriptionDetail (com.microsoft.azuretools.authmanage.models.SubscriptionDetail)1 Nullable (com.microsoft.azuretools.azurecommons.helpers.Nullable)1 AzureManager (com.microsoft.azuretools.sdkmanage.AzureManager)1 IOException (java.io.IOException)1 URI (java.net.URI)1