Search in sources :

Example 1 with DataLakeAnalyticsAccount

use of com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.DataLakeAnalyticsAccount in project azure-tools-for-java by Microsoft.

the class AzureSparkServerlessAccount method getStorageRootPath.

@Nullable
public String getStorageRootPath() {
    String storageRootPath = null;
    DataLakeAnalyticsAccount accountDetail = this.getDetailResponse();
    if (accountDetail != null) {
        // find default storage account name and suffix
        String defaultStorageAccountName = accountDetail.defaultDataLakeStoreAccount();
        storageRootPath = accountDetail.dataLakeStoreAccounts().stream().filter(info -> info.name().equals(defaultStorageAccountName)).findFirst().map(DataLakeStoreAccountInformation::suffix).map(suffix -> String.format("adl://%s.%s/", defaultStorageAccountName, suffix)).orElse(null);
    }
    return storageRootPath;
}
Also used : AuthMethodManager(com.microsoft.azuretools.authmanage.AuthMethodManager) JobInfoListResult(com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.job.models.JobInfoListResult) SparkSubmitStorageTypeOptionsForCluster(com.microsoft.azure.hdinsight.spark.common.SparkSubmitStorageTypeOptionsForCluster) NotNull(com.microsoft.azuretools.azurecommons.helpers.NotNull) ODataParam(com.microsoft.azure.hdinsight.sdk.common.ODataParam) DataLakeAnalyticsAccountBasic(com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.DataLakeAnalyticsAccountBasic) SparkSubmitStorageType(com.microsoft.azure.hdinsight.spark.common.SparkSubmitStorageType) AzureManager(com.microsoft.azuretools.sdkmanage.AzureManager) ClusterContainer(com.microsoft.azure.hdinsight.sdk.cluster.ClusterContainer) AzureDataLakeHttpObservable(com.microsoft.azure.hdinsight.sdk.common.AzureDataLakeHttpObservable) StringUtils(org.apache.commons.lang3.StringUtils) ILogger(com.microsoft.azure.hdinsight.common.logger.ILogger) DataLakeAnalyticsAccount(com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.DataLakeAnalyticsAccount) Observable(rx.Observable) ImmutableList(com.google.common.collect.ImmutableList) SubscriptionDetail(com.microsoft.azuretools.authmanage.models.SubscriptionDetail) URI(java.net.URI) JobState(com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.job.models.JobState) IClusterDetail(com.microsoft.azure.hdinsight.sdk.cluster.IClusterDetail) ImmutableSortedSet(com.google.common.collect.ImmutableSortedSet) Nullable(com.microsoft.azuretools.azurecommons.helpers.Nullable) VERSION(com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.job.models.ApiVersion.VERSION) HttpResponse(com.microsoft.azure.hdinsight.sdk.common.HttpResponse) StringEntity(org.apache.http.entity.StringEntity) StandardCharsets(java.nio.charset.StandardCharsets) List(java.util.List) DataLakeStoreAccountInformation(com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.DataLakeStoreAccountInformation) HttpGet(org.apache.http.client.methods.HttpGet) com.microsoft.azure.hdinsight.sdk.rest.azure.serverless.spark.models(com.microsoft.azure.hdinsight.sdk.rest.azure.serverless.spark.models) NameValuePair(org.apache.http.NameValuePair) Collections(java.util.Collections) AzureHttpObservable(com.microsoft.azure.hdinsight.sdk.common.AzureHttpObservable) DataLakeAnalyticsAccount(com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.DataLakeAnalyticsAccount) Nullable(com.microsoft.azuretools.azurecommons.helpers.Nullable)

Example 2 with DataLakeAnalyticsAccount

use of com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.DataLakeAnalyticsAccount in project azure-tools-for-java by Microsoft.

the class SynapseCosmosSparkPool method getConfigurationInfo.

@Override
public void getConfigurationInfo() throws IOException {
    if (!isConfigInfoAvailable()) {
        // Extract Subscription ID from ADLA resource ID
        // Sample adlaResouceId: /subscriptions/a00b00a0-00a0-0000-b000-a000b0a00000/resourceGroups/testRG/providers/Microsoft.DataLakeAnalytics/accounts/testAccount
        Matcher matcher = ADLA_RESOURCE_ID_PATTERN.matcher(getAdlaResourceId());
        if (!matcher.matches()) {
            String errorMsg = String.format("ADLA resource ID doesn't match with pattern. AdlaResourceId: %s. Pattern: %s", getAdlaResourceId(), ADLA_RESOURCE_ID_PATTERN);
            throw new IOException(errorMsg);
        }
        String subscriptionId = matcher.group("sid");
        // Get SubscriptionDetail from subscription ID
        AzureManager azureManager = AuthMethodManager.getInstance().getAzureManager();
        SubscriptionManager subscriptionManager = azureManager.getSubscriptionManager();
        SubscriptionDetail subscription = subscriptionManager.getSubscriptionIdToSubscriptionDetailsMap().getOrDefault(subscriptionId, null);
        if (subscription == null) {
            throw new IOException("User has no permission to access subscription " + subscriptionId + ".");
        }
        // Get ADLA account details through Azure REST API
        AzureHttpObservable managementHttp = new AzureManagementHttpObservable(subscription, ApiVersion.VERSION);
        String resourceManagerEndpoint = CommonSettings.getAdEnvironment().resourceManagerEndpoint();
        URI accountDetailUri = URI.create(resourceManagerEndpoint).resolve(getAdlaResourceId());
        AzureSparkServerlessAccount azureSparkServerlessAccount = managementHttp.withUuidUserAgent().get(accountDetailUri.toString(), null, null, DataLakeAnalyticsAccount.class).doOnError(err -> log().warn("Error getting ADLA account details with error: " + err.getMessage())).map(dataLakeAnalyticsAccount -> new AzureSparkServerlessAccount(subscription, URI.create("https://" + dataLakeAnalyticsAccount.endpoint()), dataLakeAnalyticsAccount.name()).setDetailResponse(dataLakeAnalyticsAccount)).subscribeOn(Schedulers.io()).toBlocking().singleOrDefault(null);
        // Get default storage account info from ADLA account
        String storageRootPath = azureSparkServerlessAccount.getStorageRootPath();
        IHDIStorageAccount adlsGen1StorageAccount = storageRootPath == null ? null : new AzureSparkCosmosCluster.StorageAccount(azureSparkServerlessAccount.getDetailResponse().defaultDataLakeStoreAccount(), storageRootPath, azureSparkServerlessAccount.getSubscription().getSubscriptionId());
        synchronized (this) {
            if (!isConfigInfoAvailable()) {
                this.http = managementHttp;
                this.storageAccount = adlsGen1StorageAccount;
                isConfigInfoAvailable = true;
            }
        }
    }
}
Also used : AuthMethodManager(com.microsoft.azuretools.authmanage.AuthMethodManager) AzureSparkServerlessAccount(com.microsoft.azure.hdinsight.sdk.common.azure.serverless.AzureSparkServerlessAccount) SparkSubmitStorageTypeOptionsForCluster(com.microsoft.azure.hdinsight.spark.common.SparkSubmitStorageTypeOptionsForCluster) ArcadiaWorkSpace(com.microsoft.azure.projectarcadia.common.ArcadiaWorkSpace) SubscriptionManager(com.microsoft.azuretools.authmanage.SubscriptionManager) Nullable(com.microsoft.azuretools.azurecommons.helpers.Nullable) SparkSubmitStorageType(com.microsoft.azure.hdinsight.spark.common.SparkSubmitStorageType) AzureManager(com.microsoft.azuretools.sdkmanage.AzureManager) IOException(java.io.IOException) ArcadiaSparkCompute(com.microsoft.azure.projectarcadia.common.ArcadiaSparkCompute) DataLakeAnalyticsAccount(com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.DataLakeAnalyticsAccount) BigDataPoolResourceInfo(com.microsoft.azure.hdinsight.sdk.rest.azure.synapse.models.BigDataPoolResourceInfo) Matcher(java.util.regex.Matcher) Schedulers(rx.schedulers.Schedulers) AzureSparkCosmosCluster(com.microsoft.azure.hdinsight.sdk.common.azure.serverless.AzureSparkCosmosCluster) SubscriptionDetail(com.microsoft.azuretools.authmanage.models.SubscriptionDetail) CommonSettings(com.microsoft.azuretools.authmanage.CommonSettings) URI(java.net.URI) Pattern(java.util.regex.Pattern) AzureManagementHttpObservable(com.microsoft.azure.hdinsight.sdk.common.AzureManagementHttpObservable) IHDIStorageAccount(com.microsoft.azure.hdinsight.sdk.storage.IHDIStorageAccount) AzureHttpObservable(com.microsoft.azure.hdinsight.sdk.common.AzureHttpObservable) ApiVersion(com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.ApiVersion) ExceptionUtils(org.apache.commons.lang3.exception.ExceptionUtils) IHDIStorageAccount(com.microsoft.azure.hdinsight.sdk.storage.IHDIStorageAccount) AzureManager(com.microsoft.azuretools.sdkmanage.AzureManager) Matcher(java.util.regex.Matcher) IOException(java.io.IOException) SubscriptionManager(com.microsoft.azuretools.authmanage.SubscriptionManager) AzureSparkCosmosCluster(com.microsoft.azure.hdinsight.sdk.common.azure.serverless.AzureSparkCosmosCluster) URI(java.net.URI) AzureHttpObservable(com.microsoft.azure.hdinsight.sdk.common.AzureHttpObservable) AzureSparkServerlessAccount(com.microsoft.azure.hdinsight.sdk.common.azure.serverless.AzureSparkServerlessAccount) SubscriptionDetail(com.microsoft.azuretools.authmanage.models.SubscriptionDetail) DataLakeAnalyticsAccount(com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.DataLakeAnalyticsAccount) AzureManagementHttpObservable(com.microsoft.azure.hdinsight.sdk.common.AzureManagementHttpObservable)

Aggregations

AzureHttpObservable (com.microsoft.azure.hdinsight.sdk.common.AzureHttpObservable)2 DataLakeAnalyticsAccount (com.microsoft.azure.hdinsight.sdk.rest.azure.datalake.analytics.accounts.models.DataLakeAnalyticsAccount)2 SparkSubmitStorageType (com.microsoft.azure.hdinsight.spark.common.SparkSubmitStorageType)2 SparkSubmitStorageTypeOptionsForCluster (com.microsoft.azure.hdinsight.spark.common.SparkSubmitStorageTypeOptionsForCluster)2 AuthMethodManager (com.microsoft.azuretools.authmanage.AuthMethodManager)2 SubscriptionDetail (com.microsoft.azuretools.authmanage.models.SubscriptionDetail)2 Nullable (com.microsoft.azuretools.azurecommons.helpers.Nullable)2 AzureManager (com.microsoft.azuretools.sdkmanage.AzureManager)2 URI (java.net.URI)2 ImmutableList (com.google.common.collect.ImmutableList)1 ImmutableSortedSet (com.google.common.collect.ImmutableSortedSet)1 ILogger (com.microsoft.azure.hdinsight.common.logger.ILogger)1 ClusterContainer (com.microsoft.azure.hdinsight.sdk.cluster.ClusterContainer)1 IClusterDetail (com.microsoft.azure.hdinsight.sdk.cluster.IClusterDetail)1 AzureDataLakeHttpObservable (com.microsoft.azure.hdinsight.sdk.common.AzureDataLakeHttpObservable)1 AzureManagementHttpObservable (com.microsoft.azure.hdinsight.sdk.common.AzureManagementHttpObservable)1 HttpResponse (com.microsoft.azure.hdinsight.sdk.common.HttpResponse)1 ODataParam (com.microsoft.azure.hdinsight.sdk.common.ODataParam)1 AzureSparkCosmosCluster (com.microsoft.azure.hdinsight.sdk.common.azure.serverless.AzureSparkCosmosCluster)1 AzureSparkServerlessAccount (com.microsoft.azure.hdinsight.sdk.common.azure.serverless.AzureSparkServerlessAccount)1