Search in sources :

Example 1 with FileSystemType

use of com.sequenceiq.common.model.FileSystemType in project cloudbreak by hortonworks.

the class CloudStorageConverter method requestToFileSystem.

public FileSystem requestToFileSystem(CloudStorageBase cloudStorageRequest) {
    FileSystem fileSystem = new FileSystem();
    fileSystem.setName(nameGenerator.generateName(FILESYSTEM));
    FileSystemType fileSystemType = fileSystemResolver.determineFileSystemType(cloudStorageRequest);
    fileSystem.setType(fileSystemType);
    CloudStorage cloudStorage = new CloudStorage();
    String s3GuardDynamoTableName = getS3GuardDynamoTableName(cloudStorageRequest);
    cloudStorage.setS3GuardDynamoTableName(s3GuardDynamoTableName);
    List<StorageLocation> storageLocations = cloudStorageRequest.getLocations().stream().map(this::storageLocationRequestToStorageLocation).collect(Collectors.toList());
    cloudStorage.setLocations(storageLocations);
    if (cloudStorageRequest.getIdentities() != null) {
        List<CloudIdentity> cloudIdentities = cloudStorageRequest.getIdentities().stream().map(this::identityRequestToCloudIdentity).collect(Collectors.toList());
        cloudStorage.setCloudIdentities(cloudIdentities);
    }
    cloudStorage.setAccountMapping(accountMappingRequestToAccountMapping(cloudStorageRequest.getAccountMapping()));
    fileSystem.setCloudStorage(cloudStorage);
    return fileSystem;
}
Also used : CloudStorage(com.sequenceiq.cloudbreak.domain.cloudstorage.CloudStorage) SpiFileSystem(com.sequenceiq.cloudbreak.cloud.model.SpiFileSystem) FileSystem(com.sequenceiq.cloudbreak.domain.FileSystem) FileSystemType(com.sequenceiq.common.model.FileSystemType) CloudIdentity(com.sequenceiq.cloudbreak.domain.cloudstorage.CloudIdentity) StorageLocation(com.sequenceiq.cloudbreak.domain.cloudstorage.StorageLocation)

Example 2 with FileSystemType

use of com.sequenceiq.common.model.FileSystemType in project cloudbreak by hortonworks.

the class CloudStorageConverter method requestToSpiFileSystem.

public SpiFileSystem requestToSpiFileSystem(CloudStorageBase cloudStorageRequest) {
    List<CloudFileSystemView> cloudFileSystemViews = new ArrayList<>();
    FileSystemType type = null;
    if (cloudStorageRequest.getIdentities() != null && !cloudStorageRequest.getIdentities().isEmpty()) {
        for (StorageIdentityBase storageIdentity : cloudStorageRequest.getIdentities()) {
            type = getCloudFileSystemView(cloudStorageRequest, cloudFileSystemViews, storageIdentity);
        }
    }
    validateCloudFileSystemViews(cloudFileSystemViews, type);
    return new SpiFileSystem("", type, cloudFileSystemViews);
}
Also used : FileSystemType(com.sequenceiq.common.model.FileSystemType) CloudFileSystemView(com.sequenceiq.cloudbreak.cloud.model.filesystem.CloudFileSystemView) ArrayList(java.util.ArrayList) SpiFileSystem(com.sequenceiq.cloudbreak.cloud.model.SpiFileSystem) StorageIdentityBase(com.sequenceiq.common.api.cloudstorage.StorageIdentityBase)

Example 3 with FileSystemType

use of com.sequenceiq.common.model.FileSystemType in project cloudbreak by hortonworks.

the class AzureSetup method validateFileSystem.

@Override
public void validateFileSystem(CloudCredential credential, SpiFileSystem spiFileSystem) throws Exception {
    FileSystemType fileSystemType = spiFileSystem.getType();
    List<CloudFileSystemView> cloudFileSystems = spiFileSystem.getCloudFileSystems();
    if (cloudFileSystems.size() > 2) {
        throw new CloudConnectorException("More than 2 file systems (identities) are not yet supported on Azure!");
    }
    if (cloudFileSystems.isEmpty()) {
        LOGGER.info("No filesystem was configured.");
        return;
    }
    if (FileSystemType.ADLS.equals(fileSystemType)) {
        validateAdlsFileSystem(credential, spiFileSystem);
    } else if (FileSystemType.ADLS_GEN_2.equals(fileSystemType)) {
        validateAdlsGen2FileSystem(spiFileSystem);
    } else {
        validateWasbFileSystem(spiFileSystem);
    }
}
Also used : CloudConnectorException(com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException) FileSystemType(com.sequenceiq.common.model.FileSystemType) CloudFileSystemView(com.sequenceiq.cloudbreak.cloud.model.filesystem.CloudFileSystemView)

Example 4 with FileSystemType

use of com.sequenceiq.common.model.FileSystemType in project cloudbreak by hortonworks.

the class CloudStorageConfigDetails method queryParameters.

public Set<ConfigQueryEntry> queryParameters(BlueprintTextProcessor blueprintTextProcessor, ConfigQueryEntries configQueryEntries, FileSystemConfigQueryObject request) {
    Set<ConfigQueryEntry> filtered = new HashSet<>();
    Map<String, Set<String>> componentsByHostGroup = blueprintTextProcessor.getComponentsByHostGroup();
    boolean attachedCluster = request.isAttachedCluster();
    for (Map.Entry<String, Set<String>> serviceHostgroupEntry : componentsByHostGroup.entrySet()) {
        for (String service : serviceHostgroupEntry.getValue()) {
            Set<ConfigQueryEntry> collectedEntries = configQueryEntries.getEntries().stream().filter(configQueryEntry -> configQueryEntry.getRelatedServices().stream().anyMatch(relatedService -> relatedService.equalsIgnoreCase(service))).filter(configQueryEntry -> {
                if ((configQueryEntry.isRequiredForAttachedCluster() && attachedCluster) || !attachedCluster) {
                    return true;
                }
                return false;
            }).filter(configQueryEntry -> configQueryEntry.getSupportedStorages().contains(request.getFileSystemType().toUpperCase())).collect(Collectors.toSet());
            filtered.addAll(collectedEntries);
        }
    }
    Set<ConfigQueryEntry> collectedEntries = configQueryEntries.getEntries().stream().filter(configQueryEntry -> blueprintDoesNotContainActual(configQueryEntry.getRelatedMissingServices(), componentsByHostGroup)).collect(Collectors.toSet());
    filtered.addAll(collectedEntries);
    String fileSystemTypeRequest = request.getFileSystemType();
    FileSystemType fileSystemType = FileSystemType.valueOf(fileSystemTypeRequest);
    String protocol = fileSystemType.getProtocol();
    Map<String, Object> templateObject = getTemplateObject(request, protocol);
    for (ConfigQueryEntry configQueryEntry : filtered) {
        try {
            boolean secure = request.isSecure();
            configQueryEntry.setProtocol(secure ? protocol + "s" : protocol);
            configQueryEntry.setSecure(secure);
            configQueryEntry.setDefaultPath(generateConfigWithParameters(configQueryEntry.getDefaultPath(), fileSystemType, templateObject));
        } catch (IOException e) {
        }
    }
    filtered = filtered.stream().sorted(Comparator.comparing(ConfigQueryEntry::getPropertyName)).collect(Collectors.toCollection(LinkedHashSet::new));
    return filtered;
}
Also used : Logger(org.slf4j.Logger) HandlebarTemplate(com.sequenceiq.cloudbreak.handlebar.HandlebarTemplate) ConfigQueryEntries(com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntries) LoggerFactory(org.slf4j.LoggerFactory) Set(java.util.Set) IOException(java.io.IOException) HashMap(java.util.HashMap) BlueprintTextProcessor(com.sequenceiq.cloudbreak.template.processor.BlueprintTextProcessor) Collectors(java.util.stream.Collectors) StringUtils(org.apache.commons.lang3.StringUtils) HashSet(java.util.HashSet) Service(org.springframework.stereotype.Service) Map(java.util.Map) FileSystemType(com.sequenceiq.common.model.FileSystemType) ConfigQueryEntry(com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntry) HandlebarUtils(com.sequenceiq.cloudbreak.handlebar.HandlebarUtils) Handlebars(com.github.jknack.handlebars.Handlebars) Comparator(java.util.Comparator) LinkedHashSet(java.util.LinkedHashSet) Template(com.github.jknack.handlebars.Template) LinkedHashSet(java.util.LinkedHashSet) ConfigQueryEntry(com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntry) Set(java.util.Set) HashSet(java.util.HashSet) LinkedHashSet(java.util.LinkedHashSet) IOException(java.io.IOException) FileSystemType(com.sequenceiq.common.model.FileSystemType) HashMap(java.util.HashMap) Map(java.util.Map) HashSet(java.util.HashSet) LinkedHashSet(java.util.LinkedHashSet)

Example 5 with FileSystemType

use of com.sequenceiq.common.model.FileSystemType in project cloudbreak by hortonworks.

the class CloudStorageLocationValidator method validate.

public void validate(String storageLocation, Environment environment, ValidationResultBuilder resultBuilder) {
    Optional<FileSystemType> fileSystemType = getFileSystemType(environment);
    if (fileSystemType.isPresent() && FileSystemType.GCS.equals(fileSystemType.get()) && !storageLocation.startsWith("gs://")) {
        throw new EnvironmentServiceException(String.format("The Google storage location [%s] should be a gs:// URL. %s", storageLocation, getDocLink(environment.getCloudPlatform())));
    }
    String bucketName = getBucketName(fileSystemType, storageLocation);
    CloudCredential cloudCredential = credentialToCloudCredentialConverter.convert(environment.getCredential());
    ObjectStorageMetadataRequest request = createObjectStorageMetadataRequest(environment.getCloudPlatform(), cloudCredential, bucketName);
    ObjectStorageMetadataResponse response = ThreadBasedUserCrnProvider.doAsInternalActor(regionAwareInternalCrnGeneratorFactory.iam().getInternalCrnForServiceAsString(), () -> cloudProviderServicesV4Endopint.getObjectStorageMetaData(request));
    resultBuilder.ifError(() -> response.getStatus() == ResponseStatus.OK && !environment.getLocation().equals(response.getRegion()), String.format("Object storage location [%s] of bucket '%s' must match environment location [%s].%s", response.getRegion(), bucketName, environment.getLocation(), getDocLink(environment.getCloudPlatform())));
}
Also used : ObjectStorageMetadataResponse(com.sequenceiq.cloudbreak.cloud.model.objectstorage.ObjectStorageMetadataResponse) CloudCredential(com.sequenceiq.cloudbreak.cloud.model.CloudCredential) FileSystemType(com.sequenceiq.common.model.FileSystemType) EnvironmentServiceException(com.sequenceiq.environment.exception.EnvironmentServiceException) ObjectStorageMetadataRequest(com.sequenceiq.cloudbreak.cloud.model.objectstorage.ObjectStorageMetadataRequest)

Aggregations

FileSystemType (com.sequenceiq.common.model.FileSystemType)13 CloudFileSystemView (com.sequenceiq.cloudbreak.cloud.model.filesystem.CloudFileSystemView)4 SpiFileSystem (com.sequenceiq.cloudbreak.cloud.model.SpiFileSystem)3 StorageIdentityBase (com.sequenceiq.common.api.cloudstorage.StorageIdentityBase)3 CloudCredential (com.sequenceiq.cloudbreak.cloud.model.CloudCredential)2 ObjectStorageMetadataRequest (com.sequenceiq.cloudbreak.cloud.model.objectstorage.ObjectStorageMetadataRequest)2 ObjectStorageMetadataResponse (com.sequenceiq.cloudbreak.cloud.model.objectstorage.ObjectStorageMetadataResponse)2 FileSystem (com.sequenceiq.cloudbreak.domain.FileSystem)2 CloudIdentity (com.sequenceiq.cloudbreak.domain.cloudstorage.CloudIdentity)2 CloudStorage (com.sequenceiq.cloudbreak.domain.cloudstorage.CloudStorage)2 StorageLocation (com.sequenceiq.cloudbreak.domain.cloudstorage.StorageLocation)2 CloudStorageRequest (com.sequenceiq.common.api.cloudstorage.CloudStorageRequest)2 AdlsGen2CloudStorageV1Parameters (com.sequenceiq.common.api.cloudstorage.old.AdlsGen2CloudStorageV1Parameters)2 WasbCloudStorageV1Parameters (com.sequenceiq.common.api.cloudstorage.old.WasbCloudStorageV1Parameters)2 ConfigQueryEntry (com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntry)2 CloudStorageCdpService (com.sequenceiq.common.model.CloudStorageCdpService)2 IOException (java.io.IOException)2 ArrayList (java.util.ArrayList)2 HashMap (java.util.HashMap)2 JsonProcessingException (com.fasterxml.jackson.core.JsonProcessingException)1