Search in sources :

Example 1 with ConfigQueryEntries

use of com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntries in project cloudbreak by hortonworks.

the class ClusterTerminationService method deleteFileSystemResources.

private void deleteFileSystemResources(Long stackId, FileSystem fileSystem, boolean force) {
    try {
        FileSystemConfigurator<BaseFileSystemConfigurationsView> fsConfigurator = fileSystemConfigurators.get(fileSystem.getType());
        ConfigQueryEntries configQueryEntries = cmCloudStorageConfigProvider.getConfigQueryEntries();
        BaseFileSystemConfigurationsView fsConfiguration = fileSystemConfigurationsViewProvider.propagateConfigurationsView(fileSystem, configQueryEntries);
        if (fsConfiguration != null) {
            fsConfiguration.setStorageContainer("cloudbreak" + stackId);
            fsConfigurator.deleteResources(fsConfiguration);
        }
    } catch (Exception e) {
        if (force) {
            LOGGER.error("Error during file system deletion, moving on based on the force flag, ", e);
        } else {
            throw new TerminationFailedException("File system resources could not be deleted: ", e);
        }
    }
}
Also used : BaseFileSystemConfigurationsView(com.sequenceiq.cloudbreak.template.filesystem.BaseFileSystemConfigurationsView) TerminationFailedException(com.sequenceiq.cloudbreak.service.stack.flow.TerminationFailedException) TerminationFailedException(com.sequenceiq.cloudbreak.service.stack.flow.TerminationFailedException) NotFoundException(com.sequenceiq.cloudbreak.common.exception.NotFoundException) TransactionExecutionException(com.sequenceiq.cloudbreak.common.service.TransactionService.TransactionExecutionException) ConfigQueryEntries(com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntries)

Example 2 with ConfigQueryEntries

use of com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntries in project cloudbreak by hortonworks.

the class CloudStorageConfigDetails method queryParameters.

public Set<ConfigQueryEntry> queryParameters(BlueprintTextProcessor blueprintTextProcessor, ConfigQueryEntries configQueryEntries, FileSystemConfigQueryObject request) {
    Set<ConfigQueryEntry> filtered = new HashSet<>();
    Map<String, Set<String>> componentsByHostGroup = blueprintTextProcessor.getComponentsByHostGroup();
    boolean attachedCluster = request.isAttachedCluster();
    for (Map.Entry<String, Set<String>> serviceHostgroupEntry : componentsByHostGroup.entrySet()) {
        for (String service : serviceHostgroupEntry.getValue()) {
            Set<ConfigQueryEntry> collectedEntries = configQueryEntries.getEntries().stream().filter(configQueryEntry -> configQueryEntry.getRelatedServices().stream().anyMatch(relatedService -> relatedService.equalsIgnoreCase(service))).filter(configQueryEntry -> {
                if ((configQueryEntry.isRequiredForAttachedCluster() && attachedCluster) || !attachedCluster) {
                    return true;
                }
                return false;
            }).filter(configQueryEntry -> configQueryEntry.getSupportedStorages().contains(request.getFileSystemType().toUpperCase())).collect(Collectors.toSet());
            filtered.addAll(collectedEntries);
        }
    }
    Set<ConfigQueryEntry> collectedEntries = configQueryEntries.getEntries().stream().filter(configQueryEntry -> blueprintDoesNotContainActual(configQueryEntry.getRelatedMissingServices(), componentsByHostGroup)).collect(Collectors.toSet());
    filtered.addAll(collectedEntries);
    String fileSystemTypeRequest = request.getFileSystemType();
    FileSystemType fileSystemType = FileSystemType.valueOf(fileSystemTypeRequest);
    String protocol = fileSystemType.getProtocol();
    Map<String, Object> templateObject = getTemplateObject(request, protocol);
    for (ConfigQueryEntry configQueryEntry : filtered) {
        try {
            boolean secure = request.isSecure();
            configQueryEntry.setProtocol(secure ? protocol + "s" : protocol);
            configQueryEntry.setSecure(secure);
            configQueryEntry.setDefaultPath(generateConfigWithParameters(configQueryEntry.getDefaultPath(), fileSystemType, templateObject));
        } catch (IOException e) {
        }
    }
    filtered = filtered.stream().sorted(Comparator.comparing(ConfigQueryEntry::getPropertyName)).collect(Collectors.toCollection(LinkedHashSet::new));
    return filtered;
}
Also used : Logger(org.slf4j.Logger) HandlebarTemplate(com.sequenceiq.cloudbreak.handlebar.HandlebarTemplate) ConfigQueryEntries(com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntries) LoggerFactory(org.slf4j.LoggerFactory) Set(java.util.Set) IOException(java.io.IOException) HashMap(java.util.HashMap) BlueprintTextProcessor(com.sequenceiq.cloudbreak.template.processor.BlueprintTextProcessor) Collectors(java.util.stream.Collectors) StringUtils(org.apache.commons.lang3.StringUtils) HashSet(java.util.HashSet) Service(org.springframework.stereotype.Service) Map(java.util.Map) FileSystemType(com.sequenceiq.common.model.FileSystemType) ConfigQueryEntry(com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntry) HandlebarUtils(com.sequenceiq.cloudbreak.handlebar.HandlebarUtils) Handlebars(com.github.jknack.handlebars.Handlebars) Comparator(java.util.Comparator) LinkedHashSet(java.util.LinkedHashSet) Template(com.github.jknack.handlebars.Template) LinkedHashSet(java.util.LinkedHashSet) ConfigQueryEntry(com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntry) Set(java.util.Set) HashSet(java.util.HashSet) LinkedHashSet(java.util.LinkedHashSet) IOException(java.io.IOException) FileSystemType(com.sequenceiq.common.model.FileSystemType) HashMap(java.util.HashMap) Map(java.util.Map) HashSet(java.util.HashSet) LinkedHashSet(java.util.LinkedHashSet)

Example 3 with ConfigQueryEntries

use of com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntries in project cloudbreak by hortonworks.

the class StackToTemplatePreparationObjectConverterTest method testConvertWhenEnvironmentBackupLocationDefinedThenBaseFileSystemConfigurationsViewShouldAddIt.

@Test
public void testConvertWhenEnvironmentBackupLocationDefinedThenBaseFileSystemConfigurationsViewShouldAddIt() throws IOException {
    String backupLocation = "s3a://test";
    FileSystem sourceFileSystem = new FileSystem();
    FileSystem clusterServiceFileSystem = new FileSystem();
    ConfigQueryEntries configQueryEntries = new ConfigQueryEntries();
    BaseFileSystemConfigurationsView expected = mock(BaseFileSystemConfigurationsView.class);
    List<StorageLocationView> storageLocationViews = mock(List.class);
    BackupResponse backupResponse = new BackupResponse();
    backupResponse.setStorageLocation(backupLocation);
    DetailedEnvironmentResponse environmentResponse = DetailedEnvironmentResponse.builder().withIdBrokerMappingSource(IdBrokerMappingSource.MOCK).withCredential(new CredentialResponse()).withAdminGroupName(ADMIN_GROUP_NAME).withCrn(TestConstants.CRN).withBackup(backupResponse).build();
    StorageLocation storageLocation = new StorageLocation();
    storageLocation.setValue(backupLocation);
    storageLocation.setProperty(RangerCloudStorageServiceConfigProvider.DEFAULT_BACKUP_DIR);
    StorageLocationView backupLocationView = new StorageLocationView(storageLocation);
    when(sourceCluster.getFileSystem()).thenReturn(sourceFileSystem);
    when(cluster.getFileSystem()).thenReturn(clusterServiceFileSystem);
    when(fileSystemConfigurationProvider.fileSystemConfiguration(eq(clusterServiceFileSystem), eq(stackMock), any(), eq(new Json("")), eq(configQueryEntries))).thenReturn(expected);
    when(cmCloudStorageConfigProvider.getConfigQueryEntries()).thenReturn(configQueryEntries);
    when(environmentClientService.getByCrn(anyString())).thenReturn(environmentResponse);
    when(blueprintViewProvider.getBlueprintView(any())).thenReturn(getBlueprintView());
    when(expected.getLocations()).thenReturn(storageLocationViews);
    TemplatePreparationObject result = underTest.convert(stackMock);
    assertThat(result.getFileSystemConfigurationView().isPresent()).isTrue();
    assertThat(result.getFileSystemConfigurationView().get()).isEqualTo(expected);
    verify(fileSystemConfigurationProvider, times(1)).fileSystemConfiguration(eq(clusterServiceFileSystem), eq(stackMock), any(), eq(new Json("")), eq(configQueryEntries));
    verify(expected, times(1)).getLocations();
    verify(storageLocationViews, times(1)).add(eq(backupLocationView));
}
Also used : TemplatePreparationObject(com.sequenceiq.cloudbreak.template.TemplatePreparationObject) StorageLocationView(com.sequenceiq.cloudbreak.template.filesystem.StorageLocationView) BaseFileSystemConfigurationsView(com.sequenceiq.cloudbreak.template.filesystem.BaseFileSystemConfigurationsView) FileSystem(com.sequenceiq.cloudbreak.domain.FileSystem) DetailedEnvironmentResponse(com.sequenceiq.environment.api.v1.environment.model.response.DetailedEnvironmentResponse) CredentialResponse(com.sequenceiq.environment.api.v1.credential.model.response.CredentialResponse) ArgumentMatchers.anyString(org.mockito.ArgumentMatchers.anyString) Json(com.sequenceiq.cloudbreak.common.json.Json) BackupResponse(com.sequenceiq.common.api.backup.response.BackupResponse) StorageLocation(com.sequenceiq.cloudbreak.domain.StorageLocation) ConfigQueryEntries(com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntries) Test(org.junit.jupiter.api.Test)

Example 4 with ConfigQueryEntries

use of com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntries in project cloudbreak by hortonworks.

the class StackV4RequestToTemplatePreparationObjectConverterTest method testConvertWhenClusterHasCloudStorageThenConvertedFileSystemShouldBeStoredComingFromFileSystemConfigurationProvider.

@Test
public void testConvertWhenClusterHasCloudStorageThenConvertedFileSystemShouldBeStoredComingFromFileSystemConfigurationProvider() throws IOException {
    BaseFileSystemConfigurationsView expected = mock(BaseFileSystemConfigurationsView.class);
    CloudStorageRequest cloudStorageRequest = new CloudStorageRequest();
    FileSystem fileSystem = new FileSystem();
    ConfigQueryEntries configQueryEntries = new ConfigQueryEntries();
    when(cloudStorageValidationUtil.isCloudStorageConfigured(cloudStorageRequest)).thenReturn(true);
    when(cluster.getCloudStorage()).thenReturn(cloudStorageRequest);
    when(cloudStorageConverter.requestToFileSystem(cloudStorageRequest)).thenReturn(fileSystem);
    when(cmCloudStorageConfigProvider.getConfigQueryEntries()).thenReturn(configQueryEntries);
    when(fileSystemConfigurationProvider.fileSystemConfiguration(fileSystem, source, credential.getAttributes(), configQueryEntries)).thenReturn(expected);
    TemplatePreparationObject result = underTest.convert(source);
    assertTrue(result.getFileSystemConfigurationView().isPresent());
    assertEquals(expected, result.getFileSystemConfigurationView().get());
}
Also used : TemplatePreparationObject(com.sequenceiq.cloudbreak.template.TemplatePreparationObject) CloudStorageRequest(com.sequenceiq.common.api.cloudstorage.CloudStorageRequest) BaseFileSystemConfigurationsView(com.sequenceiq.cloudbreak.template.filesystem.BaseFileSystemConfigurationsView) FileSystem(com.sequenceiq.cloudbreak.domain.FileSystem) ConfigQueryEntries(com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntries) Test(org.junit.Test)

Example 5 with ConfigQueryEntries

use of com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntries in project cloudbreak by hortonworks.

the class StackToTemplatePreparationObjectConverterTest method testConvertWhenClusterProvidesFileSystemThenBaseFileSystemConfigurationsViewShouldBeExists.

@Test
public void testConvertWhenClusterProvidesFileSystemThenBaseFileSystemConfigurationsViewShouldBeExists() throws IOException {
    FileSystem sourceFileSystem = new FileSystem();
    FileSystem clusterServiceFileSystem = new FileSystem();
    ConfigQueryEntries configQueryEntries = new ConfigQueryEntries();
    BaseFileSystemConfigurationsView expected = mock(BaseFileSystemConfigurationsView.class);
    when(sourceCluster.getFileSystem()).thenReturn(sourceFileSystem);
    when(cluster.getFileSystem()).thenReturn(clusterServiceFileSystem);
    when(fileSystemConfigurationProvider.fileSystemConfiguration(eq(clusterServiceFileSystem), eq(stackMock), any(), eq(new Json("")), eq(configQueryEntries))).thenReturn(expected);
    when(cmCloudStorageConfigProvider.getConfigQueryEntries()).thenReturn(configQueryEntries);
    when(blueprintViewProvider.getBlueprintView(any())).thenReturn(getBlueprintView());
    TemplatePreparationObject result = underTest.convert(stackMock);
    assertThat(result.getFileSystemConfigurationView().isPresent()).isTrue();
    assertThat(result.getFileSystemConfigurationView().get()).isEqualTo(expected);
    verify(fileSystemConfigurationProvider, times(1)).fileSystemConfiguration(eq(clusterServiceFileSystem), eq(stackMock), any(), eq(new Json("")), eq(configQueryEntries));
}
Also used : TemplatePreparationObject(com.sequenceiq.cloudbreak.template.TemplatePreparationObject) BaseFileSystemConfigurationsView(com.sequenceiq.cloudbreak.template.filesystem.BaseFileSystemConfigurationsView) FileSystem(com.sequenceiq.cloudbreak.domain.FileSystem) Json(com.sequenceiq.cloudbreak.common.json.Json) ConfigQueryEntries(com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntries) Test(org.junit.jupiter.api.Test)

Aggregations

ConfigQueryEntries (com.sequenceiq.common.api.cloudstorage.query.ConfigQueryEntries)7 FileSystem (com.sequenceiq.cloudbreak.domain.FileSystem)5 BaseFileSystemConfigurationsView (com.sequenceiq.cloudbreak.template.filesystem.BaseFileSystemConfigurationsView)5 Json (com.sequenceiq.cloudbreak.common.json.Json)4 TemplatePreparationObject (com.sequenceiq.cloudbreak.template.TemplatePreparationObject)4 Test (org.junit.jupiter.api.Test)4 IOException (java.io.IOException)2 ArgumentMatchers.anyString (org.mockito.ArgumentMatchers.anyString)2 Handlebars (com.github.jknack.handlebars.Handlebars)1 Template (com.github.jknack.handlebars.Template)1 CloudbreakServiceException (com.sequenceiq.cloudbreak.common.exception.CloudbreakServiceException)1 NotFoundException (com.sequenceiq.cloudbreak.common.exception.NotFoundException)1 TransactionExecutionException (com.sequenceiq.cloudbreak.common.service.TransactionService.TransactionExecutionException)1 StorageLocation (com.sequenceiq.cloudbreak.domain.StorageLocation)1 HandlebarTemplate (com.sequenceiq.cloudbreak.handlebar.HandlebarTemplate)1 HandlebarUtils (com.sequenceiq.cloudbreak.handlebar.HandlebarUtils)1 TerminationFailedException (com.sequenceiq.cloudbreak.service.stack.flow.TerminationFailedException)1 StorageLocationView (com.sequenceiq.cloudbreak.template.filesystem.StorageLocationView)1 BlueprintTextProcessor (com.sequenceiq.cloudbreak.template.processor.BlueprintTextProcessor)1 BackupResponse (com.sequenceiq.common.api.backup.response.BackupResponse)1