Search in sources :

Example 1 with FolderWithMetadata

use of com.epam.pipeline.entity.metadata.FolderWithMetadata in project cloud-pipeline by epam.

the class ParameterMapper method resolveConfigurations.

public List<ResolvedConfiguration> resolveConfigurations(AnalysisConfiguration<? extends AbstractRunConfigurationEntry> configuration) {
    FolderWithMetadata project = folderManager.getProject(configuration.getConfigurationId(), AclClass.CONFIGURATION);
    Map<String, PipeConfValue> projectData = project == null ? new HashMap<>() : project.getData();
    List<? extends AbstractRunConfigurationEntry> entries = configuration.getEntries();
    if (CollectionUtils.isEmpty(configuration.getEntitiesIds())) {
        return Collections.singletonList(resolveParameters(entries, projectData));
    }
    // In case of array references one entity may be expanded to
    // list of references entities, e.g. SampleSet is expanded
    // to list of Sample entities
    // TODO: The only reason to store it as map - is to add association to run
    // TODO: to initial entity, from which link comes. Find better solution.
    Map<Long, List<MetadataEntity>> targetEntities = fetchAndExpandInputEntities(configuration);
    // resolve all parameter references in configurations
    Map<Long, ResolvedConfiguration> resolvedConfigurations = targetEntities.values().stream().flatMap(Collection::stream).collect(Collectors.toMap(BaseEntity::getId, entity -> resolveParameters(entity, entries, projectData)));
    return targetEntities.entrySet().stream().map(idToEntities -> idToEntities.getValue().stream().map(entity -> {
        ResolvedConfiguration currentConfiguration = resolvedConfigurations.get(entity.getId());
        currentConfiguration.getAssociatedEntityIds().add(idToEntities.getKey());
        return currentConfiguration;
    }).collect(Collectors.toList())).flatMap(Collection::stream).collect(Collectors.toList());
}
Also used : EntityTypeField(com.epam.pipeline.manager.metadata.parser.EntityTypeField) Setter(lombok.Setter) Getter(lombok.Getter) MessageConstants(com.epam.pipeline.common.MessageConstants) RequiredArgsConstructor(lombok.RequiredArgsConstructor) LoggerFactory(org.slf4j.LoggerFactory) HashMap(java.util.HashMap) PipeConfValueVO(com.epam.pipeline.entity.configuration.PipeConfValueVO) StringUtils(org.apache.commons.lang3.StringUtils) Function(java.util.function.Function) CollectionUtils(org.apache.commons.collections4.CollectionUtils) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) PipeConfValue(com.epam.pipeline.entity.metadata.PipeConfValue) MessageHelper(com.epam.pipeline.common.MessageHelper) AnalysisConfiguration(com.epam.pipeline.manager.pipeline.runner.AnalysisConfiguration) Service(org.springframework.stereotype.Service) AbstractRunConfigurationEntry(com.epam.pipeline.entity.configuration.AbstractRunConfigurationEntry) FolderWithMetadata(com.epam.pipeline.entity.metadata.FolderWithMetadata) Map(java.util.Map) TypeReference(com.fasterxml.jackson.core.type.TypeReference) PipelineConfiguration(com.epam.pipeline.entity.configuration.PipelineConfiguration) MapUtils(org.apache.commons.collections4.MapUtils) BaseEntity(com.epam.pipeline.entity.BaseEntity) Logger(org.slf4j.Logger) Collection(java.util.Collection) JsonMapper(com.epam.pipeline.config.JsonMapper) EqualsAndHashCode(lombok.EqualsAndHashCode) Collectors(java.util.stream.Collectors) ResolvedConfiguration(com.epam.pipeline.entity.pipeline.ResolvedConfiguration) Objects(java.util.Objects) List(java.util.List) MetadataEntity(com.epam.pipeline.entity.metadata.MetadataEntity) MetadataEntityManager(com.epam.pipeline.manager.metadata.MetadataEntityManager) AclClass(com.epam.pipeline.entity.security.acl.AclClass) PipelineStart(com.epam.pipeline.entity.pipeline.run.PipelineStart) AllArgsConstructor(lombok.AllArgsConstructor) Collections(java.util.Collections) Assert(org.springframework.util.Assert) ResolvedConfiguration(com.epam.pipeline.entity.pipeline.ResolvedConfiguration) PipeConfValue(com.epam.pipeline.entity.metadata.PipeConfValue) ArrayList(java.util.ArrayList) List(java.util.List) FolderWithMetadata(com.epam.pipeline.entity.metadata.FolderWithMetadata)

Example 2 with FolderWithMetadata

use of com.epam.pipeline.entity.metadata.FolderWithMetadata in project cloud-pipeline by epam.

the class FirecloudRunner method runAnalysis.

@Override
public List<PipelineRun> runAnalysis(AnalysisConfiguration<FirecloudRunConfigurationEntry> configuration) {
    Assert.state(CollectionUtils.isNotEmpty(configuration.getEntitiesIds()), "Entities are required to run analysis in Firecloud");
    FolderWithMetadata project = folderManager.getProject(configuration.getConfigurationId(), AclClass.CONFIGURATION);
    Assert.notNull(project, "Project folder must be specified to run analysis in Firecloud.");
    List<Long> entities = parameterMapper.fetchAndExpandInputEntities(configuration).values().stream().flatMap(Collection::stream).map(BaseEntity::getId).collect(Collectors.toList());
    // TODO: after merge - add check that FC class is filled
    configuration.getEntries().forEach(this::validateFirecloudConfiguration);
    FirecloudConfiguration settings = buildFirecloudConfig(configuration.getRefreshToken());
    return configuration.getEntries().stream().map(entry -> runFirecloudAnalysis(settings, entry, configuration.getConfigurationId(), entities, project.getId())).collect(Collectors.toList());
}
Also used : CredentialsManager(com.epam.pipeline.manager.google.CredentialsManager) PasswordGenerator(com.epam.pipeline.utils.PasswordGenerator) Arrays(java.util.Arrays) ParameterMapper(com.epam.pipeline.manager.pipeline.ParameterMapper) FirecloudCredentials(com.epam.pipeline.manager.google.FirecloudCredentials) PipelineRunManager(com.epam.pipeline.manager.pipeline.PipelineRunManager) RequiredArgsConstructor(lombok.RequiredArgsConstructor) SystemPreferences(com.epam.pipeline.manager.preference.SystemPreferences) HashMap(java.util.HashMap) StringUtils(org.apache.commons.lang3.StringUtils) CollectionUtils(org.apache.commons.collections4.CollectionUtils) ArrayList(java.util.ArrayList) PipelineRun(com.epam.pipeline.entity.pipeline.PipelineRun) FireCloudClass(com.epam.pipeline.entity.metadata.FireCloudClass) Service(org.springframework.stereotype.Service) FolderWithMetadata(com.epam.pipeline.entity.metadata.FolderWithMetadata) PipelineConfiguration(com.epam.pipeline.entity.configuration.PipelineConfiguration) BaseEntity(com.epam.pipeline.entity.BaseEntity) FirecloudPreferences(com.epam.pipeline.entity.pipeline.run.FirecloudPreferences) PipelineConfigurationManager(com.epam.pipeline.manager.pipeline.PipelineConfigurationManager) PreferenceManager(com.epam.pipeline.manager.preference.PreferenceManager) FolderManager(com.epam.pipeline.manager.pipeline.FolderManager) Collection(java.util.Collection) FirecloudRunConfigurationEntry(com.epam.pipeline.entity.configuration.FirecloudRunConfigurationEntry) Collectors(java.util.stream.Collectors) List(java.util.List) Builder(lombok.Builder) MetadataEntityManager(com.epam.pipeline.manager.metadata.MetadataEntityManager) Data(lombok.Data) AclClass(com.epam.pipeline.entity.security.acl.AclClass) SystemParams(com.epam.pipeline.manager.execution.SystemParams) PipelineStart(com.epam.pipeline.entity.pipeline.run.PipelineStart) MetadataClass(com.epam.pipeline.entity.metadata.MetadataClass) Assert(org.springframework.util.Assert) Collection(java.util.Collection) FolderWithMetadata(com.epam.pipeline.entity.metadata.FolderWithMetadata)

Example 3 with FolderWithMetadata

use of com.epam.pipeline.entity.metadata.FolderWithMetadata in project cloud-pipeline by epam.

the class FolderDaoTest method testLoadParentFolders.

@Test
@Transactional(propagation = Propagation.REQUIRES_NEW, rollbackFor = Exception.class)
public void testLoadParentFolders() {
    Folder folder = getFolder();
    Folder folder1 = createFolderWithParent(folder, "folder1");
    Folder folder2 = createFolderWithParent(folder1, "folder2");
    // folder that exists but mustn't be included in actualFolders
    createFolderWithParent(folder1, "folder3");
    List<Folder> expectedFolders = Stream.of(folder, folder1, folder2).collect(Collectors.toList());
    List<FolderWithMetadata> expectedFoldersWithMetadata = createObjectMetadata(expectedFolders);
    List<FolderWithMetadata> actualFoldersWithMetadata = folderDao.loadParentFolders(folder2.getId());
    assertTrue(CollectionUtils.isEqualCollection(actualFoldersWithMetadata, expectedFoldersWithMetadata));
    List<Folder> actualFolders = folderDao.loadFolderWithParents(folder2.getId());
    assertTrue(CollectionUtils.isEqualCollection(actualFolders, expectedFolders));
}
Also used : Folder(com.epam.pipeline.entity.pipeline.Folder) FolderWithMetadata(com.epam.pipeline.entity.metadata.FolderWithMetadata) AbstractSpringTest(com.epam.pipeline.AbstractSpringTest) Test(org.junit.Test) Transactional(org.springframework.transaction.annotation.Transactional)

Example 4 with FolderWithMetadata

use of com.epam.pipeline.entity.metadata.FolderWithMetadata in project cloud-pipeline by epam.

the class FolderDaoTest method convertToFolderWithMetadata.

private FolderWithMetadata convertToFolderWithMetadata(Folder folder) {
    FolderWithMetadata folderWithMetadata = new FolderWithMetadata();
    folderWithMetadata.setId(folder.getId());
    folderWithMetadata.setParentId(folder.getParentId());
    folderWithMetadata.setName(folder.getName());
    folderWithMetadata.setOwner(folder.getOwner());
    return folderWithMetadata;
}
Also used : FolderWithMetadata(com.epam.pipeline.entity.metadata.FolderWithMetadata)

Example 5 with FolderWithMetadata

use of com.epam.pipeline.entity.metadata.FolderWithMetadata in project cloud-pipeline by epam.

the class FolderManagerGetProjectTest method getProjectShouldWorkWithoutParent.

@Test
public void getProjectShouldWorkWithoutParent() {
    Mockito.when(entityManager.load(Matchers.any(AclClass.class), Matchers.any(Long.class))).thenReturn(folder1);
    FolderWithMetadata project = folderManager.getProject(folder1.getId(), AclClass.FOLDER);
    assertNull(project);
}
Also used : AclClass(com.epam.pipeline.entity.security.acl.AclClass) FolderWithMetadata(com.epam.pipeline.entity.metadata.FolderWithMetadata) FolderManagerTest.initFolderWithMetadata(com.epam.pipeline.manager.pipeline.FolderManagerTest.initFolderWithMetadata) AbstractSpringTest(com.epam.pipeline.AbstractSpringTest) Test(org.junit.Test)

Aggregations

FolderWithMetadata (com.epam.pipeline.entity.metadata.FolderWithMetadata)12 PipeConfValue (com.epam.pipeline.entity.metadata.PipeConfValue)5 AclClass (com.epam.pipeline.entity.security.acl.AclClass)5 AbstractSpringTest (com.epam.pipeline.AbstractSpringTest)4 Folder (com.epam.pipeline.entity.pipeline.Folder)4 Test (org.junit.Test)4 BaseEntity (com.epam.pipeline.entity.BaseEntity)3 MetadataEntityManager (com.epam.pipeline.manager.metadata.MetadataEntityManager)3 ArrayList (java.util.ArrayList)3 HashMap (java.util.HashMap)3 List (java.util.List)3 Collectors (java.util.stream.Collectors)3 MessageConstants (com.epam.pipeline.common.MessageConstants)2 MessageHelper (com.epam.pipeline.common.MessageHelper)2 EntityVO (com.epam.pipeline.controller.vo.EntityVO)2 AbstractSecuredEntity (com.epam.pipeline.entity.AbstractSecuredEntity)2 PipelineConfiguration (com.epam.pipeline.entity.configuration.PipelineConfiguration)2 MetadataEntry (com.epam.pipeline.entity.metadata.MetadataEntry)2 PipelineStart (com.epam.pipeline.entity.pipeline.run.PipelineStart)2 FolderManagerTest.initFolderWithMetadata (com.epam.pipeline.manager.pipeline.FolderManagerTest.initFolderWithMetadata)2