Search in sources :

Example 36 with CodeLocation

use of com.synopsys.integration.detectable.detectable.codelocation.CodeLocation in project synopsys-detect by blackducksoftware.

the class YarnLockDetectableTest method assertExtraction.

@Override
public void assertExtraction(@NotNull Extraction extraction) {
    Assertions.assertEquals(1, extraction.getCodeLocations().size());
    CodeLocation codeLocation = extraction.getCodeLocations().get(0);
    Assertions.assertEquals("babel", extraction.getProjectName());
    Assertions.assertEquals("1.2.3", extraction.getProjectVersion());
    NameVersionGraphAssert graphAssert = new NameVersionGraphAssert(Forge.NPMJS, codeLocation.getDependencyGraph());
    graphAssert.hasRootSize(2);
    graphAssert.hasRootDependency("async", "2.5.0");
    graphAssert.hasRootDependency("lodash", "4.17.4");
    graphAssert.hasParentChildRelationship("async", "2.5.0", "lodash", "4.17.4");
}
Also used : CodeLocation(com.synopsys.integration.detectable.detectable.codelocation.CodeLocation) NameVersionGraphAssert(com.synopsys.integration.detectable.util.graph.NameVersionGraphAssert)

Example 37 with CodeLocation

use of com.synopsys.integration.detectable.detectable.codelocation.CodeLocation in project synopsys-detect by blackducksoftware.

the class ExtractionUtil method assertAndGetCodeLocationNamed.

public static CodeLocation assertAndGetCodeLocationNamed(String name, Extraction extraction) {
    CodeLocation found = null;
    for (CodeLocation it : extraction.getCodeLocations()) {
        if (it.getExternalId().isPresent() && it.getExternalId().get().getName().equals(name)) {
            found = it;
        }
    }
    Assertions.assertNotNull(found, "Missing code location with name: " + name);
    return found;
}
Also used : CodeLocation(com.synopsys.integration.detectable.detectable.codelocation.CodeLocation)

Example 38 with CodeLocation

use of com.synopsys.integration.detectable.detectable.codelocation.CodeLocation in project synopsys-detect by blackducksoftware.

the class CodeLocationConverter method toDetectCodeLocation.

public Map<CodeLocation, DetectCodeLocation> toDetectCodeLocation(File detectSourcePath, Extraction extraction, File overridePath, String overrideName) {
    Map<CodeLocation, DetectCodeLocation> detectCodeLocations = new HashMap<>();
    for (CodeLocation codeLocation : extraction.getCodeLocations()) {
        File sourcePath = codeLocation.getSourcePath().orElse(overridePath);
        ExternalId externalId;
        if (!codeLocation.getExternalId().isPresent()) {
            logger.debug("The detector was unable to determine an external id for this code location, so an external id will be created using the file path.");
            String relativePath = FileNameUtils.relativize(detectSourcePath.getAbsolutePath(), sourcePath.getAbsolutePath());
            if (StringUtils.isNotBlank(relativePath)) {
                externalId = externalIdFactory.createPathExternalId(DETECT_FORGE, relativePath);
            } else {
                // Relativize from the parent.
                externalId = externalIdFactory.createPathExternalId(DETECT_FORGE, FileNameUtils.relativizeParent(detectSourcePath.getAbsolutePath(), sourcePath.getAbsolutePath()));
            }
            logger.debug("The external id that was created is: {}", Arrays.asList(externalId.getExternalIdPieces()));
        } else {
            externalId = codeLocation.getExternalId().get();
        }
        Optional<String> dockerImageName = extraction.getMetaData(DockerExtractor.DOCKER_IMAGE_NAME_META_DATA);
        DetectCodeLocation detectCodeLocation = dockerImageName.map(s -> DetectCodeLocation.forDocker(codeLocation.getDependencyGraph(), sourcePath, externalId, s)).orElseGet(() -> DetectCodeLocation.forCreator(codeLocation.getDependencyGraph(), sourcePath, externalId, overrideName));
        detectCodeLocations.put(codeLocation, detectCodeLocation);
    }
    return detectCodeLocations;
}
Also used : Arrays(java.util.Arrays) Logger(org.slf4j.Logger) Forge(com.synopsys.integration.bdio.model.Forge) Extraction(com.synopsys.integration.detectable.extraction.Extraction) LoggerFactory(org.slf4j.LoggerFactory) ExternalIdFactory(com.synopsys.integration.bdio.model.externalid.ExternalIdFactory) HashMap(java.util.HashMap) FileNameUtils(com.synopsys.integration.detect.workflow.codelocation.FileNameUtils) StringUtils(org.apache.commons.lang3.StringUtils) File(java.io.File) DetectorEvaluation(com.synopsys.integration.detector.base.DetectorEvaluation) CodeLocation(com.synopsys.integration.detectable.detectable.codelocation.CodeLocation) Map(java.util.Map) Optional(java.util.Optional) ExternalId(com.synopsys.integration.bdio.model.externalid.ExternalId) DockerExtractor(com.synopsys.integration.detectable.detectables.docker.DockerExtractor) DetectCodeLocation(com.synopsys.integration.detect.workflow.codelocation.DetectCodeLocation) CodeLocation(com.synopsys.integration.detectable.detectable.codelocation.CodeLocation) DetectCodeLocation(com.synopsys.integration.detect.workflow.codelocation.DetectCodeLocation) HashMap(java.util.HashMap) DetectCodeLocation(com.synopsys.integration.detect.workflow.codelocation.DetectCodeLocation) ExternalId(com.synopsys.integration.bdio.model.externalid.ExternalId) File(java.io.File)

Example 39 with CodeLocation

use of com.synopsys.integration.detectable.detectable.codelocation.CodeLocation in project synopsys-detect by blackducksoftware.

the class DetectorTool method performDetectors.

public DetectorToolResult performDetectors(File directory, DetectorRuleSet detectorRuleSet, DetectorFinderOptions detectorFinderOptions, DetectorEvaluationOptions evaluationOptions, String projectDetector, List<DetectorType> requiredDetectors, FileFinder fileFinder) {
    logger.debug("Initializing detector system.");
    Optional<DetectorEvaluationTree> possibleRootEvaluation;
    logger.debug("Starting detector file system traversal.");
    possibleRootEvaluation = detectorFinder.findDetectors(directory, detectorRuleSet, detectorFinderOptions, fileFinder);
    if (!possibleRootEvaluation.isPresent()) {
        logger.error("The source directory could not be searched for detectors - detector tool failed.");
        logger.error("Please ensure the provided source path is a directory and detect has access.");
        exitCodePublisher.publishExitCode(ExitCodeType.FAILURE_CONFIGURATION, "Detector tool failed to run on the configured source path.");
        return new DetectorToolResult();
    }
    DetectorEvaluationTree rootEvaluation = possibleRootEvaluation.get();
    List<DetectorEvaluation> detectorEvaluations = rootEvaluation.allDescendentEvaluations();
    logger.trace("Setting up detector events.");
    // DetectorNameVersionHandler detectorNameVersionHandler = createNameVersionHandler(projectDetector);
    DetectorEvaluatorBroadcaster eventBroadcaster = new DetectorEvaluatorBroadcaster(eventSystem);
    DetectorEvaluator detectorEvaluator = new DetectorEvaluator(evaluationOptions, extractionEnvironmentProvider::createExtractionEnvironment);
    detectorEvaluator.setDetectorEvaluatorListener(eventBroadcaster);
    detectorEvaluator.registerPostApplicableCallback(detectorAggregateEvaluationResult -> {
        detectorEventPublisher.publishApplicableCompleted(detectorAggregateEvaluationResult.getApplicableDetectorTypesRecursively());
        detectorEventPublisher.publishSearchCompleted(detectorAggregateEvaluationResult.getEvaluationTree());
        logger.info("");
    });
    detectorEvaluator.registerPostExtractableCallback(detectorAggregateEvaluationResult -> {
        detectorEventPublisher.publishPreparationsCompleted(detectorAggregateEvaluationResult.getEvaluationTree());
        logger.debug("Counting detectors that will be evaluated.");
        Integer extractionCount = detectorAggregateEvaluationResult.getExtractionCount();
        detectorEventPublisher.publishExtractionCount(extractionCount);
        logger.debug("Total number of detectors: {}", extractionCount);
    });
    detectorEvaluator.registerPostExtractionCallback(detectorAggregateEvaluationResult -> detectorEventPublisher.publishExtractionsCompleted(detectorAggregateEvaluationResult.getEvaluationTree()));
    DetectorAggregateEvaluationResult evaluationResult = detectorEvaluator.evaluate(rootEvaluation);
    // TODO- finished extractions?
    logger.debug("Finished detectors.");
    printExplanations(rootEvaluation);
    Map<DetectorType, StatusType> statusMap = extractStatus(detectorEvaluations);
    publishStatusEvents(statusMap);
    publishFileEvents(detectorEvaluations);
    detectorIssuePublisher.publishEvents(statusEventPublisher, rootEvaluation);
    publishMissingDetectorEvents(requiredDetectors, evaluationResult.getApplicableDetectorTypesRecursively());
    Map<CodeLocation, DetectCodeLocation> codeLocationMap = createCodeLocationMap(detectorEvaluations, directory);
    DetectorEvaluationNameVersionDecider detectorEvaluationNameVersionDecider = new DetectorEvaluationNameVersionDecider(new DetectorNameVersionDecider());
    Optional<NameVersion> bomToolProjectNameVersion = detectorEvaluationNameVersionDecider.decideSuggestion(detectorEvaluations, projectDetector);
    logger.debug("Finished evaluating detectors for project info.");
    DetectorToolResult detectorToolResult = new DetectorToolResult(bomToolProjectNameVersion.orElse(null), new ArrayList<>(codeLocationMap.values()), evaluationResult.getApplicableDetectorTypes(), new HashSet<>(), rootEvaluation, codeLocationMap);
    // Completed.
    logger.debug("Finished running detectors.");
    detectorEventPublisher.publishDetectorsComplete(detectorToolResult);
    return detectorToolResult;
}
Also used : DetectCodeLocation(com.synopsys.integration.detect.workflow.codelocation.DetectCodeLocation) CodeLocation(com.synopsys.integration.detectable.detectable.codelocation.CodeLocation) NameVersion(com.synopsys.integration.util.NameVersion) DetectorEvaluator(com.synopsys.integration.detector.evaluation.DetectorEvaluator) DetectorEvaluation(com.synopsys.integration.detector.base.DetectorEvaluation) DetectorAggregateEvaluationResult(com.synopsys.integration.detector.evaluation.DetectorAggregateEvaluationResult) DetectorEvaluationTree(com.synopsys.integration.detector.base.DetectorEvaluationTree) DetectorType(com.synopsys.integration.detector.base.DetectorType) StatusType(com.synopsys.integration.detect.workflow.status.StatusType) DetectCodeLocation(com.synopsys.integration.detect.workflow.codelocation.DetectCodeLocation) DetectorEvaluationNameVersionDecider(com.synopsys.integration.detect.workflow.nameversion.DetectorEvaluationNameVersionDecider) DetectorNameVersionDecider(com.synopsys.integration.detect.workflow.nameversion.DetectorNameVersionDecider)

Example 40 with CodeLocation

use of com.synopsys.integration.detectable.detectable.codelocation.CodeLocation in project synopsys-detect by blackducksoftware.

the class DetectableTool method extract.

public DetectableToolResult extract() {
    // TODO: Move docker/bazel out of detectable and drop this notion of a detctable tool. Will simplify this logic and make this unneccessary.
    DetectableResult extractable;
    try {
        extractable = detectable.extractable();
    } catch (DetectableException e) {
        extractable = new ExceptionDetectableResult(e);
    }
    if (!extractable.getPassed()) {
        logger.error(String.format("Was not extractable: %s", extractable.toDescription()));
        statusEventPublisher.publishIssue(new DetectIssue(DetectIssueType.DETECTABLE_TOOL, "Detectable Tool Issue", Arrays.asList(extractable.toDescription())));
        statusEventPublisher.publishStatusSummary(new Status(name, StatusType.FAILURE));
        exitCodePublisher.publishExitCode(ExitCodeType.FAILURE_GENERAL_ERROR, extractable.toDescription());
        return DetectableToolResult.failed(extractable);
    }
    logger.debug("Extractable passed.");
    ExtractionEnvironment extractionEnvironment = extractionEnvironmentProvider.createExtractionEnvironment(name);
    Extraction extraction;
    try {
        extraction = detectable.extract(extractionEnvironment);
    } catch (ExecutableFailedException | ExecutableRunnerException | JsonSyntaxException | IOException | CycleDetectedException | DetectableException | MissingExternalIdException | ParserConfigurationException | SAXException e) {
        extraction = new Extraction.Builder().exception(e).build();
    }
    if (!extraction.isSuccess()) {
        logger.error("Extraction was not success.");
        List<String> errorMessages = collectErrorMessages(extraction);
        statusEventPublisher.publishIssue(new DetectIssue(DetectIssueType.DETECTABLE_TOOL, "Detectable Tool Issue", errorMessages));
        statusEventPublisher.publishStatusSummary(new Status(name, StatusType.FAILURE));
        exitCodePublisher.publishExitCode(new ExitCodeRequest(ExitCodeType.FAILURE_GENERAL_ERROR, extraction.getDescription()));
        return DetectableToolResult.failed();
    } else {
        logger.debug("Extraction success.");
        statusEventPublisher.publishStatusSummary(new Status(name, StatusType.SUCCESS));
    }
    Map<CodeLocation, DetectCodeLocation> detectCodeLocationMap = codeLocationConverter.toDetectCodeLocation(sourcePath, extraction, sourcePath, name);
    List<DetectCodeLocation> detectCodeLocations = new ArrayList<>(detectCodeLocationMap.values());
    DockerTargetData dockerTargetData = DockerTargetData.fromExtraction(extraction);
    DetectToolProjectInfo projectInfo = null;
    if (StringUtils.isNotBlank(extraction.getProjectName()) || StringUtils.isNotBlank(extraction.getProjectVersion())) {
        NameVersion nameVersion = new NameVersion(extraction.getProjectName(), extraction.getProjectVersion());
        projectInfo = new DetectToolProjectInfo(detectTool, nameVersion);
    }
    logger.debug("Tool finished.");
    return DetectableToolResult.success(detectCodeLocations, projectInfo, dockerTargetData);
}
Also used : ExecutableFailedException(com.synopsys.integration.detectable.detectable.executable.ExecutableFailedException) NameVersion(com.synopsys.integration.util.NameVersion) CycleDetectedException(com.synopsys.integration.detectable.util.CycleDetectedException) ArrayList(java.util.ArrayList) SAXException(org.xml.sax.SAXException) DetectIssue(com.synopsys.integration.detect.workflow.status.DetectIssue) ExtractionEnvironment(com.synopsys.integration.detectable.extraction.ExtractionEnvironment) Extraction(com.synopsys.integration.detectable.extraction.Extraction) ParserConfigurationException(javax.xml.parsers.ParserConfigurationException) DockerTargetData(com.synopsys.integration.detect.lifecycle.run.data.DockerTargetData) Status(com.synopsys.integration.detect.workflow.status.Status) DetectCodeLocation(com.synopsys.integration.detect.workflow.codelocation.DetectCodeLocation) CodeLocation(com.synopsys.integration.detectable.detectable.codelocation.CodeLocation) ExitCodeRequest(com.synopsys.integration.detect.lifecycle.shutdown.ExitCodeRequest) IOException(java.io.IOException) MissingExternalIdException(com.synopsys.integration.bdio.graph.builder.MissingExternalIdException) ExecutableRunnerException(com.synopsys.integration.executable.ExecutableRunnerException) JsonSyntaxException(com.google.gson.JsonSyntaxException) DetectCodeLocation(com.synopsys.integration.detect.workflow.codelocation.DetectCodeLocation) DetectableResult(com.synopsys.integration.detectable.detectable.result.DetectableResult) ExceptionDetectableResult(com.synopsys.integration.detectable.detectable.result.ExceptionDetectableResult) ExceptionDetectableResult(com.synopsys.integration.detectable.detectable.result.ExceptionDetectableResult) DetectToolProjectInfo(com.synopsys.integration.detect.workflow.project.DetectToolProjectInfo) DetectableException(com.synopsys.integration.detectable.detectable.exception.DetectableException)

Aggregations

CodeLocation (com.synopsys.integration.detectable.detectable.codelocation.CodeLocation)104 DependencyGraph (com.synopsys.integration.bdio.graph.DependencyGraph)60 Extraction (com.synopsys.integration.detectable.extraction.Extraction)41 File (java.io.File)24 ArrayList (java.util.ArrayList)23 Test (org.junit.jupiter.api.Test)22 ExternalId (com.synopsys.integration.bdio.model.externalid.ExternalId)21 Dependency (com.synopsys.integration.bdio.model.dependency.Dependency)20 NameVersion (com.synopsys.integration.util.NameVersion)19 IOException (java.io.IOException)19 BasicDependencyGraph (com.synopsys.integration.bdio.graph.BasicDependencyGraph)16 ExternalIdFactory (com.synopsys.integration.bdio.model.externalid.ExternalIdFactory)16 List (java.util.List)14 UnitTest (com.synopsys.integration.detectable.annotations.UnitTest)13 ExecutableOutput (com.synopsys.integration.executable.ExecutableOutput)11 ExecutableRunnerException (com.synopsys.integration.executable.ExecutableRunnerException)10 Optional (java.util.Optional)10 Collectors (java.util.stream.Collectors)10 Logger (org.slf4j.Logger)10 LoggerFactory (org.slf4j.LoggerFactory)10