Search in sources :

Example 1 with DockerClient

use of com.epam.pipeline.manager.docker.DockerClient in project cloud-pipeline by epam.

the class AggregatingToolScanManager method getActualScan.

private Optional<ToolVersionScanResult> getActualScan(Tool tool, String tag, DockerRegistry registry) {
    Optional<ToolVersionScanResult> versionScanResult = toolManager.loadToolVersionScan(tool.getId(), tag);
    if (versionScanResult.isPresent() && versionScanResult.get().getLastLayerRef() != null) {
        ToolVersionScanResult vs = versionScanResult.get();
        LOGGER.info(messageHelper.getMessage(MessageConstants.INFO_TOOL_SCAN_ALREADY_SCANNED, tool.getImage()));
        DockerClient dockerClient = getDockerClient(tool.getImage(), registry);
        String dockerRef = dockerClient.getVersionAttributes(registry, tool.getImage(), tag).getDigest();
        boolean isActual = vs.getDigest() != null && dockerRef.equals(vs.getDigest());
        if (isActual) {
            vs.setScanDate(DateUtils.now());
            return Optional.of(vs);
        } else {
            LOGGER.info(messageHelper.getMessage(MessageConstants.INFO_TOOL_SCAN_NEW_LAYERS, tool.getImage(), tag, vs.getDigest(), dockerRef));
        }
    }
    return Optional.empty();
}
Also used : ToolVersionScanResult(com.epam.pipeline.entity.scan.ToolVersionScanResult) DockerClient(com.epam.pipeline.manager.docker.DockerClient)

Example 2 with DockerClient

use of com.epam.pipeline.manager.docker.DockerClient in project cloud-pipeline by epam.

the class AggregatingToolScanManager method fetchLayers.

private List<String> fetchLayers(Tool tool, String tag, DockerRegistry registry) throws ToolScanExternalServiceException {
    DockerClient dockerClient = getDockerClient(tool.getImage(), registry);
    ManifestV2 manifest = dockerClient.getManifest(registry, tool.getImage(), tag).orElseThrow(() -> new ToolScanExternalServiceException(tool, messageHelper.getMessage(MessageConstants.ERROR_REGISTRY_COULD_NOT_GET_MANIFEST, tool.getImage())));
    return manifest.getLayers().stream().map(c -> c.getDigest()).collect(Collectors.toList());
}
Also used : DockerComponentScanResult(com.epam.pipeline.manager.docker.scan.dockercompscan.DockerComponentScanResult) Arrays(java.util.Arrays) LoggerFactory(org.slf4j.LoggerFactory) SystemPreferences(com.epam.pipeline.manager.preference.SystemPreferences) Autowired(org.springframework.beans.factory.annotation.Autowired) ManifestV2(com.epam.pipeline.entity.docker.ManifestV2) Vulnerability(com.epam.pipeline.entity.scan.Vulnerability) URLUtils(com.epam.pipeline.utils.URLUtils) StringUtils(org.apache.commons.lang3.StringUtils) DeserializationFeature(com.fasterxml.jackson.databind.DeserializationFeature) ToolDependency(com.epam.pipeline.entity.scan.ToolDependency) MessageHelper(com.epam.pipeline.common.MessageHelper) ListUtils(org.apache.commons.collections4.ListUtils) Map(java.util.Map) VulnerabilitySeverity(com.epam.pipeline.entity.scan.VulnerabilitySeverity) DateUtils(com.epam.pipeline.entity.utils.DateUtils) ClairService(com.epam.pipeline.manager.docker.scan.clair.ClairService) UUID(java.util.UUID) Instant(java.time.Instant) Collectors(java.util.stream.Collectors) Retrofit(retrofit2.Retrofit) Tool(com.epam.pipeline.entity.pipeline.Tool) List(java.util.List) ClairScanResult(com.epam.pipeline.manager.docker.scan.clair.ClairScanResult) Stream(java.util.stream.Stream) JacksonConverterFactory(retrofit2.converter.jackson.JacksonConverterFactory) PostConstruct(javax.annotation.PostConstruct) Optional(java.util.Optional) StringPreference(com.epam.pipeline.manager.preference.AbstractSystemPreference.StringPreference) DockerClientFactory(com.epam.pipeline.manager.docker.DockerClientFactory) Call(retrofit2.Call) UnsupportedEncodingException(java.io.UnsupportedEncodingException) ToolScanPolicy(com.epam.pipeline.entity.scan.ToolScanPolicy) MessageConstants(com.epam.pipeline.common.MessageConstants) ToolManager(com.epam.pipeline.manager.pipeline.ToolManager) ToolScanExternalServiceException(com.epam.pipeline.exception.ToolScanExternalServiceException) HashMap(java.util.HashMap) Response(retrofit2.Response) ClairScanRequest(com.epam.pipeline.manager.docker.scan.clair.ClairScanRequest) DockerRegistryManager(com.epam.pipeline.manager.docker.DockerRegistryManager) MapperFeature(com.fasterxml.jackson.databind.MapperFeature) Service(org.springframework.stereotype.Service) DockerComponentScanService(com.epam.pipeline.manager.docker.scan.dockercompscan.DockerComponentScanService) ToolScanStatus(com.epam.pipeline.entity.pipeline.ToolScanStatus) DockerComponentLayerScanResult(com.epam.pipeline.manager.docker.scan.dockercompscan.DockerComponentLayerScanResult) AbstractSystemPreference(com.epam.pipeline.manager.preference.AbstractSystemPreference) PreferenceManager(com.epam.pipeline.manager.preference.PreferenceManager) Logger(org.slf4j.Logger) DockerComponentScanRequest(com.epam.pipeline.manager.docker.scan.dockercompscan.DockerComponentScanRequest) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) IOException(java.io.IOException) TimeUnit(java.util.concurrent.TimeUnit) DockerRegistry(com.epam.pipeline.entity.pipeline.DockerRegistry) URLEncoder(java.net.URLEncoder) OkHttpClient(okhttp3.OkHttpClient) ToolVersionScanResult(com.epam.pipeline.entity.scan.ToolVersionScanResult) DockerClient(com.epam.pipeline.manager.docker.DockerClient) ManifestV2(com.epam.pipeline.entity.docker.ManifestV2) DockerClient(com.epam.pipeline.manager.docker.DockerClient) ToolScanExternalServiceException(com.epam.pipeline.exception.ToolScanExternalServiceException)

Example 3 with DockerClient

use of com.epam.pipeline.manager.docker.DockerClient in project cloud-pipeline by epam.

the class ToolScanScheduler method scheduledToolScan.

/**
 * A scheduled scan, that runs for all the registries, all tools and all tool versions, sends them to Tool Scanning
 * System and stores scanning results to the database.
 */
public void scheduledToolScan() {
    if (!preferenceManager.getPreference(SystemPreferences.DOCKER_SECURITY_TOOL_SCAN_ENABLED)) {
        LOGGER.info(messageHelper.getMessage(MessageConstants.ERROR_TOOL_SCAN_DISABLED));
        return;
    } else {
        LOGGER.info(messageHelper.getMessage(MessageConstants.INFO_TOOL_SCAN_SCHEDULED_STARTED));
    }
    boolean scanAllRegistries = preferenceManager.getPreference(SystemPreferences.DOCKER_SECURITY_TOOL_SCAN_ALL_REGISTRIES);
    List<DockerRegistry> registries = scanAllRegistries ? dockerRegistryDao.loadAllDockerRegistry() : dockerRegistryDao.loadDockerRegistriesWithSecurityScanEnabled();
    for (DockerRegistry registry : registries) {
        LOGGER.info(messageHelper.getMessage(MessageConstants.INFO_TOOL_SCAN_REGISTRY_STARTED, registry.getPath()));
        for (Tool tool : registry.getTools()) {
            DockerClient dockerClient = getDockerClient(registry, tool);
            try {
                List<String> versions = toolManager.loadTags(tool.getId());
                for (String version : versions) {
                    try {
                        ToolVersionScanResult result = toolScanManager.scanTool(tool, version, false);
                        toolManager.updateToolVulnerabilities(result.getVulnerabilities(), tool.getId(), version);
                        toolManager.updateToolDependencies(result.getDependencies(), tool.getId(), version);
                        toolManager.updateToolVersionScanStatus(tool.getId(), ToolScanStatus.COMPLETED, new Date(), version, result.getLastLayerRef(), result.getDigest());
                        updateToolVersion(tool, version, registry, dockerClient);
                    } catch (ToolScanExternalServiceException e) {
                        LOGGER.error(messageHelper.getMessage(MessageConstants.ERROR_TOOL_SCAN_FAILED, tool.getImage(), version), e);
                        toolManager.updateToolVersionScanStatus(tool.getId(), ToolScanStatus.FAILED, new Date(), version, null, null);
                    }
                }
            } catch (Exception e) {
                LOGGER.error(messageHelper.getMessage(MessageConstants.ERROR_TOOL_SCAN_FAILED, tool.getImage()), e);
                toolManager.updateToolVersionScanStatus(tool.getId(), ToolScanStatus.FAILED, new Date(), "latest", null, null);
            }
        }
    }
    LOGGER.info(messageHelper.getMessage(MessageConstants.INFO_TOOL_SCAN_SCHEDULED_DONE));
}
Also used : DockerRegistry(com.epam.pipeline.entity.pipeline.DockerRegistry) DockerClient(com.epam.pipeline.manager.docker.DockerClient) ToolVersionScanResult(com.epam.pipeline.entity.scan.ToolVersionScanResult) ToolScanExternalServiceException(com.epam.pipeline.exception.ToolScanExternalServiceException) Date(java.util.Date) ToolScanExternalServiceException(com.epam.pipeline.exception.ToolScanExternalServiceException) PipelineException(com.epam.pipeline.exception.PipelineException) Tool(com.epam.pipeline.entity.pipeline.Tool)

Example 4 with DockerClient

use of com.epam.pipeline.manager.docker.DockerClient in project cloud-pipeline by epam.

the class ToolManager method getCurrentImageSize.

public long getCurrentImageSize(final String dockerImage) {
    LOGGER.info("Getting size of image {}", dockerImage);
    Tool tool = loadByNameOrId(dockerImage);
    DockerRegistry dockerRegistry = dockerRegistryManager.load(tool.getRegistryId());
    String imageWithoutTag = getImageWithoutTag(tool.getImage());
    String tag = getTagFromImageName(dockerImage);
    DockerClient dockerClient = dockerRegistryManager.getDockerClient(dockerRegistry, imageWithoutTag);
    try {
        ToolVersion toolVersion = dockerClient.getVersionAttributes(dockerRegistry, imageWithoutTag, tag);
        if (Objects.isNull(toolVersion) || Objects.isNull(toolVersion.getSize())) {
            LOGGER.warn(messageHelper.getMessage(MessageConstants.ERROR_TOOL_VERSION_INVALID_SIZE, dockerImage));
            return 0;
        }
        return toolVersion.getSize();
    } catch (IllegalArgumentException e) {
        LOGGER.error("An error occurred while getting image size: {} ", e.getMessage());
        return 0;
    }
}
Also used : DockerRegistry(com.epam.pipeline.entity.pipeline.DockerRegistry) DockerClient(com.epam.pipeline.manager.docker.DockerClient) ToolVersion(com.epam.pipeline.entity.docker.ToolVersion) Tool(com.epam.pipeline.entity.pipeline.Tool)

Aggregations

DockerClient (com.epam.pipeline.manager.docker.DockerClient)4 DockerRegistry (com.epam.pipeline.entity.pipeline.DockerRegistry)3 Tool (com.epam.pipeline.entity.pipeline.Tool)3 ToolVersionScanResult (com.epam.pipeline.entity.scan.ToolVersionScanResult)3 ToolScanExternalServiceException (com.epam.pipeline.exception.ToolScanExternalServiceException)2 MessageConstants (com.epam.pipeline.common.MessageConstants)1 MessageHelper (com.epam.pipeline.common.MessageHelper)1 ManifestV2 (com.epam.pipeline.entity.docker.ManifestV2)1 ToolVersion (com.epam.pipeline.entity.docker.ToolVersion)1 ToolScanStatus (com.epam.pipeline.entity.pipeline.ToolScanStatus)1 ToolDependency (com.epam.pipeline.entity.scan.ToolDependency)1 ToolScanPolicy (com.epam.pipeline.entity.scan.ToolScanPolicy)1 Vulnerability (com.epam.pipeline.entity.scan.Vulnerability)1 VulnerabilitySeverity (com.epam.pipeline.entity.scan.VulnerabilitySeverity)1 DateUtils (com.epam.pipeline.entity.utils.DateUtils)1 PipelineException (com.epam.pipeline.exception.PipelineException)1 DockerClientFactory (com.epam.pipeline.manager.docker.DockerClientFactory)1 DockerRegistryManager (com.epam.pipeline.manager.docker.DockerRegistryManager)1 ClairScanRequest (com.epam.pipeline.manager.docker.scan.clair.ClairScanRequest)1 ClairScanResult (com.epam.pipeline.manager.docker.scan.clair.ClairScanResult)1