-3-2.webp)
Container Scanning + SCA = Better Together
We’re excited to announce that Endor Labs now extends our software supply chain platform to include container scanning. To really understand our approach, we suggest starting with a quick dive into the strategies and challenges present in today’s market. But if you’re eager to see what we’ve cooked up, you can skip to the solution overview.
Container Security Strategies
Containerizing applications — creating a lightweight, portable application bundle that can be deployed on just about any infrastructure — is a staple of cloud-native software deployments. Containers carry not only the application itself, but all of its OS dependencies and libraries as well, allowing high portability and easy deployment without the overhead of a traditional virtual machine. This means that containers not only include the Open-Source Software (OSS) libraries that your application depends on, but the chain of OSS libraries and tools that are part of the bundled execution environment — and like your application dependencies, these container dependencies have their own risks like known vulnerabilities and malware.
To guard against software supply chain attacks and adhere to industry standards like FedRAMP, organizations generally use a variety of strategies:
- Container hardening, where the engineering team (or a 3rd party) reduces a container’s attack surface and makes it less vulnerable to exploits. Otherwise, developers end up building on top of vulnerable containers full of unnecessary dependencies, typically referred to as “bloated containers”. This involves removing unneeded components as well as adjusting configurations to reduce the likelihood of successful attacks.
- Container image scanning, where a security team uses tools that examine the contents of each container image (the base package that creates containers at run time) for risks, such as known vulnerabilities (like CVEs) or evidence of malicious code. This is a similar approach to that of Software Composition Analysis (SCA) tools, except that container image scanning is inspecting both OS- and package-level vulnerabilities. Also like SCA, this can be done through static analysis or at the runtime.
- Container registry hardening, where security and operations teams ensure that the library of deployable container images — the container registry — has proper controls to ensure that only authorized images are available there.
- Container deployment and runtime protections, commonly grouped under the banner “CNAPP”, where operational controls are placed in cloud environments to ensure that only authorized containers produced from approved images are deployed (admission control), that running containers are monitored for evidence of risky behaviors (adversarial behavior, serious defects, etc.), and that known risky behavior patterns are prevented where possible.
Types of Container Scanning Solutions
There are several ways a security team can implement container scanning, but most options make you compromise on either cost or coverage. Here we break them down by type with some pros and cons for each.
- Kubernetes platform or registry providers (e.g. Dockerhub)
- Pros: Free or low cost runtime container scanning within registries, low or no deployment required.
- Cons: Basic features and can’t detect risks until containers are already built and stored in the registry, increasing threat exposure.
- CNAPP vendors
- Pros: Consolidate with cloud security for reduced tool sprawl and costs
- Cons: Don’t detect risks until image is deployed
- AppSec vendors
- Pros: Scan during code/build SDLC stages and consolidate with other AppSec tools (e.g. SCA, SAST, secret detection) for reduced tool sprawl and lower costs.
- Cons: No runtime scanning and can lack robust coverage, for example if it only looks at the base image imported from the Dockerfile without visibility into the application packages inside the container.
Three Challenges with Container Scanning Solutions
The problems with container scanners are actually very similar to the problems with software composition analysis (SCA) tools:
Container Scanning Problem #1: Alert Fatigue
Because open source (OSS) application packages are subsequently included in containers, the container scanners create additional findings for the same source of issues, thereby increasing toil for teams involved. Being able to correlate findings across multiple scanners, requires connective tissue. Application security posture management (ASPM) tools try to solve this problem by consolidating alerts, but adding on another layer of tooling doesn’t always work — nevermind the added overhead of another tool.
Mo scanners, mo noise.
Container Scanning Problem #2: Insufficient Alert Context
As with application dependencies, there are direct and transitive container dependencies. But unlike with SCA findings, container scanners don’t usually differentiate between the two. This matters because vulnerabilities in container direct dependencies should absolutely get fixed, but there’s more nuance with transitive dependencies because the vulnerability might not be reachable. When you have no idea what’s important, you are left with three options to stay compliant and secure, all of which are lose-lose scenarios:
- Fix Everything: Spend a lot of time and money on fixing things that may not even affect you, and risks might get remediated in the order they are discovered or based on severity.
- Prioritize Based on Severity (CVSS) Alone: Spend resources fixing some “critical” things that don’t affect you, while leaving you exposed to the less-critical risks you didn’t have time for.
- Do Extra Research: Spend a lot of time and money trying to figure out what really matters, and you overextend your team.
Container Scanning Problem #3: Delayed Remediation
Delayed remediation can be caused by alert fatigue and insufficient context, which delay getting started on addressing risks. But there’s also a third barrier: traceability, the lack of which increases the time taken to finish. Organizations that identify container vulnerabilities after they’ve shipped to production have serious trouble tracing the artifact back to the source repo, release artifact, and development team. This substantially increases the time it takes to fix issues and runs the risk of making organizations become non-compliant with standards like PCI DSS, FedRAMP, etc. This is one of the ways CISOs end up in hot water.
Solve Container Challenges by Rethinking “What is a Dependency?”
You’re probably used to thinking of a dependency as the libraries that become part of your applications. But as we talked about in Surprise! Your GitHub Actions Are Dependencies, Too, the tools we use to build, test, and operate our applications are also dependencies (and part of the software supply chain). Container images (standalone, executable files that contain all the files, libraries, and dependencies needed to run a container) use OSS — and just like software application dependencies, they can be vulnerable to the OWASP Open Source Top 10. And just like any kind of dependency we have, we need visibility and a system for identifying and prioritizing risks.
At Endor Labs, we break dependencies into three categories that align to SDLC stages:
- Application dependencies are all the libraries and frameworks that your first-party code relies on. If you’re following current software development practices, the majority of these components will be OSS. This is classic software composition analysis (SCA) territory (though traditional SCA has some limits). Application dependencies can also be AI or ML models responsible for generating new first-party code.
- Build-and-Deploy dependencies are all the things you rely on to turn your first-party code into a distributable application. You could also think of this category as "workflow dependencies”, and it can include GitHub Actions, pre-built GitLab Steps, compilers…you name it! Management of these components is an emerging requirement for organizations wanting to prevent anything from breaking builds to an adversary using a compromised Action to insert malicious code in the application you just built (and will now be deploying to production or distributing to your customers).
- Operational dependencies put the “Ops” in “DevOps”. These are components needed to operate your applications in production. It includes things like container images, operating systems, cloud configurations and management tools, application servers (like WebSphere), and so on. Your application depends on these to operate safely, so your supply chain threat model has to include them.
Once you agree that all of these items are “dependencies”, then the next step is to take a hard look at the concept of dependency management. We’ll go into that topic more in another blog, but for now here’s the TL;DR: An effective dependency management program reduces toil and noise by consolidating scanning and prioritization activities (whenever possible) for all types of dependencies.
Container Scanning with Endor Labs
At Endor Labs, we think the best way to incorporate containers into a dependency management program is with a combination of best-in-class dependency management (Endor Labs) and runtime container scanning tools (e.g. Wiz, Prisma Cloud), as follows:
- Code: Endor Labs scans the source project, dependency tree, and application artifacts
- Build: Endor Labs scans the base image and application dependencies, then signs the result
- Deploy: Endor Labs verifies the signature on, then scans the published container image; signature and scan results can be consumed by CNAPP/admission controllers
- Run: Runtime container scanner/CNAPP monitors the operating containers in production
- Respond: If your CNAPP detects a new issue in production, Endor Labs quickly traces the affected container back to the build and code that generated it, speeding response.
This solution offers the benefits of bundling (i.e. an affordable price tag) alongside the benefits of a robust tool. Container scanning with Endor Labs solves the Three Challenges with Container Scanning Solutions while supporting a holistic dependency management program. With Endor Labs you can:
- Find container risks sooner with pre-deployment scans
- Reduce alert noise with deep visibility
- Accelerate remediation with insights
- Establish container provenance with artifact signing
- Support compliance obligations with SBOM and VEX
Find Container Risks Sooner with Pre-Deployment Scans
Don’t wait until your images are deployed to find out they’re vulnerable!
Endor Labs supports scanning both the base operating system (OS) and application packages for container images. By scanning before an image is deployed, you can identify images with known vulnerabilities when it’s easier to do something about it — and prevent those risks from being exploited.
Reduce Alert Noise with Deep Visibility
Reduce the need for ASPM with a comprehensive view of your OSS vulnerability exposure across all your dependencies.
As shown in the following gif, you can easily group findings by vulnerability (e.g. GHSA-xh6m-7cr7-xx66 and GO-2022-0523) and immediately see all the impacted dependencies — libraries, containers, and more. And this isn’t just in the UI! PR comments will also include all the vulnerabilities in the same comment, so the developer can quickly see what needs fixing, and every bit of data is available via our ReSTful API thanks to our “API-first” design philosophy.
As with our application findings, container findings provide deep visibility required to reduce noise and prioritize next steps. Each finding includes layer annotations for container images, telling you exactly which layer the dependency was introduced in. And each container has a dependency graph illustrating how the impacted dependency relates to your container. This visibility helps you quickly triage vulnerabilities in both base image and application packages since you know what is directly imported versus transitive.
Accelerate Remediation with Insights
Reducing noise with visibility and prioritization is the first step to accelerating remediation; after all the faster you figure out what needs fixing, the faster you can get on with remediation. But don’t stop there!
There are several capabilities included with Endor Labs that shorten mean time to repair (MTTR), including:
- Container traceability— When a risk is discovered in a running container can be hard to trace back to the source. Using artifact metadata, you can quickly identify where the container came from and assign ownership to the right team or individual.
- Remediation recommendation— Automatically provide recommended upgrade path to remove the vulnerable code.
- Searchable intelligence— Quickly discover relationships, context, and organization-wide impacts through high-speed searches across your application footprint; pull important context data from our ReSTful API into SIEM, business analysis, and issue aggregation tools.
Establish Container Provenance with Artifact Signing
Artifact traceability is a new area of focus for us in 2024 because we’ve heard time and again that security teams need to determine provenance — that is, reliable data connecting an artifact to its source code and build pipelines — in support of admission control, troubleshooting, remediation, and compliance efforts. For example, organizations seeking SLSA Level 2 must be able to attest to container (and other artifact) provenance. With containers signed by Endor Labs, all that manual effort (of tracing problems back to the source repo, release artifact, and development team) just evaporates.
When Endor Labs is deployed into your existing build pipeline and the pipeline produces an artifact, it makes a signature request to Endor Labs. The signature and the transparency log data are stored into the tamper-resistant Endor Labs database for later verification and audit. Read Signing Your Artifacts For Security, Quality, and Compliance to learn more about this process and use cases.
Support Compliance Obligations with SBOM and VEX
Artifact signatures are important for establishing whether an artifact has gone through security processes (such as container scanning) required by various regulations. But also important — and required by several regulations and frameworks including PCI DSS, DORA, and the FDA’s Ensuring Cybersecurity in Medical Devices — is the ability to attest to an accurate software inventory.
With one click, you can generate a complete, accurate container software bill of materials (SBOM) for OS and application dependencies (in SPDX or CycloneDX format) and a Vulnerability Exploitability eXchange (VEX) document with details about each vulnerability, such as its severity, the software it affects — or, more importantly, doesn’t affect. While there are point solutions that can provide some of this value, many struggle with accuracy. Endor Labs includes SBOM and VEX generation at no extra cost, helping you further consolidate your security tool stack.
Build an Open Source Security Program with Endor Labs
Contact us to join the container scanning early access program!
In addition to container scanning, Endor Labs Open Source includes:
- Reachability-Based SCA: Prioritize the handful of vulnerabilities that actually matter, and help developers manage the security and health of their direct and transitive open source packages.
- Artifact Signing: Enable application and container provenance to enhance admission control, incident response, and other essential capabilities.
- AI-Assisted OSS Selection: Curate OSS components with 150+ checks and scoring based on security, popularity, activity, and quality.
- SBOM & VEX Generation: Produce accurate SBOMs with automated VEX for your applications and containers.