How Endor Patches Are Built and Tested
Endor Patches are backported open-source security fixes. Learn how we build and test Endor Patches for compatibility and security.
Endor Patches are backported open-source security fixes. Learn how we build and test Endor Patches for compatibility and security.
Endor Patches are backported open-source security fixes. Learn how we build and test Endor Patches for compatibility and security.
Endor Patches are backported open-source security fixes. Learn how we build and test Endor Patches for compatibility and security.
Endor Patches are backported open-source security fixes. Learn how we build and test Endor Patches for compatibility and security.
Every time a vulnerability strikes an open source project, downstream developers are left wondering how to fix the vulnerability without breaking everything else. Updating to the latest non-vulnerable version can result in regressions and incompatibilities—the older the application and its dependencies, the higher the risk.
At Endor Labs, we understand this challenge. That’s why we created Endor Patches—to help security and development teams navigate vulnerabilities with confidence.
They help teams react quickly to emerging threats, buying time to plan and execute complex upgrades. Through focusing on rigorous testing, transparency, and minimal changes, we ensure patches you can trust, reducing security risk without disrupting software operations.
Guiding Principles to Deliver Secure, Minimal Viable Fixes
Endor Patches are backported fixes designed to address vulnerabilities in older versions of open source components while leaving everything else untouched. These minimally viable security patches prioritize:
- Minimal changes: Provide only what’s necessary to fix the vulnerability.
- Maximum compatibility: Avoiding breaking changes that disrupt workflows.
- Focus on security: No attempts to fix non-security bugs or other functionality.
This approach is well-known from Linux distributions like Debian or Ubuntu, which follow comparable guidelines when downporting security patches. As Debian put it in their security FAQs, “most important [...] when making a new package that fixes a security problem is to make as few changes as possible”.
Building Patches: From Simple to Complex Backports
Ideally, the upstream version of a patch can be applied with no or little modifications to previous component versions, and Endor Labs follows this approach whenever possible.
However, “simply” taking the upstream version of a patch is sometimes impossible, e.g. when significant refactoring happened between the upstream version that was fixed by the project maintainers, and the older version to which it should be backported.
But that doesn’t mean that a patch is impossible, just that it requires custom development, as shown by the following examples.
Example 1: Fixes may depend on code not present in older versions
CVE-2023-50164, an HTTP parameter pollution vulnerability in Apache Struts, was fixed in 2.5.x and the 6.x.x series of Apache Struts. However, it was not fixed in versions 2.3.x or earlier, because they reached end of life, hence, were no longer maintained.
The difficulty of downporting the fix to 2.3.x versions is that the fix for 2.5.33 and 6.3.0.2 consisted of modifying the class “HttpParameters”, which is not present in older 2.3.x releases.
In cases like this, fixing vulnerabilities may require a custom solution.
Example 2: Fixes may be implemented through the removal of functionality
In the case of Spring Web, vulnerable functionality wasn’t fixed, but rather it was deprecated and removed, rendering a non-breaking backport impossible.
Java's serialization mechanism has been used by several Spring communication APIs since the early days, long before Java deserialization vulnerabilities became well known in 2015, following a prominent vulnerability in Apache Commons.
However, as general awareness of deserialization vulnerabilities grew, the use of Java’s built-in mechanism was considered risky and discouraged in conjunction with untrusted data, which led to the publication of CVE-2016-1000027 for the Spring Framework. Following this advisory, the corresponding APIs were deprecated with Spring 5.3 (which is not a fix, but “only” results in a warning message for developers) and entirely removed as of Spring 6.
The removal of those communication APIs represents a breaking change, which makes upgrading to Spring 6 a difficult endeavor for the many developers depending on this feature. Such cases also require a custom fix, namely the development of a custom sanitization mechanism and its integration into the affected communication APIs of older Spring releases.
Testing Patches: Meeting Security and Developer Expectations
Security teams need vulnerabilities fixed so that they know they aren’t exposed to risk. But development teams need security patches to be backwards compatible so they don’t have to expend significant effort to fix a vulnerability.
These two fundamental expectations can conflict, which is a foundational source of friction in vulnerability management. The constant rate of change in the open source ecosystem makes staying up to date an ongoing and costly tax on development. Those who don’t pay that cost can find themselves at odds when a bug fix or security vulnerability forces them to pay the cost of upgrading in one lump sum.
At Endor Labs, we understand the importance of meeting the requirements of both developers and security teams. Therefore we build and test Endor Patches to:
- Resolve the respective vulnerability without introducing new ones.
- Ensure backwards compatibility. In other words, limit breaking changes!
- Provide transparency to establish trust in what you’ve done.
A patch must meet these requirements to serve as a drop-in replacement of a vulnerable component.
The vulnerability must no longer be exploitable
The high-level process used to backport and validate each and every vulnerability fix in a given component version comprises the following activities:
- Endor Labs’ security research team identifies the fix commit of the upstream repository and — for each commit — the minimal set of required function changes. This is necessary, as fix commits often contain auxiliary changes. Upstream commits are applied as-is where possible, in order to align as closely as possible with authoritative upstream changes.
- Following, we check whether the upstream commits also comprise fix-related test cases, which is often, but not always the case. If no test cases have been provided, or if we had to develop a custom fix, we decide on a case-by-case basis whether to develop custom test cases. This decision is primarily driven by the complexity of the patch, and whether it deviates from the original fix.
- Finally, we developed a light-weight, container-based framework to automate running PoC exploits for given vulnerabilities against both the vulnerable version and the patched version of the open source component: The former validates the PoC, while the latter demonstrates that the PoC does not work any longer on the patched version.
The patch must be backwards compatible
To minimize the risk of producing incompatible patches, we set the following key guidelines for all of our build processes:
- Prefer the original fix commit from the upstream project over home-made, custom fixes.
- When original fix commits cannot be applied as-is, pick a minimal set of parent commits to make them applicable.
- Use the same build tooling (Maven & JDK 8 for example) that was used to build the vulnerable version of the artifact.
Once a patch is created, it is tested to assure its functional compatibility with the previous version, whereby we consider both syntactic compatibility and semantic compatibility.
Syntactic compatibility concerns the public API of a software component. Changes such as the removal of classes, functions or the modification of function parameters can lead to compilation or runtime errors of client code – also called breaking changes. According to Ochoa et al., method removals are by far the most frequent reason for breaking changes [Ochoa 2022, Fig 5.2], but many more reasons exist.
When developing Endor patches, we check the syntactic compatibility by comparing the vulnerable and patched artifact using an automated change impact analysis solution we call upgrade impact analysis. In case there are backward-incompatible API changes, we revisit the patch to minimize or entirely eliminate such changes. This makes sure that the code of client applications, or that of other dependencies, works well with the patch. Further, when we perform upgrade impact analysis on our customers’ direct dependency upgrades, we surface if the Endor Patch contains any API-incompatible changes.
Semantic compatibility, on the other hand, is much more difficult to handle. In fact, the semantic equivalence of two non-trivial software programs (component versions in our case) cannot be formally proven according to Rice’s theorem. This holds true no matter where those versions are coming from – the upstream open source project, Endor Labs or from anywhere else.
The best option to produce and test equivalence is to run extensive tests. To this end, we make sure the test suite of the upstream project succeeds on the vulnerable and patched versions.
The fix must be transparent
Consumers of Endor Patches need to trust that patches fix the vulnerability without introducing any regressions or breaking changes into their application, and they need to understand what has changed in the patch. To this end, we meet this requirement by running a comprehensive test suite before shipping a patch, including both manual and automated quality gates.
The actual patch, build and test commands as well as all other process inputs and outputs are made available to consumers. As such, consumers do not only get visibility into the patch, build and tests performed by Endor Labs – but can reproduce and re-run the very same steps on their end.
Test Endor Patches yourself
Trust and transparency is paramount to everything that we do and we hold ourselves to the highest standards of transparency. This is the only way to build trust and create a testing strategy that truly builds confidence with our customers and our community.
Read our whitepaper to learn more about how Endor Patches work, and how you can deploy them in your organization.
