Introducing Upgrades & Remediation: Give Developers the Confidence to Fix
Upgrade Impact Analysis shows you what breaking changes a fix could cause. Endor Magic Patches are trusted patches you can use when upgrades are too painful. Your SCA tool should do more than show you problems: It should give you solutions.
Upgrade Impact Analysis shows you what breaking changes a fix could cause. Endor Magic Patches are trusted patches you can use when upgrades are too painful. Your SCA tool should do more than show you problems: It should give you solutions.
Upgrade Impact Analysis shows you what breaking changes a fix could cause. Endor Magic Patches are trusted patches you can use when upgrades are too painful. Your SCA tool should do more than show you problems: It should give you solutions.
Upgrade Impact Analysis shows you what breaking changes a fix could cause. Endor Magic Patches are trusted patches you can use when upgrades are too painful. Your SCA tool should do more than show you problems: It should give you solutions.
Upgrade Impact Analysis shows you what breaking changes a fix could cause. Endor Magic Patches are trusted patches you can use when upgrades are too painful. Your SCA tool should do more than show you problems: It should give you solutions.
You’ve probably seen this scenario play out… Your SCA tool discovered a reachable vulnerability. It’s a critical severity and you prioritize getting it upgraded ASAP.
Engineering gets into a war-room and decides to push back because they’re uncertain about the level of effort involved in the upgrade. Or perhaps they attempt the upgrade but something goes wrong…it’s a breaking change! Maybe you luck out and it’s an easy fix, or maybe the app is down for days. Everyone’s stressed out and they’re probably mad at you because you’re the one who imposed the upgrade with urgency. The outage could have been avoided if someone had manually researched the upgrade impacts, but either they didn’t have time (it was critical!) or didn’t have the skills.
You’re left wondering “What’s the point of my AppSec program if I can’t get anything fixed?”
The Cost of Remediation: Developer Productivity
Determining the impact (effort, risk reduction, ROI) of remediation requires manual research by developers which takes time away from developing software for business value and it extends the amount of time it takes to fix vulnerabilities. In fact, developers hate dependency management so much that they’ve coined the term dependency hell! Sometimes this dependency hell even requires code changes because one or more dependencies that are part of the upgrade changed enough that the way you use them no longer works right.
It's so painful, some organizations have adopted the “upgrade only when necessary” philosophy. Software updates have become a de facto tax on software reuse and regardless of your philosophy, you have to pay a tax when you update. Some organizations incrementally pay that cost by staying as up-to-date as reasonably possible on their software as an operating cost. Others pay it in a lump sum to address an issue.
This developer productivity tax is caused by two remediation challenges:
Pain Point 1: Unintended Consequences of Upgrading
We looked at 1250 patch-to-fix vulnerabilities in our customers’ environments and found that nearly all of them contained breaking changes.
- 24% of vulnerability fixes required a major version upgrade to remediate, which by nature will have breaking changes.
- Of the remaining 76% that could be remediated by upgrading to a minor or patch version, 94% also contained at least one potentially breaking change.
Understandably, developers are often afraid to make upgrades because they don’t know what will impact the application…and no one wants to be responsible for breaking something that the business relies on! This justifiable fear slows down the resolution of vulnerabilities.
Pain Point 2: Upgrading Foundational Packages Can Take Months or Years
Some OSS packages are so foundational to applications that upgrading them is no easy feat. In such cases, development organizations may continue on old and unsupported versions for years to come, rather than accept the tax of upgrading. While it’s an engineering best practice to stay at the latest and have a CI/CD pipeline to test everything, in reality most organizations can’t meet this bar for various reasons that involve significant technical debt or major incompatibilities. In practice it’s typical to upgrade only when absolutely necessary. In these cases, often the cost of upgrading is too great (e.g. introduces too many breaking changes, performance degradations ), and the organization accepts the risk of the vulnerability rather than breaking business-critical apps. This exposes the organization to breaches and compliance failures.
To illustrate this problem, we took a look at our customer data for projects using Spring. We found that 82% of customer projects are still using the 5.x series, with the majority being at Spring 5.3.27 (the latest of the 5 series) and the second-most-commonly used version is 5.3.23. There are numerous vulnerabilities in the 5.x series, but upgrading to 6.x involves new minimum requirements, removed APIs, and other changes that make this a very painful project. It’s no wonder so many of our customers haven’t upgraded yet! That’s why these customers are establishing month-to-year long projects to remediate these issues.
Fix What Matters, Even When It’s Hard!
At Endor Labs, we think your SCA tool should do more than show you problems: It should also give you solutions. We’re excited to share new capabilities that help you fix what matters, even when it’s hard. With upgrade impact analysis, you better understand the impact of dependency upgrades, enabling more informed remediation conversations with engineering. And in the event that the upgrade is too costly (risky/complicated/effort intensive), you can use Endor Magic Patches to stay safe with a minimal patch that reduces the changes down to just what’s required to eliminate the vulnerability, and therefore the operational risk of changing versions. You can make it easy for developers to do the work and lower the risk of upgrades breaking your apps, while still reducing vulnerability risks to your organization.
A special shoutout goes to the brilliant engineers who are the backbone of this announcement. To Alexandre Wilhelm, Sebastian Cai, Henrik Plate, George Apostolopoulos, Joseph Hejderup, and Philip Hamer: Your relentless dedication and expertise have made this effort a reality. Thank you for your hard work and commitment!
Upgrade Impact Analysis: Know How Painful an Upgrade Will Be
With upgrade impact analysis, we help you manage how you pay your upgrade taxes and identify the updates that will be more costly. By extending our program analysis engine to identify unintended consequences such as breaking changes to your application you can now manage risk in the context of difficulty. By understanding how various fix options will impact your application, you can:
- Improve Return on Investment of Remediation Efforts: Identify which upgrades can have the highest security impact in conjunction with the effort it takes.
- Give Time Back to Developers: Reduce the need for manual research by providing developers with a prioritized list of upgrades ranked by complexity and impact.
- Address Risks Faster: Make informed estimations of fix efforts with standardized research so you can quickly implement low effort/low risk fixes and make prioritization decisions for complex fixes.
Endor Magic Patches: Mitigate Vulnerabilities When Upgrading is Too Painful
Some upgrades will be more expensive than others. This is a cost we accept to innovate quickly by adopting open source code. Yet your C-Suite wants it fixed now…what are your options? You can accept the risk, you can accelerate the issue’s resolution, or you can put in compensating controls to reduce it. With any of these approaches, you’ll suffer hours of arguing priorities.
With Endor Magic Patches, we eliminate the hassle of hard-to-perform upgrades by providing security patches we backport to your vulnerable version for your security and convenience. Because it’s a minimal patch, it just contains changes to address vulnerabilities, so you can turn a high effort investigation into something simple.
But we believe that magicians should reveal their tricks. With security risk, trust and transparency is paramount. The source code, patches, test, build and deploy steps are available to you. Our builds are completely reproducible and hermetic. You can reproduce them with steps we can provide. That way you can see exactly what you are getting.
By applying an Endor Magic Patch you can:
- Respond to Emerging Threats: Be ready for the next Spring4Shell with peace of mind that you can obtain a patch to ensure you stay safe while you work to upgrade dependencies.
- Balance Developer Workloads: Reduce the urgency of upgrading so you can let developers focus on releasing their planned features without unexpected delays.
- Support FedRAMP Compliance: Mitigate vulnerability risk to protect sensitive information in alignment with government requirements.
Using Endor Labs to Identify Update Risk and Simplify Upgrades
With an understanding of your software bill of materials (SBOM) and a roadmap on how your code interacts with your software dependencies, Endor Labs builds a database of evidence of how your software interacts with vulnerable code. This is the secret sauce behind our function-level reachability, and this same information can be used to help you understand risks associated with vulnerability remediation because we can identify:
- Likely breaking changes (through removed or modified code) associated with upgrades
- Version conflicts that exist in your application, which can increase the risk of build failures from version constraints.
All this information is useful by itself. But to really speed up how you address risk, you need to be able to prioritize which upgrades to do first. We perform an analysis to estimate the risk of each upgrade — for your application specifically — and we rank upgrades based on the evidence we have that the task will be non-trivial:
- High Risk = Upgrading is very likely to break your application at multiple points
- Medium Risk = Proceed with caution: upgrading has a moderate chance of breaking your application
- Low Risk = you're probably good: upgrading has a very low chance of breaking your application
With an informed understanding of risk, your organization can make faster decisions on what to remediate now (or soon) vs. what to mitigate using Endor Magic Patches. To illustrate this process we’ll look at two examples:
- A low complexity upgrade for postgresql
- A high complexity upgrade for Spring
Remediation Example: A Low Complexity Upgrade (postgresql)
In this scenario, we’re using version 42.6.0 of postgresql, which is vulnerable to SQL Injection via line comment generation. Endor Labs recommends upgrading to version 42.6.1 to remediate the vulnerability and gauges this to be a “low” risk upgrade because we found no evidence of breaking changes. This upgrade should be added to the remediation backlog or addressed immediately.
Mitigation Example: A High Complexity Upgrade (spring)
Take for example a major upgrade of org.springframework:spring-core from 3.2.0.RELEASE to 5.3.31. It fixes several vulnerabilities, which may be reachable depending on your application. But there are 37 potential breaking changes identified and Endor Labs has high confidence that 12 will be impactful based on our analysis of your program. You can see the breaking changes and our level of confidence in them directly associated with the upgrade. Even if you’re unfamiliar with Spring, this analysis tells you that this remediation won’t be a trivial task. It’s going to require some grit to accomplish. And if you do know Spring, you were crying as soon as this upgrade hit your inbox (even before you saw the evidence)!
In this scenario, it could take months or years to upgrade Spring. Instead, you can mitigate using Endor Magic Patches in three simple steps:
- Connect to the Endor Magic Patch repository
- Update your manifest to the Endor Labs version of your package.
- Watch your vulnerabilities disappear the next time you build. It's like magic.
Get Started with Endor Labs
We think you should expect more from your SCA and we’re here to make it a reality. When you use Endor Labs, you can:
- Find the vulnerabilities that matter most with reachability analysis,
- Prioritize which risks to remediate using upgrade impact analysis, and
- Patch those hard-to-upgrade packages with Endor Magic Patches.
Book a demo to see how Endor Labs helps you secure everything your code depends on.