The Uncomfortable Truth of Vulnerable and Outdated Software Components
Learn where common industry sayings such as “stay up to date” come from and how you can help Endor Labs help you overcome those challenges.
Learn where common industry sayings such as “stay up to date” come from and how you can help Endor Labs help you overcome those challenges.
Learn where common industry sayings such as “stay up to date” come from and how you can help Endor Labs help you overcome those challenges.
Learn where common industry sayings such as “stay up to date” come from and how you can help Endor Labs help you overcome those challenges.
Learn where common industry sayings such as “stay up to date” come from and how you can help Endor Labs help you overcome those challenges.
Developers are increasingly turning to automated upgrading tools to keep their open source dependencies reasonably current. This trend is driven by the growing recognition that staying up-to-date is valuable, especially in light of OWASP's inclusion of "Vulnerable and Outdated Components" in their Top 10 list. However, it's important to remember that simply staying current is a means to an end. The real goals are twofold:
- Accelerated Innovation: By promptly adopting new features, developers can stay at the forefront of innovation.
- Efficient Problem Resolution: Staying updated allows for quick and cost-effective fixes for security vulnerabilities and bugs.
It's worth emphasizing that using older versions isn't always a problem. Sometimes, a stable, older version is perfectly adequate. However, if you choose to rely on unmaintained software, you must accept the responsibility for maintaining it – software reuse is not the same as outsourcing maintenance. Staying up-to-date can act as a preventative measure, for example:
- Addressing known issues: This is a clear and immediate benefit.
- Benefitting from community oversight: Maintained software has more eyes on it, increasing the chances of spotting and fixing issues.
- Mitigating future upgrade challenges: Staying current reduces the likelihood of encountering breaking changes that could delay crucial updates when addressing future vulnerabilities. This fear of breaking changes is a significant driver behind the emphasis on staying up-to-date.
Dependency updates often fail, keeping you on old versions
As we discussed in Breaking Changes, Breaking Trust, a major barrier to staying up-to-date is the risk an upgrade will introduce unexpected behavior. This is compounded by several factors: First, test suites often fail to provide a reliable safety net due to incomplete code coverage, especially for transitive dependencies. Second, developers frequently encounter "update fatigue" due to the sheer volume of updates and the perceived high cost-to-benefit ratio of upgrading. This reluctance is further fueled by concerns about breaking changes, the effort involved in understanding update implications, and the potential for complex migration processes.
Problem 1: The test coverage gap
As an industry, we rely on test suites and successful builds to detect breaking changes in updates. However, recent research has shown these strategies to be unreliable for several reasons. For example, 100% test coverage is an unrealistic practice at scale, so anything that isn’t tested could cause an unforeseen break. For example, Google's general guidelines for code coverage are:
- 60%: Acceptable
- 75%: Commendable
- 90%: Exemplary
A comprehensive study by Hejderup et al. [1] further reveals the limitations of using tests for dependency updates. Their findings show that in well-tested Java projects:
- Direct dependency calls: Only half the projects had test coverage exceeding 60%.
- Transitive dependencies: Test coverage plummeted to a mere 21%.
- Fault injection: On average, test suites detected only 47% of faults in direct dependencies and 35% in transitive dependencies.
These results highlight a critical gap in the effectiveness of current automated update strategies. The common assumption that project tests can reliably catch breaking changes in dependencies is often overly optimistic. Many projects suffer from incomplete test coverage, and breaking changes in dependencies can be subtle or occur in areas not directly exercised by existing tests.
Problem 2: Developer reluctance and update fatigue
Despite the availability of many tools to help, many developers remain reluctant to update dependencies. Mirhosseini and Parnin [2] conducted a study on developer attitudes towards automated dependency updates. They found that:
- When it comes to dependency management, 62% of developers said breaking changes were a top concern. Understanding the implications of changes and migration effort were also major concerns.
- Only 32% of automated pull requests for dependency updates were merged, indicating a high level of caution among developers.
- Developers reported feeling overwhelmed by the frequency of update notifications, leading to what the authors term "notification fatigue."
Building on these findings, Sawant et al. [3] and Kula et al. [4] further investigated the reasons for not upgrading dependencies. Their research revealed additional factors:
- Cost-benefit concerns: The perceived cost of upgrading often outweighs the benefits.
- Maintenance burden: Keeping up with updates is seen as a "full-time job."
- Complex upgrades: Some developers prefer switching to a different library altogether.
The real problem lies in the combination of inadequate test coverage, the complexity of dependency relationships, and developers' justifiable caution. While automated tools can help, they don't fully address the underlying issues of breaking changes and the effort required to understand and implement updates safely. As a result, library updates remain a significant dilemma for developers, requiring careful consideration and often manual intervention despite the availability of automated solutions.
While there is a general fear of introducing unexpected problems from our library dependencies, how do we move forward? We need to create mechanisms for improved transparency and visibility so that we can work more efficiently.
Endor Labs' two-step approach to breaking change analysis
When it comes to upgrading software dependencies, there’s always a trade-off between improving security and the risk of breaking existing functionality. At Endor Labs, we’ve developed a solution that minimizes these risks and ensures your upgrades are both secure and smooth. Let’s walk through how we do this using a two-step process.
Step 1: Identify potential breaking changes
Whenever you upgrade a software library from one version to another—say from version 1 to version 2—the first concern is that changes in the new version could cause parts of your application to stop working as expected. We identify these potential problem areas by comparing the public interfaces of the two versions and flagging the differences.
These differences are what we call breaking change candidates. They are changes in the new version that could cause your application to malfunction. From a business perspective, the risk here is clear: if the upgrade breaks a critical part of your system, it could lead to downtime, security vulnerabilities, or lost productivity.
However, just knowing what might break isn’t enough. What matters is knowing whether these changes actually impact your specific application.
For the technically inclined, this means that we are combining class hierarchy analysis with our reachability analysis to establish breaking changes that may affect your application.
Step 2: Use reachability analysis to identify which breaking change candidates are used
Here’s where we get more precise: after identifying the potential breaking changes, we use reachability analysis to determine whether your application actually uses the parts of the library that have changed.
Think of it this way: not every change in the new version will necessarily affect your software. For example, if an update in the library changes how a certain feature works, but your application doesn’t use that feature, it’s not a risk for you. This analysis helps us pinpoint the exact parts of your application that might be affected by the upgrade, allowing you to focus only on the areas that need attention. Unlike traditional diffing tools that merely provide a set of changes, this approach contextualizes those changes within your specific codebase, offering a more targeted and relevant assessment of potential impacts.
From a business perspective, this reduces unnecessary work, minimizes disruption, and ensures that upgrades can be prioritized effectively with the right expectations.
Again, if you want to get into the technicality of it, this means we stitch the updated call graph of the dependency in place of the current version to see what might break or change and calculate a confidence score based on how the client application interacts with the breaking change candidates.
Endor Labs’ call graph stitching shows how your code interacts
A call graph is a graph where the nodes represent functions and the edges represent calls between functions. We generate a call graph for each dependency, and we stitch together call graphs to provide a detailed map of how your application interacts with not just the library you’re upgrading, but all the other libraries that depend on it.
For example, your application might rely on multiple third-party components that themselves use the upgraded library. If a change in the new version causes an issue for one of those components, it could affect your entire system—even if your direct use of the library seems fine. This mapping of the entire dependency tree lets us see the full picture, ensuring there are no hidden risks.
By stitching together these interactions, we can also detect more subtle issues, like changes in how different parts of your application and its dependencies communicate with each other. These can introduce incompatibilities that are harder to spot without this in-depth analysis.
How is call graph stitching different from automated upgrade tools??
In comparison to automated dependency updating tools that rely on tests to identify breaking changes, program analysis offers several distinct advantages. Notably, it operates independently of test quality and coverage, enabling a more comprehensive assessment of potential impacts.
Consider the case of widely-used logging libraries like log4j (yes the example is overkill but I promise It’s being used reasonably this time). Many developers implement them out-of-the-box without extensive testing of their functionality. In the event of a vulnerability (such as log4shell), there may not be sufficient test coverage or effective tests to trigger regressions. Program analysis, however, leverages call graphs to approximate execution paths, providing insights even in areas lacking explicit tests and identifying whether there are paths to problematic or vulnerable code. This approach shifts the burden away from users, reducing the need to focus heavily on test creation and maintenance for dependency management purposes.
Program analysis provides a more comprehensive understanding of dependency changes. It offers a clear picture of the number of modifications between versions and, crucially, how many of these changes directly impact your project. This insight enables more accurate planning and estimation of refactoring efforts, particularly for major updates.
Unlike test-based approaches, program analysis delivers a more robust and nuanced evaluation of how library changes might affect your specific codebase. By highlighting affected areas and potential vulnerabilities, it empowers developers to make informed decisions about dependency management with greater confidence and precision.
Ready to change relationships between your security and development teams?
Upgrading software is a constant balancing act between staying up to date with security patches and ensuring your system continues to function without interruption. Endor Labs’ two-step approach—combining breaking change detection with reachability analysis and call graph stitching—gives you both security and stability. You’ll know exactly which parts of an upgrade affect your system, allowing your teams to address only the areas that matter, reducing downtime and helping to prioritize vulnerability updates with the right expectations.
This approach not only ensures smoother upgrades but also helps your business avoid costly disruptions that can arise from blindly upgrading software without understanding the full impact.
Book a demo to understand how Endor Labs turns your vulnerability prioritization workflows dreams into reality or start a full-featured free trial that includes test projects and the ability to scan your own projects.
References
[1] Hejderup, J., & Gousios, G. (2022). Can we trust tests to automate dependency updates? A case study of Java projects. The Journal of Systems and Software, 183, 111097.
[2] Mirhosseini, S., & Parnin, C. (2017). Can automated pull requests encourage software developers to upgrade out-of-date dependencies?. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) (pp. 84-94). IEEE.
[3] Sawant, Anand Ashok, Romain Robbes, and Alberto Bacchelli. "To react, or not to react: Patterns of reaction to API deprecation." Empirical Software Engineering 24 (2019): 3824-3870.
[4] Kula, Raula Gaikovina, et al. "Do developers update their library dependencies? An empirical study on the impact of security advisories on library migration." Empirical Software Engineering 23 (2018): 384-417.