.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
Remediating Vulnerabilities vs. Maintaining Current Dependencies
Learn about the pros and cons of maintaining current dependencies, backed by a TU Delft study on 262 Java projects on Github.
Learn about the pros and cons of maintaining current dependencies, backed by a TU Delft study on 262 Java projects on Github.
Learn about the pros and cons of maintaining current dependencies, backed by a TU Delft study on 262 Java projects on Github.
Learn about the pros and cons of maintaining current dependencies, backed by a TU Delft study on 262 Java projects on Github.
Learn about the pros and cons of maintaining current dependencies, backed by a TU Delft study on 262 Java projects on Github.
Maintaining up-to-date open source (OSS) dependencies is often touted as a best practice in software development, and it’s not uncommon to hear “keeping your dependencies up to date” is the solution to vulnerability management. However, the reality of implementing this advice is far from straightforward. The act of updating dependencies encompasses a myriad of challenges, extending beyond the mere technical to include operational, security, and compliance considerations. This nuanced exploration delves into the intricacies of dependency updates, supported by recent studies, to uncover the multifaceted risks and strategies for effective management.
I recently spoke at FOSDEM 2024 on this topic. In this blog I’ll summarize my research findings and you can watch the 26-minute lecture for more details.
Understanding Stability and Compatibility Risks
At the heart of dependency management lies the risk of introducing instability or compatibility issues to your project. Despite an active development team and testing suite, updating dependencies can inadvertently disrupt existing functionalities, leading to performance degradation, breaking changes, or elusive bugs. This scenario is exacerbated in the face of complex library interdependencies, often resulting in the dreaded "dependency hell”.
Recent research, including the comprehensive study BUMP: A Benchmark of Reproducible BreakingDependency Updates (analyzing 571 breaking dependency updates across 153 Java projects), categorizes the primary challenges into two:
- API changes, which may be flagged by compilers, allowing for manageable refactoring; and
- Behavioral modifications of dependencies, which are insidious, requiring extensive testing to detect.
But Are Our Tests Sufficient?
Members of our research team at Endor Labs have experience with answering this very question. The team undertook an academic study (Can we trust tests to automate dependency updates? A case study of Java Projects) at TU Delft which included analysis of 521 GitHub projects and concluded that the tests included in these projects rarely cover the full functional use of dependencies. The research involved identifying all of the functions which were reachable from first-party code, instrumenting these functions (so we know if the tests reached them) and then verifying if the project’s test suite invoked the dependency functions. The analysis concluded that the median test coverage for these functions in direct dependencies was only 60%. The situation with transitive dependencies (dependencies of dependencies) was even worse at only 20% of functions reached by the tests. It’s important to remember that this research was also based on public GitHub projects which are typically built by a community of conscientious maintainers. The chances of having good test coverage in a corporate environment may be even lower.
But What about SemVer?
Semantic Versioning (SemVer) proposes a structured approach to versioning that promises to mitigate the risk of breaking changes. However, its effectiveness is contingent upon strict adherence, which, as studies like Breaking Bad? Semantic Versioning and Impact of Breaking Changes in Maven Central reveal, is not always observed. As an example, up to 20% of Maven projects have been found to inconsistently implement SemVer, undermining its potential as a safeguard against compatibility issues. It’s also often the case that a vulnerability often requires a minor or major update to resolve, meaning some refactoring of code is inevitable.
The Hidden Dangers of Malware
Another overlooked aspect of updating dependencies is the potential for making things worse. If developers don’t pin their dependencies and their build tools automatically grab later versions of a library, then they may be introducing more risk than they attempt to remediate. It's also worth noting that malware is generally not covered by CVEs (as it's an intentional attempt to compromise a library) malware might not be visible in many SCA tool alerts.
For example, in 2018 it was discovered that the `event-stream` package on npm had been compromised. A malicious actor gained the trust of the original maintainer of event-stream, who then transferred the ownership of the package. The new owner added a dependency called `flatmap-stream`, which contained malicious code and this was downloaded millions of times before the malware was detected a few months later.
If you’re interested in reading more, check out Small World with High Risks: A Study of Security Threats in the npm Ecosystem. This paper investigates the npm ecosystem, highlighting how its expansive array of third-party packages and limited maintainer oversight lead to significant security vulnerabilities, including the potential for widespread use of vulnerable or malicious code.
Compliance Risks Amidst Licensing Changes
Developers who routinely update dependencies without reviewing changes may also overlook license modifications, like HashiCorp's Terraform shift from MPL 2.0 to BSL 1.1 in August 2023. This change, aimed at tighter commercial control, restricted certain uses in commercial settings without a purchased license, potentially catching some developers off guard. Thankfully these changes in licenses tend to only occur with major version updates.
Balancing the Opportunity Cost
Last but certainly not least we should also consider the opportunity cost of upgrading dependencies. Every time we ask a developer to update a dependency we’re taking them away from delivering new value to the business. In some instances, this might be justified but for example if they’re being asked to update dependencies which are only used in test suites then the risks were extremely low. Likewise, if we’re asking developers to move off a known, reliable and stable version of a library, we should consider whether the application is making use of the vulnerable code before causing a fire drill.
What We Recommend
In managing dependencies, our guidance at Endor Labs focuses on ensuring both security and ease of maintenance:
- Don’t Blindly Trust Your Tools: If you’re working with projects with limited test coverage it’s recommended that you perform a full regression test to ensure you haven’t inadvertently introduced new bugs by updating dependencies.
- Test Third-Party Functionality: Treat external libraries as if they were your own code by writing tests for their critical functionalities. This helps catch any regressions that may arise from updates, safeguarding your application.
- Use Mocking Judiciously: While mocking is useful for isolating tests, ensure you also have tests that run without mocks to fully assess third-party integrations. This strategy helps identify issues that mocks might conceal, enhancing the reliability of your application.
- Evaluate Library Size and Dependencies: Consider the size and additional dependencies of third-party libraries against the functionality they provide. If a library is too large for the benefits it offers, it might be more efficient to develop the needed functionality in-house, reducing complexity and potential security risks.
By implementing these strategies, teams can better manage their application dependencies, leading to more secure and maintainable software.
Prioritizing Updates with Reachability-Based SCA
So while it might seem ideal to keep all your dependencies up-to-date, in practice it causes more work and potential problems than a risk-based approach. Reachability-based SCA can help you quickly identify dependencies in need of upgrade - whether that be because of a vulnerability or some operational problem - without the noise for which security tools are notorious. Endor Labs is available to try free for 30 days. Start now or contact us to discuss your use cases.