XZ Backdoor: How to Prepare for the Next One
Over the last week I’ve heard the question “How do I prevent my organization from being exposed to open source supply chain attacks like the xz backdoor?” dozens of times. While it’s unlikely an organization can effectively prevent any exposure to supply chain risks, organizations can absolutely focus on a strategy to reduce the probability that a supply chain attack is successful, and reduce the time it takes them to respond to such threats.
Open Source software reuse has become a nine-trillion dollar public resource that has dramatically accelerated innovation. Making the choice to not use open source would be a choice to set yourself at a competitive disadvantage. Yet open-source adoption is not without risks, as we saw again from this recent backdoor discovered in the popular xz-utils package.
Background on the XZ Backdoor
The open source library xz is a general-purpose data compression library used by many applications to shrink files and data messages to save on storage and network costs. An adversary was able to compromise the build of one of xz’s components in a way that introduces a backdoor into common Linux systems that run an ssh server which uses the systemd init manager and glibc. Compromised systems allow the adversary to upload and execute code using administrative privilege level.
Thanks to the efforts of Andres Freund from Microsoft, who discovered the attack while attempting to understand an SSH performance problem, this vulnerability was caught before there was widespread adoption of the compromised xz version.
Wired has a great general overview of the issue and its impact, and a detailed writeup on GitHub provides an in-depth technical discussion.
This supply chain attack uses a well-known vector (AV-500 in Endor Labs’ Risk Explorer for Software Supply Chains) in which an adversary hijacks a legitimate package and distributes a malicious version. This particular instance is one of the most mature open source supply chain attacks discovered to date, in that it was well-obfuscated, relies on component interaction (thus resisting many types of review and analysis), and involved the identity of a legitimate maintainer (it is unknown as of this writing if the maintainer was always malicious and spent time building trust, or if an adversary compromised the maintainer’s accounts).
How Can You Reduce Your Risk of Consuming Supply Chain Attacks?
Recognize They're Here to Stay and Educate Yourself
The proliferation of open source software has made open source supply chain attacks one of the quickest and most scalable attack patterns.
When you choose to reuse software you also implicitly trust all other software that their maintainers have chosen to reuse. When you trust one software package you often implicitly choose to trust a large number of software packages. Our analysis of some of the most important open source packages in the world in the Endor Labs State of Dependency Management report shows just how rampant this problem is. For example, when you trust one Maven package, on average there are an additional 14 dependencies you implicitly trust as a result. This number is even larger in certain software ecosystems such as NPM where you on average import 77 other software components for every one you trust. This trust relationship is established with all maintainers of all software components. This problem can be mitigated with regular software audits, but to ensure that this is not a cost-prohibitive effort most organizations need to rely on the group collective efforts of the open source community. The xz backdoor discovery is a significant example of the advantages of the open source model to enabling discovery of supply chain attacks.
These types of attacks are difficult to detect, but leading indicators of risk like Endor Labs’ Score Factors can be used to target analysis efforts and reduce the chances that risky packages are introduced into an organization’s environment. Additionally, artifact signing provides some resilience to this type of attack by ensuring that only authorized pipelines can produce valid, signed artifacts. While this won’t provide perfect protection, it significantly raises the cost for an adversary to get malicious packages adopted, and also aids in effective incident response by allowing responders to quickly and accurately block use of the malicious package once it is identified.
For those who want to educate themselves on supply chain attacks, a full taxonomy of attack vectors and compensating controls for open source supply chain attacks can be found in Endor Labs’ Risk Explorer for Software Supply Chains.
Focus Efforts on Improving Your Response Effectiveness
Foundational controls such as maintaining a mature software inventory remain one of the highest value programs you can have in place to quickly identify, scope, and respond to software risks once they’re identified.
This software inventory should be not just for the software your organization builds but also the software your organization procures. Using SCA solutions; deploying network vulnerability scanners, EDR/XDR systems, and other operational controls; and maintaining an inventory of SBOMs for the software you deploy and rely on can help you to quickly identify where you rely on a compromised software component, such as xz. Organizations should focus their efforts on maintaining visibility into their software inventory so that when a headline hits and the board or executive team demands to know their risk exposure you can answer quickly and begin planning your remediation, rather than conducting a fire drill and emailing all your vendors asking if you are potentially impacted.
Combined with operational controls like artifact signing and admission control, a good software inventory can also ensure that once an issue is remediated, it stays remediated; admission controllers can ensure that unsigned artifacts and known-vulnerable versions of artifacts can’t be redeployed.
Endor Labs' software inventory capabilities help you get visibility into the software that you consume and produce so that you can respond quickly to future issues. We provide you a searchable inventory of the third-party and first-party software components used in your environment so you can quickly respond when you discover an issue in an open source software library; and we can even ingest 3rd-party SBOMs so that your inventory can include vendor application components. We also offer a simple-to-deploy artifact signing solution to help you defend against shadow deployments and avoid redeploying unremediated artifacts.
Actionable Approaches to Defense in Depth Can Help Reduce the Probability of Exploitation
Defense in depth or independent lines of security defenses can help detect and prevent supply chain attacks from being successful. Some actionable approaches to defense in depth to mitigate supply chain attacks such as XZ include:
- Implement Least Privilege - In the case of XZ a least privilege approach to server and container configuration would help. To state the obvious for some of you reading this blog: Backdoor SSH keys allow you to access SSH when SSH services are accessible. Please don’t expose SSH ports to the internet. Container based services probably don’t need to expose SSH availability in most reasonable cases. You are asking for trouble by exposing SSH to a container or to the public internet. While this wouldn’t have stopped anyone from adopting the vulnerable component, least privilege is a powerful defense to exploitation of vulnerabilities and backdoors.
- Create governance policies for open source usage - Many organizations have governance policies for open source software usage. These policies range from “Pin all versions of open source software used so that you don’t download the latest version always” to “don’t use versions of open source software less than 60 days old.” Policies focused on staying away from bleeding edge software versions add value by preventing scenarios such as Fedora-Rawhide, the existing development branch of Fedora, being compromised. These policies allow the open source community to vet a software package more completely. Endor Labs can help teams to enforce and triage policy violations to help scale such governance programs
- Remove unused software to reduce supply chain risk - Unused software can create unnecessary bloat in your software and introduce supply chain risk over time. Regularly review your open source software usage for each application and ensure that you remove unused software dependencies as they are no longer needed over time. You can use Endor Labs to help identify unused software in your application packages to systematically reduce risk over time.
- Regularly review auxiliary dependencies for potential risk - Auxiliary dependencies, or “small” one-off-packages, often maintained by one or two individuals often have fewer eyes on them. These dependencies can often be replaced and using them is a “choice” that any software organization has. These software components can be reviewed to ask questions such as: “Does this project contain binary files?”, “Does this package make network calls or calls to the host operating system that could be indicative of an attack?” or “Does this project have no automated releases?”. Reviewing these software dependencies for leading indicators of risk can be a high value task for many organizations.
Conclusion
Supply chain attacks will continue to evolve and are here to stay. The fundamentals of security remain the same. The best thing we can do is stay prepared and use the opportunity to improve our security programs:
- Don’t over-react! Instead, educate yourself and your teams about the attack and the governance programs that could be implemented.
- Never waste a good incident. Incidents are opportunities to improve your security program and explain the importance of supply chain security.
- Ensure that you do the fundamentals of security, they might just save you next time.