What Breaking Changes Teach Us about Security
GitHub rolled out a release that had some breaking changes to Git. Here's what we learned from it.
GitHub rolled out a release that had some breaking changes to Git. Here's what we learned from it.
GitHub rolled out a release that had some breaking changes to Git. Here's what we learned from it.
GitHub rolled out a release that had some breaking changes to Git. Here's what we learned from it.
GitHub rolled out a release that had some breaking changes to Git. Here's what we learned from it.
By Jamie Scott, Camilla Odlund, and Alexandre Wilhelm.
The industry has always maintained a delicate balance between security and productivity. To ensure we can trust the code we download we must know that it's reproducible and be able to verify its integrity. Modern software relies on integrity checking of software checksums to verify that downloaded software is trustworthy and as expected. Today, GitHub updated to a version of Git that changed the compression algorithm for GitHub releases, which are a common way that code is distributed and consumed. GitHub’s updates to the compression algorithm changed the software checksums and as a result software supply chains around the world began to crumble.
Popular developer ecosystems such as homebrew, the most prominent package manager for MacOS, and Bazel, a popular build system, rely heavily on GitHub's release files. They also verify those files' trustworthiness with checksum integrity checks. When those checks failed today, developers using these systems experienced an outage and software supply chains around the world lost a little trust.
GitHub’s process for release distribution is known to cause occasional problems with integrity checks. GitHub release files are either regenerated each time you use the link, or periodically cached and regenerated. As a result, even if the source code remains unchanged, the contents of the tarball file can be different each time it is generated.
At Endor Labs, we rely on Bazel as our primary build system. We run tests and rebuild our product to check every pull request and every commit that goes into our software. Our Bazel builds verify the integrity of each version of a package we depend on, ensuring that the package we download and put in our software is the package we’ve chosen to trust. Below is an example configuration of our own internal implementation of the software package protobuf:
When GitHub’s change went into effect today every single change to our code base was impacted and our software development efforts came to a halt as we were forced to troubleshoot the below error.
What did we end up doing? We ended up updating the checksums for each of our packages to fix this and then GitHub reverted their change and our build systems broke, again. We then reverted that change and our team was able to get back to building software. As a security conscious company, this was the approach we took but it is just one example of the tradeoffs between security and productivity.
Many development teams removed integrity checks to address this outage. Some of these won’t be put back because of a lack of trust and some will be put back over time. Outages create distrust of security controls, which disincentivize their use and create less secure supply chains. Disabled security controls, after all, can no longer provide their intended protections. GitHub provides a way for software package maintainers to allow software consumers to verify their packages and ensure software builds are reproducible, but this requires additional work from developers and with such a widespread ecosystem, it's unreasonable to assume this will work everywhere. This is the impact of security taxes on development teams: We lost tens to hundreds of hours of productivity.
Why do people use GitHub releases? Well, they are incredibly easy to use and reduce the friction of release management. The world has shifted towards an age of software assembly from many disparate open source communities. We can’t expect every open source community to provide a release outside of the frictionless GitHub release workflow. In such an interconnected ecosystem, we can't afford for simple to not also mean secure. It is the only way we can establish trust across our supply chains.
We can create awareness around GitHub releases and encourage users to rely on processes and systems that provide guarantees for reproducible builds and therefore can be verified as trusted. After all, GitHub provides a path to ensure that releases can be consistently verified. All you need to do is upload the release files yourself to achieve this and not strictly rely on GitHub to publish them for you. This approach can solve the problem limitedly, but, as we have all seen through years of corporate phishing and awareness campaigns, awareness efforts can only produce limited impact on the outcomes we want.
To create real outcomes in software supply chain security, we need to make supply chain security simple and transparent to users. The more disconnected the easy path and the secure path are, the more likely the less secure path will be taken. This is simply a matter of human nature. Let’s learn from this incident to encourage not only stronger security awareness but easier paths for developers to make the right choices. We don’t need to just make better security. We need to make security easier.