NOTE: The views I'm expressing here are solely my own and say nothing about my employers views.
I think that the tech industry has a severe code supply chain issue. Supply chains are a super hard problem with physical products (Raise a hand if you (a) have tried tracing the supply chain of cocoa (b) Can tell a midnight factory run of luxury clothes from a legitimate one (c) remember the supermicro controversy ) but with software we have the ability to do a much better job on solving it. I find it really disappointing that we have failed to do so. Reading through the comments here I've seen discussions on deterministic builds, code signing, and other practices. I think that they are parts of a unified whole, but all the pieces need to be there and need to be correctly done. Below I outline where I think the industry should be.
A complete, secure, code supply chain should do the following:
* Validate signatures on all 3rd party dependencies
* Ensure that all internally written code relies on signed commits
* Have builds be reproducible
* Sign the output of those builds
Taken together all of these form a complete supply chain that applies to both closed and open source software. There is nothing technically infeasible about implementing much of it as well - to me it feels like a culture issue.
The gap between where we seem to be and where we should be seems to be:
* Validate signatures on all 3rd party dependencies
** Present Day: Many vendors cannot be bothered to properly sign the outputs of their builds. Microsoft updates, openssh releases, and things like that remain the exception rather than the rule. This problem becomes even more egregious when looking at enterprise to enterprise products such as drivers which are either massive sets of source code or precompiled blobs, both of which run with lots of privileges in the context of the product they are integrated into. Even Fedora provides lots of packages from their Koji build system, the majority of which are not signed.
** Where we could be: Normalize signing these, and normalize validating the signatures prior to any use in a build environment. This is one of the easiest places in the supply chain to insert malware due to the lack of verifications.
* Ensure that all internally written code relies on signed commits
** Present Day: Outside of git, most VCS systems don't even support signed commits. Within git, signed commits are not popular. I personally blame the tooling. Signing is based on PGP keys which have all sorts of known issues with use, tooling, and a general disdain due to their initial use case for email being broken. Places like Github attempt features like mandatory signing, but that falls short. Keys are still sourced from unknown places, each developer is responsible for their own key, there is no support for validating prior commits once the signing key is rotated, and using the webui totally bypasses the signing requirements (https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/about-commit-signature-verification).
** Where we could be: Let's imagine a future where git is used as a VCS. Signing keys should be centrally controlled by an authority with developers issued code signing subkeys that are rotated and can be revoked by the central authority. By having a history of all code signing keys over time, the repository can also be audited at any point in time. Even if malicious insiders directly alter the VCS, it can be flagged! I lead a project to implement such a system at my work ( https://eos.arista.com/commit-signing-with-git-at-enterprise-scale/ ) which I am posting on every discussion here to try and normalize a discussion around how to do this at other companies.
* Have builds be reproducible
** Present Day: This is probably the biggest gap in having a secure supply chain. Builds today are not reproducible nor are they deterministic. The best which I know of is NixOS which is around 99% reproducible ( https://r13y.com/ ). Debian appears 95% on a specific target ( https://isdebianreproducibleyet.com/ ). Most other products are much lower than that.
** Where we could be: The first step would be deterministic builds, where building with the same inputs always results in the same outputs. Once you have a way to store what those inputs are, you can then reproduce builds later. Securing build environments becomes much easier at that point. You can build in multiple places, at multiple security levels, and check the same output comes out each time. You can even build at a much later point in time since you should have your whole set of dependencies clearly documented and saved. Validating outputs is super easy later on since you can recreate exactly what it should have been. This is also great for build systems in general since it makes dependency graphs more accurate and reduces problems with building in different environments. With the existence of VMs and containers, this is also a problem that should be super solvable. The devil is in the details here, but there should not be any reason it cannot be solved other than a lack of proper investment.
* Sign the output of those builds
** Where we are today: This is one of the items that is actually the most popular, since it is so easy to do. There are lots of methods to sign any sort of data and the tooling around them is pretty straightforward. By signing this data, it closes the loop on someone downstream validating that data as an input to their own system.
** Where we could be: Keeping up the good work and going further to normalize signing build system outputs!