Compromising OpenWrt Supply Chain

It's easy to defeat right now because very few are currently thinking about secure build systems.

As an example, systemd's latest source tarball has two Web Open Font Format (WOFF) files in a documentation folder, a favicon.png, two few small screenshots and error messages that have unexplained 128bit identifiers. There are also translated strings (PO files) which could include obscure writing systems few can quickly audit, and thus could be advantageous to an attacker wanting to hide malicious code.

The problem with most build systems is the entire source tarball is extracted into a single location and any build scripts executed during build have access to everything in the source tarball. Gentoo's portage does apply sandboxing around an entire build process, just not internally between the different build stages.

Continuing the Gentoo example (being one of the better examples of sandboxed builds), ideally src_unpack could take a compressed tarball distributed from an upstream project and split files into multiple separate paths such as:

- source code files and build scripts needed to build binaries

- supporting test files needed to test the the built binaries

- supporting documentation/binary/high entropy data such as application icons, game data, documentation, etc that should be copied upon installation but aren't required to build binaries

Then src_prepare, src_configure, src_compile, src_test and src_install all have different sandbox configurations restricting which types of files separated during src_unpack each build phase can interact with.

For the systemd example above, some possible improvements could be:

1. Remove WOFF font files and use system default fonts, omit favicon.png and omit screenshots. Or only make WOFF font files, favicon.png and screenshots available for copying during the src_install phase, and ensure they are not readable by build scripts during src_configure and src_compile.

2. Generate error message identifiers using an explained approach such as hash_algorithm("systemd_error_" + name_of_error_constant) to generate nothing-up-my-sleeve identifiers.

3. Only provide access to and include in the build process any translation files of languages the user cares about. Or only make translation files available for copying during the src_install phase and possibly src_test too, and ensure they are not readable by build scripts.

These build system security techniques are obviously more work, but are generally straightforward to understand and implement. These technqiues are in the realm of possibility for smaller embedded Linux systems that may just be kernel + BusyBox + a few small scripts + a bespoke application + a handful of dynamic libraries. And for more complex Linux systems, these techniques are within the realm of possibility when targeted towards high value software, such as software requiring or often executed with root permissions and software requiring simultaneous access to the Internet and access to user files.

Very cool article, security researchers are incredibly creative and scary smart.

That was an excellent report and a really decent technical explanation. Good to see how quickly OpenWrt (one of my favorite open-source projects) fixed and addressed this vulnerability!

This is a nice idea, and one I also advocate for, however it's important to keep in mind that the idea of reproducibility relies on determinism. So much of what goes into a build pipeline is inherently nondeterministic, because we're making decisions at compile time which can differ from compilation run to compilation run, setting aside flags. In fact, that's the point of an optimizing compiler, as many reproducible build projects have discovered, turning on optimizations pretty much guarantees no reproducibility.

Counterpoint: Archlinux is 89% reproducible with optimizations enabled. The only thing I see which is difficult to make reproducible is optimizations with a timeout.