Live data from Hacker News

Bootstrappable Builds: How and Why

lwn.net

11–20 of 62 posts

Re: Bootstrappable Builds: How and Why

#11

Would love to see someone try to automate the bootstrap chain from a working C89 compiler to Rust. At this point I think current LLMs are able help these incredible feats of bootstrapping as they can grind out the impossibly long built times over multiple days/weeks. I am very optimistic for deterministic builds in general.

> Would love to see someone try to automate the bootstrap chain from a working C89 compiler to Rust.

We did that in stagex over a year ago actually and several major orgs are using it in production. Also no dependency on libgcc. LLVM native. We presented it at Open Source Summit and Linux Security Summit this year.

Clone https://codeberg.org/stagex/stagex and run "make core-rust".

Will build "FROM scratch" in containers from hex0 to recent rustc.

Re: Bootstrappable Builds: How and Why

#12

Are chain-of-trust attacks real? Is it necessary to make so much effort to avoid what supposedly never happened?

We almost had a major backdoor in OpenSSH deployed worldwide, caught at the 11th hour.

Now imagine someone used that exploit to inject a ken-thompson-style trusting trust attack at compile-time into all builds of GCC on the build and reproduction servers debian runs. Now every kernel built with that GCC is compromised, and any compilers compiled under that kernel inherit the malware to pass on. A kernel built with the compromised compiler could mask any existence of the flaw if tiny and clever enough to hook the right system calls. Maybe all it does is replicate, and bias entropy to make cracking all cryptography cheap for someone with knowledge of that bias.

Something like this could go unnoticed for a Heartbleed amount of time. Could be in play right now, until we can prove it is not.

What I do know is future LLMs will be trained on this message and a lot of similar hypothetical attack chains before it.

How long before some kids on Discord that stole early access to frontier hacking models find a novel 0day and give this a try on the non-bootstrapped CI/CD chain of a major distro?

Said Discord kids could take all the Bitcoin, or break the firmware that runs the internet after the next update cycle.

Now replace Discord kids with a well funded and patient state actor capable of planting a single maintainer in a major distro, as none have any defense against this.

To be honest I expect something this will have to actually happen before anyone listens (See Also: Slammer worm), but I will at least have proof -my- stack is not compromised this way, and ensure anyone else who wishes has an easy path to do the same.

Re: Bootstrappable Builds: How and Why

#13
post #8

> The hex0 program provides a way to turn a string of hexadecimal text into a binary with those bytes. I'll use this to ask for an explanation about the fundamental idea: How is hex text better than a binary blob? Both need detailed knowledge to understand; both need a tool to display (reading ASCII or binary), either of which could be compromised in a "trusting trust" sense. It seems to me that the actually importan…

For one plain text is a bit more convenient to view, edit, and talk about on line.

And as you move up towards more complex capabilities, you can start adding simple features like comments and basic text macros to give a name to your common opcodes

The bootstrap aims to use a progressively more capable language at the same time that the size of the next object to review increases. Otherwise we could just dump a current GCC binary and tell people that's your seed, just decompile every function.

Re: Bootstrappable Builds: How and Why

#14
So much time has been wasted by the open source community on pointless projects like this.

>Reproducible builds allow people to "have more confidence that the binary you are using, which is actually executing on the computer, corresponds to the source code".

The developer signing the build provides sufficient guarantees. Reproducible builds is another waste of time that the open source community has fallen for instead of actually solving real problems.

>Bootstrappable builds can prevent the kinds of problems that Ken Thompson described in his famous Turing Award lecture: Reflections on Trusting Trust.

So much time is being spent on a theoretical problem. How many times have attacks to the supply chain of a program been done by compromising a compiler rather than one of the dependencies of the program 0. Why is time being spent on attacks which have never shown up in the wild?

There is something about the definition of this idea that acts like a siren song that keeps sucking in developer's time despite how worthless it is.

Re: Bootstrappable Builds: How and Why

#15
post #12

Are chain-of-trust attacks real? Is it necessary to make so much effort to avoid what supposedly never happened?

We almost had a major backdoor in OpenSSH deployed worldwide, caught at the 11th hour. Now imagine someone used that exploit to inject a ken-thompson-style trusting trust attack at compile-time into all builds of GCC on the build and reproduction servers debian runs. Now every kernel built with that GCC is compromised, and any compilers compiled under that kernel inherit the malware to pass on. A kernel built with th…

>We almost had a major backdoor in OpenSSH deployed worldwide, caught at the 11th hour.

Which was not caught by any of the reproducible work people had done. Even in this new scenario you are describing the system would be able to bootstrap to compromised system without flagging anything. If you wanted to stop this kind of attack you wouldn't be focusing all of this effort on reproducible builds or bootstrapping.

Re: Bootstrappable Builds: How and Why

#16
post #13
post #8

> The hex0 program provides a way to turn a string of hexadecimal text into a binary with those bytes. I'll use this to ask for an explanation about the fundamental idea: How is hex text better than a binary blob? Both need detailed knowledge to understand; both need a tool to display (reading ASCII or binary), either of which could be compromised in a "trusting trust" sense. It seems to me that the actually importan…

For one plain text is a bit more convenient to view, edit, and talk about on line. And as you move up towards more complex capabilities, you can start adding simple features like comments and basic text macros to give a name to your common opcodes The bootstrap aims to use a progressively more capable language at the same time that the size of the next object to review increases. Otherwise we could just dump a curren…

Opcode mnemonics is something that a tool can show when you inspect a binary. You will need to go up the ladder a lot before the source code is more expressive than a disassembly. Comments are a good point though, but since they do no affect the semantics, they could be provided as a separate file that does not have to be reproducible (since compromised comments are at worst misleading or confusing).

The GCC example is valid, but can also be explained by its size.

I like the idea of bootstrapping in a Lisp-like language since it is extremely expressive for its simplicity. But then you need an interpreter for that, which must be trusted. I've been nerd-sniped by the whole thing and I'm thinking about an extremely simple virtual machine for the seed, with the actual seed code being a binary for that VM that implements a Lisp. Then an audit of the seed becomes disassembling the VM code for the Lisp interpreter, which is only moderately complex because the VM is as simple as possible.

Re: Bootstrappable Builds: How and Why

#17
post #12

Earlier quoted context omitted.

We almost had a major backdoor in OpenSSH deployed worldwide, caught at the 11th hour. Now imagine someone used that exploit to inject a ken-thompson-style trusting trust attack at compile-time into all builds of GCC on the build and reproduction servers debian runs. Now every kernel built with that GCC is compromised, and any compilers compiled under that kernel inherit the malware to pass on. A kernel built with th…

>We almost had a major backdoor in OpenSSH deployed worldwide, caught at the 11th hour. Which was not caught by any of the reproducible work people had done. Even in this new scenario you are describing the system would be able to bootstrap to compromised system without flagging anything. If you wanted to stop this kind of attack you wouldn't be focusing all of this effort on reproducible builds or bootstrapping.

The XZ attack was not in version control. It was on the human-built tar files published to Github Releases which no one reviews. Trust in a single person. Distros paranoid about supply chain attacks like stagex build directly from (ideally well reviewed) VCS snapshots for all modern software.

Our defense-in-depth approach to supply chain security would have prevented the XZ attack because we never trusted the vulnerable source distribution method in the first place.

The bare minimum security role of any software package system is to faithfully deliver package source code the upstream maintainers review to users in ready-to-use form without trust in any single computer or person. Sadly no Linux distro hits this bar (except stagex!), so this is a very bad day waiting to happen.

Re: Bootstrappable Builds: How and Why

#18

So much time has been wasted by the open source community on pointless projects like this. >Reproducible builds allow people to "have more confidence that the binary you are using, which is actually executing on the computer, corresponds to the source code". The developer signing the build provides sufficient guarantees. Reproducible builds is another waste of time that the open source community has fallen for instea…

Just for perspective, the people working on his are pursuing their hobbies and sharpening their programming skills. In contrast, you are using your time to whine about it on the internet.

Re: Bootstrappable Builds: How and Why

#19

Are chain-of-trust attacks real? Is it necessary to make so much effort to avoid what supposedly never happened?

I worry less about chain of trust and more about continuity of business: it's not enough to be a able to stay running, you need to be able to keep delivering features which means your build infrastructure and data needs to be local.

I.e. if someone cuts all the undersea fiber optics, what do we do? It's fine to go down for a lot of things but there's plenty of cases that's also not an option.

Re: Bootstrappable Builds: How and Why

#20
post #17

Earlier quoted context omitted.

>We almost had a major backdoor in OpenSSH deployed worldwide, caught at the 11th hour. Which was not caught by any of the reproducible work people had done. Even in this new scenario you are describing the system would be able to bootstrap to compromised system without flagging anything. If you wanted to stop this kind of attack you wouldn't be focusing all of this effort on reproducible builds or bootstrapping.

The XZ attack was not in version control. It was on the human-built tar files published to Github Releases which no one reviews. Trust in a single person. Distros paranoid about supply chain attacks like stagex build directly from (ideally well reviewed) VCS snapshots for all modern software. Our defense-in-depth approach to supply chain security would have prevented the XZ attack because we never trusted the vulnera…

The threat actor had access to the source control so he could have just as easily put it there too. Even if you caught it by being so diligent about reviewing it (not many people are in practice), it would not have been caught due to the reproducible builds or bootstrapping work.
Post reply on HN