NixOS and reproducible builds could have detected the xz backdoor • Hacker News Board

Re: NixOS and reproducible builds could have detected the xz backdoor

#21

Post by xvector » Sat, Mar 22, 2025, 10:48 PM UTC

vlovich123 wrote: post #4

ltbarcly3 wrote: post #3

Yes, if you use a trusted framework then you are safe from things until that framework is attacked. The xz backdoor might have been detected, but the xz backdoor wasn't crafted with the goal of working against the Nix ecosystem. When a nix core developer ends up being a spy or whatever then there will end up being an attack against the nix ecosystem. Don't reply to this with some claim that Nix is inherently secure u…

The standard never has been and never will be absolute security. That’s an impossible threshold nothing would ever meet even though it’s objectively true that software today is generally more secure than software 30 years ago. The strongman claim being made is “Nix is harder and more expensive to exploit than traditional build systems”. So sure, if you find a cheap way to exploit Nix, track me down. But until then, i…

> But until then, it remains at least plausible & in practice very likely that Nix is harder to exploit than alternate systems on a technical level.

Do people even read package derivations? Feels like it'd be easy to check in a derivation with an exploit.

Re: NixOS and reproducible builds could have detected the xz backdoor

#22

Post by tomjakubowski » Sat, Mar 22, 2025, 10:48 PM UTC

crote wrote: post #11

jjmarr wrote: post #7

Earlier quoted context omitted.

I still like the blogpost, because NixOS bills itself as a technical solution to prevent build artifacts that are decoupled from the source code (i.e. not reproducible), and the xz backdoor was hidden in build artifacts.

Sure, but you could achieve the same thing by requiring that the build artifacts are generated by a Github Actions runner.

That would only make the xz attack harder, not make it impossible. Just add some step to the action run that fetches blobs from an internet resource you control, then swap out the blobs for malicious ones.

Re: NixOS and reproducible builds could have detected the xz backdoor

#23

Post by lolinder » Sat, Mar 22, 2025, 10:50 PM UTC

nextos wrote: post #13

lolinder wrote: post #5

Note that NixOS and reproducible builds did not detect the xz backdoor, and in fact NixOS shipped the malicious builds of xz (though they didn't do anything because the malware didn't target NixOS): > I am a NixOS developer and I was surprised when the backdoor was revealed to see that the malicious version of xz had ended up being distributed to our users. As always theory and reality are different, and the thing th…

> NixOS and reproducible builds did not detect the xz backdoor Nix declarativeness is quite useful to increase protection against exploits in a number of ways. Unfortunately, there is still a lot of untapped potential. My number one priority would be to implement fine-grained ephemeral containers. Guix has these already. This would make it convenient to run every single process with restricted privileges, including n…

> I think the xz backdoor did not work on NixOS because its unusual non FHS-compliant filesystem structure.

Right, but this is not part of the security model, it's an incidental attribute of the OS that's there for other reasons and easily solved for if the attacker had prioritized it. The only reason why it didn't work is because the attacker didn't bother making it work on NixOS, not because he couldn't have if he'd wanted to.

Re: NixOS and reproducible builds could have detected the xz backdoor

#24

Post by Smaug123 » Sat, Mar 22, 2025, 11:05 PM UTC

lotharcable wrote: post #6

So the argument hinges on the fact that the XZ maintainer hid malicious code in the tarballs that were not checked into Git. The author demonstrates that Nix can be configured to generate the tarballs from git that go into building the binaries. What I don't see, however, is how is this a feature that requires Nix or NixOS? Any build system out there (including the stuff that goes into RPMs and Debs) can be configure…

The article does in fact cite the reproducible-builds project, in the section on "Leveraging bitwise reproducibility". From your comment I am not convinced you understood the point of the article, which is:

* the NixOS build process was unable to perform a full-source build of xz because xz is required too early in the bootstrap;

* a proposed adjustment to nixpkgs to automatically detect compromises of nixpkgs dependencies which are required early in the bootstrap.

Other ecosystems can of course also attempt full-source builds and discover the discrepancy; the entire point of the article is that nixpkgs currently cannot.

Re: NixOS and reproducible builds could have detected the xz backdoor

#25

Post by nextos » Sat, Mar 22, 2025, 11:05 PM UTC

ratorx wrote: post #20

nextos wrote: post #13

Earlier quoted context omitted.

> NixOS and reproducible builds did not detect the xz backdoor Nix declarativeness is quite useful to increase protection against exploits in a number of ways. Unfortunately, there is still a lot of untapped potential. My number one priority would be to implement fine-grained ephemeral containers. Guix has these already. This would make it convenient to run every single process with restricted privileges, including n…

You don’t even need to run in a container for this. It’s possible to do this entirely in systemd service configuration. The easiest way is just to have separate user for every service and reduce stuff running as root. You can also restrict filesystem access, network access and even syscall access (although some of this may be implemented as a container under the hood). Unfortunately, this wouldn’t help with the xz vu…

> It’s possible to do this entirely in systemd service configuration

Sure, but I think that leaves out many use cases. What if I want to e.g. start a Python shell that has access to certain directories, and nothing else, including no network access?

Nix provides a good way of doing that for common use cases, as it has decent support for Firejail. But I would like something like Guix containers, which is convenient for any ad hoc use case. This greatly reduces any security threat. It's a poor-man's QubesOS.

Re: NixOS and reproducible builds could have detected the xz backdoor

#26

Post by immibis » Sat, Mar 22, 2025, 11:06 PM UTC

nextos wrote: post #13

lolinder wrote: post #5

Note that NixOS and reproducible builds did not detect the xz backdoor, and in fact NixOS shipped the malicious builds of xz (though they didn't do anything because the malware didn't target NixOS): > I am a NixOS developer and I was surprised when the backdoor was revealed to see that the malicious version of xz had ended up being distributed to our users. As always theory and reality are different, and the thing th…

> NixOS and reproducible builds did not detect the xz backdoor Nix declarativeness is quite useful to increase protection against exploits in a number of ways. Unfortunately, there is still a lot of untapped potential. My number one priority would be to implement fine-grained ephemeral containers. Guix has these already. This would make it convenient to run every single process with restricted privileges, including n…

It's interesting to observe that every process is already restricted to only be able to do computation by default. Then along comes the OS with a plethora of holes in the sandbox to do various things. And then it's strange that we take those holes for granted and apply bandaid patches over them instead of not creating the holes to begin with. Why can't we ask the kernel to create a new extremely limited memory map and run some code until a certain software interrupt fires, then restore the previous context? Why should we have to start with a fully powered-up process and then close off its abilities, instead of starting with no capabilities except the in/out buffers and computation? In this model, there could be a deliberate backdoor in the computation and it still couldn't do anything besides DoS.

Re: NixOS and reproducible builds could have detected the xz backdoor

#27

Post by immibis » Sat, Mar 22, 2025, 11:09 PM UTC

dzaima wrote: post #10

cocoto wrote: post #8

Earlier quoted context omitted.

The backdoor was not targeting Nix but it had to not raise any suspicion during a build in Nix to not be exposed.

The backdoor build script specifically checked for things indicating that it's being built for debian, and if not, not inserting the backdoor; so it only ever was non-reproducible in situations where reproducibility wasn't expected. Not hard to make sure a backdoor with control over the build environment doesn't raise suspicions in non-targeted places.

also wasn't the backdoor reproducible to begin with? If it had targeted every system, you'd just get reproducible backdoored binaries.

Re: NixOS and reproducible builds could have detected the xz backdoor

#28

Post by immibis » Sat, Mar 22, 2025, 11:09 PM UTC

microtonal wrote: post #14

rowanG077 wrote: post #12

it's somehow immensely funny to me that some state probably had an entire project to land this backdoor in xz, spend literal years to make it happen. And then it was immediately detected and all effort was for nothing.

Or they have N other such projects in flight. The xz backdoor wasn’t that much work, just playing the long game. The person doing the xz project could easily do several other projects at the same time. A lot of issues do get undetected. E.g. the Debian OpenSSL security accident was only detected when a gazillion servers had predictable SSH keys.

AFAIK Debian OpenSSL was detected when two people had the same GitHub key and since GitHub identifies people by their key it conflicted.

Re: NixOS and reproducible builds could have detected the xz backdoor

#29

Post by immibis » Sat, Mar 22, 2025, 11:11 PM UTC

Dwedit wrote: post #19

rowanG077 wrote: post #12

it's somehow immensely funny to me that some state probably had an entire project to land this backdoor in xz, spend literal years to make it happen. And then it was immediately detected and all effort was for nothing.

Yeah, without the latency regression, it probably would have gone undetected much longer. Using a secondary thread and spreading the CPU load over a few seconds would have made it not even register as a spike in CPU usage.

Or do cheap ECDSA instead of expensive RSA. Even if the backdoor is hidden inside RSA decryption and the rest of the system thinks the thing being decrypted should be encrypted with RSA, you don't have to use it for the back door.

Re: NixOS and reproducible builds could have detected the xz backdoor

#30

Post by ratorx » Sat, Mar 22, 2025, 11:17 PM UTC

nextos wrote: post #25

ratorx wrote: post #20

Earlier quoted context omitted.

You don’t even need to run in a container for this. It’s possible to do this entirely in systemd service configuration. The easiest way is just to have separate user for every service and reduce stuff running as root. You can also restrict filesystem access, network access and even syscall access (although some of this may be implemented as a container under the hood). Unfortunately, this wouldn’t help with the xz vu…

> It’s possible to do this entirely in systemd service configuration Sure, but I think that leaves out many use cases. What if I want to e.g. start a Python shell that has access to certain directories, and nothing else, including no network access? Nix provides a good way of doing that for common use cases, as it has decent support for Firejail. But I would like something like Guix containers, which is convenient fo…

Yeah, I guess most existing Linux stuff that is actually configured (so not SELinux etc) is geared at system processes not user ones. Transparently running all user applications in properly isolated containers would be quite neat.

Does Firejail handle dynamic access (eg. I may want an xz invocation to work on my private keys, but not THIS specific one where I’ve given it a completely different file?).

I quite like pledge/unveil for this kind of thing on OpenBSD, although that’s for a different threat model.