Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NixOS and reproducible builds could have detected the xz backdoor

luj.fr

Post reply on HN
Print view View original

11–20 of 192 posts

Re: NixOS and reproducible builds could have detected the xz backdoor

#11

Post by crote »

jjmarr wrote: post #7
lolinder wrote: post #5

Note that NixOS and reproducible builds did not detect the xz backdoor, and in fact NixOS shipped the malicious builds of xz (though they didn't do anything because the malware didn't target NixOS): > I am a NixOS developer and I was surprised when the backdoor was revealed to see that the malicious version of xz had ended up being distributed to our users. As always theory and reality are different, and the thing th…

I still like the blogpost, because NixOS bills itself as a technical solution to prevent build artifacts that are decoupled from the source code (i.e. not reproducible), and the xz backdoor was hidden in build artifacts.

Sure, but you could achieve the same thing by requiring that the build artifacts are generated by a Github Actions runner.

Re: NixOS and reproducible builds could have detected the xz backdoor

#13

Post by nextos »

lolinder wrote: post #5

Note that NixOS and reproducible builds did not detect the xz backdoor, and in fact NixOS shipped the malicious builds of xz (though they didn't do anything because the malware didn't target NixOS): > I am a NixOS developer and I was surprised when the backdoor was revealed to see that the malicious version of xz had ended up being distributed to our users. As always theory and reality are different, and the thing th…

> NixOS and reproducible builds did not detect the xz backdoor

Nix declarativeness is quite useful to increase protection against exploits in a number of ways. Unfortunately, there is still a lot of untapped potential. My number one priority would be to implement fine-grained ephemeral containers. Guix has these already.

This would make it convenient to run every single process with restricted privileges, including no access to ~/, except those directories that are needed by the task. That would prevent e.g. a rogue pip package from stealing SSH keys.

Still, I think the xz backdoor did not work on NixOS because its unusual non FHS-compliant filesystem structure.

Re: NixOS and reproducible builds could have detected the xz backdoor

#14

Post by microtonal »

rowanG077 wrote: post #12

it's somehow immensely funny to me that some state probably had an entire project to land this backdoor in xz, spend literal years to make it happen. And then it was immediately detected and all effort was for nothing.

Or they have N other such projects in flight. The xz backdoor wasn’t that much work, just playing the long game. The person doing the xz project could easily do several other projects at the same time.

A lot of issues do get undetected. E.g. the Debian OpenSSL security accident was only detected when a gazillion servers had predictable SSH keys.

Re: NixOS and reproducible builds could have detected the xz backdoor

#16

Post by mcint »

Excellent descriptive analysis. Wrong, misleading title, perhaps "technically correct," but at best with a "backdoored" meaning.

It points out the need and use for build-manager tools that go a step beyond union file system layers, but track then enforce that e.g. tests cannot pollute build artifacts. Take a causal trace graph of files affecting files, in the build process, make that trace graph explicit, and then build a way to enforce that graph, or report on deviations from previous trace graphs.

Re: NixOS and reproducible builds could have detected the xz backdoor

#17

Post by rowanG077 »

microtonal wrote: post #14
rowanG077 wrote: post #12

it's somehow immensely funny to me that some state probably had an entire project to land this backdoor in xz, spend literal years to make it happen. And then it was immediately detected and all effort was for nothing.

Or they have N other such projects in flight. The xz backdoor wasn’t that much work, just playing the long game. The person doing the xz project could easily do several other projects at the same time. A lot of issues do get undetected. E.g. the Debian OpenSSL security accident was only detected when a gazillion servers had predictable SSH keys.

Most likely yes. That doesn't really impact the funny factor for me.

Re: NixOS and reproducible builds could have detected the xz backdoor

#18

Post by nialv7 »

I feel the author is a bit tunnel visioned by what happens to happen this time. The Jiatan incident has a sample size of one, it'd be a bit short sighted to think that's the only way it could happen. You can imagine various scenarios where the defenses suggested here will not have worked.

Also I (as a nix user myself) think it's unlikely NixOS would have caught it. As evidenced by the fact that it didn't. (Yeah I realize I just said next time it might happen differently but it'd be foolish to put faith in nix without evidence).

Re: NixOS and reproducible builds could have detected the xz backdoor

#19

Post by Dwedit »

rowanG077 wrote: post #12

it's somehow immensely funny to me that some state probably had an entire project to land this backdoor in xz, spend literal years to make it happen. And then it was immediately detected and all effort was for nothing.

Yeah, without the latency regression, it probably would have gone undetected much longer. Using a secondary thread and spreading the CPU load over a few seconds would have made it not even register as a spike in CPU usage.

Re: NixOS and reproducible builds could have detected the xz backdoor

#20

Post by ratorx »

nextos wrote: post #13
lolinder wrote: post #5

Note that NixOS and reproducible builds did not detect the xz backdoor, and in fact NixOS shipped the malicious builds of xz (though they didn't do anything because the malware didn't target NixOS): > I am a NixOS developer and I was surprised when the backdoor was revealed to see that the malicious version of xz had ended up being distributed to our users. As always theory and reality are different, and the thing th…

> NixOS and reproducible builds did not detect the xz backdoor Nix declarativeness is quite useful to increase protection against exploits in a number of ways. Unfortunately, there is still a lot of untapped potential. My number one priority would be to implement fine-grained ephemeral containers. Guix has these already. This would make it convenient to run every single process with restricted privileges, including n…

You don’t even need to run in a container for this. It’s possible to do this entirely in systemd service configuration. The easiest way is just to have separate user for every service and reduce stuff running as root. You can also restrict filesystem access, network access and even syscall access (although some of this may be implemented as a container under the hood).

Unfortunately, this wouldn’t help with the xz vulnerability because the SSH server is the one loading the compromised library in that case (indirectly). Since SSH itself needs to have access to the private keys, it’s not really easy to secure it against vulnerabilities in the library it loads itself.

On the flip side, unless the vulnerability is in one of the important binaries/shared libraries, the amount of damage it can cause it probably quite contained with simply having good user isolation. Nix can make this analysis really simple (because of explicitly specified dependencies), so you can crack down on critical dependencies a lot more easily.

Post reply on HN