Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pwning the Nix ecosystem

ptrpa.ws

Post reply on HN
Print view View original

51–60 of 69 posts

Re: Pwning the Nix ecosystem

#51

Post by Xylakant »

rendaw wrote: post #46
pyrox wrote: post #40

Earlier quoted context omitted.

Hey! First, a disclaimer: I do not speak for anyone officially, but I am a very regular contributor to nixpkgs and have been involved in trying to increase nixpkgs' security through adopting the Full-Source Bootstrap that Guix and Stagex use. I also assume that the RFC you're talking about is RFC 0100, "Sign Commits"(ref: https://github.com/NixOS/rfcs/pull/100 ) As mentioned in the RFC discussion, the major blocker w…

> the major blocker with this is the lack of an ability for contributors to sign from mobile devices Do you mean a significant number of nixpkgs contributors make nixpkgs PRs from their phones... via the github web editor? That seems weird to me at face value... editing code is hard enough on a phone, but this is also for a linux distro (definitely not a mobile os today), not a web app or something else you could eve…

Mobile devices are not restricted to phones, but include tablets, some of which are pretty powerful and usable for code editing.

Note that the signature for the web interface is made with a GitHub owned key on your behalf and not with your personal key.

Re: Pwning the Nix ecosystem

#52

Post by lrvick »

rendaw wrote: post #46
pyrox wrote: post #40

Earlier quoted context omitted.

Hey! First, a disclaimer: I do not speak for anyone officially, but I am a very regular contributor to nixpkgs and have been involved in trying to increase nixpkgs' security through adopting the Full-Source Bootstrap that Guix and Stagex use. I also assume that the RFC you're talking about is RFC 0100, "Sign Commits"(ref: https://github.com/NixOS/rfcs/pull/100 ) As mentioned in the RFC discussion, the major blocker w…

> the major blocker with this is the lack of an ability for contributors to sign from mobile devices Do you mean a significant number of nixpkgs contributors make nixpkgs PRs from their phones... via the github web editor? That seems weird to me at face value... editing code is hard enough on a phone, but this is also for a linux distro (definitely not a mobile os today), not a web app or something else you could eve…

Sorry to be that guy, but if someone cannot afford a $10 bit of hardware for the most basic attempt at protecting others from being harmed by someone impersonating them... then they have no business being a trusted maintainer in a Linux distribution relied on for billions of dollars in infrastructure.

That would be like someone saying they could not afford a mask in COVID or something. It is hard to believe these people really exist. I could go find $10 in change looking on the ground of a few nearby fast food pick-up windows, because I have done it. Many times. Free money!

Anyway, such people will be easy to bribe, easy to target, easy to steal from. Letting that sort of person have trust in a major OS is endangering them, and frankly irresponsible.

For anyone that makes excuses about being unable to produce a hardware signing device, of course let them contribute, but then let two confirmed real humans with hardware keys adopt, review, and sign that PR, and always have at least two real confirmed humans with hardware keys sign every change both as code, and as reproducible artifacts after.

We have taken in tons of drive-by unsigned contributions in stagex. This is no problem. We just pretend an AI bot wrote it, and require one maintainer to "adopt" the commit to sign it (maintaining attribution), and then a second maintainer reviews, and does a signed merge as usual.

Re: Pwning the Nix ecosystem

#53

Post by tennysont »

vlovich123 wrote: post #44
pyrox wrote: post #40

Earlier quoted context omitted.

Hey! First, a disclaimer: I do not speak for anyone officially, but I am a very regular contributor to nixpkgs and have been involved in trying to increase nixpkgs' security through adopting the Full-Source Bootstrap that Guix and Stagex use. I also assume that the RFC you're talking about is RFC 0100, "Sign Commits"(ref: https://github.com/NixOS/rfcs/pull/100 ) As mentioned in the RFC discussion, the major blocker w…

The reason I dislike this is this is the first thing in the article: > in nixpkgs that would have allowed us to pwn pretty much the entire nix ecosystem and inject malicious code into nixpkg OP provided a mechanism to stymie the attack. The counter from your position needs to be how the nix project otherwise solves this problem, not “this isn’t the right approach for hand wavy reasons”. Given the reasonings stated, O…

> in nixpkgs that would have allowed us to pwn pretty much the entire nix ecosystem and inject malicious code into nixpkg

Isn't that what happens when a build server or source code is compromised? I'm not sure if the existence of this exploit was egregious, but the blast radius seems normal for a build server exploit.

> how the nix project otherwise solves this problem

You can go into `/etc/nix/nix.conf` and remove `trusted-public-keys` so that you don't trust the output of the build servers. Then you just need to audit a particular commit from nixpkgs (and the source code of the packages that you build) and pin your config to that specific commit.

Otherwise, it seems like the solution is to harden the build system and source code control so that you can freely trust the source code without auditing it yourself. I'm not sure what else can be done.

If your threat model is that the 10+ nixpkg contributors are trustworthy but the github repo is untrustworthy, then git signing would make you safe.

Personally, I worry that a carelessly approved merge in nixpkg or an upstream supply attack is a bigger threat then a github repo exploit (as described here), but I imagine that reasonable minds could disagree.

Regardless, I'm very excited to see that nix builds are almost fully reproducible. That seems great! It seems like this could potentially be the foundation on which a very secure distro is built.

Re: Pwning the Nix ecosystem

#54

Post by tennysont »

lrvick wrote: post #50
pyrox wrote: post #40

Earlier quoted context omitted.

Hey! First, a disclaimer: I do not speak for anyone officially, but I am a very regular contributor to nixpkgs and have been involved in trying to increase nixpkgs' security through adopting the Full-Source Bootstrap that Guix and Stagex use. I also assume that the RFC you're talking about is RFC 0100, "Sign Commits"(ref: https://github.com/NixOS/rfcs/pull/100 ) As mentioned in the RFC discussion, the major blocker w…

My RFC was much earlier in 2018 https://github.com/NixOS/rfcs/pull/34 Lack of supply chain integrity controls as a means to reduce contribution friction to maximize the number of packages contributed is a perfectly valid strategy for a workstation distribution targeted at hobby developers. Volunteers can do what they want, so that RFC convinced me stagex needed to exist for high security use cases, as Nix was explici…

Out of curiosity, why don't/didn't you start a new version of nixpkgs with hardened source? You could forgo the build server, forcing users to build from scratch (at least to start). You could leverage the plentiful, albeit, less secure, packaging code in the nixpkgs to quickly build out your hardened versions.

Effectively, you're building out an audited copy of nixpkgs on the same build engine, but with hardened configs. Write wrappers to validate git signatures when users update, and you got yourself a chain of trust on the source code distribution for your hardened nixpkg.

I'm sure you had reasons, I'm just interested to know your thought process.

Re: Pwning the Nix ecosystem

#55

Post by otabdeveloper4 »

lrvick wrote: post #50
pyrox wrote: post #40

Earlier quoted context omitted.

Hey! First, a disclaimer: I do not speak for anyone officially, but I am a very regular contributor to nixpkgs and have been involved in trying to increase nixpkgs' security through adopting the Full-Source Bootstrap that Guix and Stagex use. I also assume that the RFC you're talking about is RFC 0100, "Sign Commits"(ref: https://github.com/NixOS/rfcs/pull/100 ) As mentioned in the RFC discussion, the major blocker w…

My RFC was much earlier in 2018 https://github.com/NixOS/rfcs/pull/34 Lack of supply chain integrity controls as a means to reduce contribution friction to maximize the number of packages contributed is a perfectly valid strategy for a workstation distribution targeted at hobby developers. Volunteers can do what they want, so that RFC convinced me stagex needed to exist for high security use cases, as Nix was explici…

NixOS is miles better from a security standpoint than any Debian or Red Hat already, so take what you can.

Re: Pwning the Nix ecosystem

#56

Post by otabdeveloper4 »

ants_everywhere wrote: post #37
otabdeveloper4 wrote: post #29

Earlier quoted context omitted.

The SPIFFE standard does something like this. It's not used by anyone because nobody actually gives a shit about security, the entire industry is basically a grift.

Lots of projects use SPIFFE, but lots of people don't like the new tech because they think the old ways are simpler

After trying to get SPIFFE mTLS to work with Python asyncio and giving up, I'm sure "lots of projects" is an overstatement.

Even basic parts of the tech stack aren't there yet.

Re: Pwning the Nix ecosystem

#57

Post by otabdeveloper4 »

YouAreWRONGtoo wrote: post #33
aftergibson wrote: post #30

As time goes on, I find myself increasingly worried about supply chain attacks—not from a “this could cost me my job” or “NixOS, CI/CD, Node, etc. are introducing new attack vectors” perspective, but from a more philosophical one. The more I rely on, the more problems I’ll inevitably have to deal with. I’m not thinking about anything particularly complex—just using things like VSCode, Emacs, Nix, Vim, Firefox, JavaSc…

Emacs itself is probably secure and you can easily audit every extension, but if you update every extension blindly via a nicely composable emacs Nix configuration, you would indeed have a problem. I guess one could automate finding obvious exploits via LLMs and if the LLM finds something abort the update. The right solution is to use Coq and just formally verify everything in your organization, which incidentally me…

Formal verification solves nothing. You can have a formally verified 100% secure backdoor exploit. (Ultimately it all depends on the semantics of "sysadmin" vs "hacker", who are really just two different roles of the same person.)

This is also why signing code commits isn't a solution, only a way to trace ends when something fucks up.

Re: Pwning the Nix ecosystem

#58

Post by friendly_wizard »

lrvick wrote: post #34

Had the Nix team rolled out signed commits/reviews and independent signed reproducible builds as my (rejected) RFC proposed, then it would not be possible to do any last mile supply chain attacks like this. In the end NixPkgs wants to be wikipedia easy for any rando to modify, and fear any attempt at security will make volunteers run screaming, because they are primarily focused on being a hobby distro. That's just f…

Stagex is honestly the most interesting discovery in this thread

Re: Pwning the Nix ecosystem

#59

Post by lrvick »

tennysont wrote: post #53
vlovich123 wrote: post #44

Earlier quoted context omitted.

The reason I dislike this is this is the first thing in the article: > in nixpkgs that would have allowed us to pwn pretty much the entire nix ecosystem and inject malicious code into nixpkg OP provided a mechanism to stymie the attack. The counter from your position needs to be how the nix project otherwise solves this problem, not “this isn’t the right approach for hand wavy reasons”. Given the reasonings stated, O…

> in nixpkgs that would have allowed us to pwn pretty much the entire nix ecosystem and inject malicious code into nixpkg Isn't that what happens when a build server or source code is compromised? I'm not sure if the existence of this exploit was egregious, but the blast radius seems normal for a build server exploit. > how the nix project otherwise solves this problem You can go into `/etc/nix/nix.conf` and remove `…

You absolutely should never trust a centralized build server. Any security critical software distribution process should have all packages independently built, verified to have identical hashes, and signed by systems controlled by as many different trusted maintainers or third parties as possible.

Then any user can prove the binary they got was built faithfully from source due to those redundant build system signatures. We designed ReprOS for this purpose.

stagex has also been 100% deterministic, full source bootstrapped, and independently reproduced/signed by multiple maintainers since our first release with a small team of 10ish regular contributors, so it can be done.

Re: Pwning the Nix ecosystem

#60

Post by lrvick »

tennysont wrote: post #54
lrvick wrote: post #50

Earlier quoted context omitted.

My RFC was much earlier in 2018 https://github.com/NixOS/rfcs/pull/34 Lack of supply chain integrity controls as a means to reduce contribution friction to maximize the number of packages contributed is a perfectly valid strategy for a workstation distribution targeted at hobby developers. Volunteers can do what they want, so that RFC convinced me stagex needed to exist for high security use cases, as Nix was explici…

Out of curiosity, why don't/didn't you start a new version of nixpkgs with hardened source? You could forgo the build server, forcing users to build from scratch (at least to start). You could leverage the plentiful, albeit, less secure, packaging code in the nixpkgs to quickly build out your hardened versions. Effectively, you're building out an audited copy of nixpkgs on the same build engine, but with hardened con…

I ultimately thought out what would be easier, a decade political fight to make massive changes to nix, or a fork of it written solo to improve auditability and security, or starting over from the top with a design that checks every dream box I wanted from a linux distro.

I had many RFCs that would have followed this rejected one if there was any change tolerance... so my fastest path to prove out my ideas for a distro with decentralized trust was to start one with that explicit goal.

If I wanted to make things maximally auditable and portable to different build engines, a published dead simple spec with multiple competing implementations that most software engineers already know how to write would be ideal. People could review an engine they use, or ensure all existing implementations on any operating system get identical results and are thus trusted that way. If it natively supports a ton of features to make deterministic builds wildly simpler, major bonus.

OCI/Containerfile was a check on all fronts, and some early maintainers and I riffed on design patterns and realized the OCI ecosystem already had specified multi party signing and verification, artifact integrity, smart layer by layer caching etc etc. This fit our dev experience and threat model perfectly and we could just skip implementing the package build and distribution layer and just start writing packages, like that day. None of us needed to learn or invent a new language or ask auditors to do so or fork nix ecosystem to have proper signing support and write a sane spec... that could be years of wheel spinning.

The time saved by choosing an existing widely used and implemented spec meant we were able to put all energy into full source bootstrapping, universal multi party hardware signing on every build, change, review, and reproduction. Just full source bootstrapped linux from scratch in containerfiles with OCI native multi party signing if all parties independently get the same oci hashes from local builds. Oh and we are going LLVM native like Chimera next week. Big sweeping changes like that are easy with our ultralight setup.

I would note that the features we need for deterministic builds in docker, the most popular OCI implementation, only landed a couple of months before we started stagex, and the full source bootstrapping work by the bootstrappable builds team only got a complete bootstrap for the first time a few months before that and Guix shortly after. Tons of reference material.

If stagex had started before 2022 I imagine we might have used a heavily trimmed down nix clone or tried to convince guix to adopt our threat model, which is much further along in supply chain security than nix but scheme would have been a very isolating choice. I think stagex got lucky starting at exactly the right time when two huge pieces of the puzzle were done for us.

Post reply on HN