NixOS Reproducible Builds: minimal ISO successfully independently rebuilt • Hacker News Board

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#171

Post by amarshall » Tue, Oct 31, 2023, 1:03 AM UTC

lrvick wrote: post #169

amarshall wrote: post #145

Earlier quoted context omitted.

What distros do this? As far as I can tell Debian doesn’t anymore (the build system signs builds), Fedora doesn’t, and Arch…I can’t tell but I don’t think it does. But maybe I’m wrong. The NixOS build system signs all build outputs with its key and the signature is verified upon download. If you’re paranoid, Nix at least makes it easy to just build everything from source.

Yes packages are reproduced and signed. I get that and this is fantastic. That is not the big gap I am talking about which is making sure the expressions themselves even use the correct sources and are not tampered with. Making sure the central signing key is not compromised is pretty critical too, but that can be mitigated with independent reproduction servers that also sign, requiring a minimum of 3 before a new pa…

Ah so you’re talking about the package sources (e.g. Nix files, PKGBUILD, etc.). I guess that’s closer to commit signing, I think? Well Nixpkgs maintainership is much more bazaar than cathedral, for better or worse, and that stymies some of that. However most Nixpkgs maintainers do not have commit access at all. Nixpkgs is at least moving the right direction (I think) by limiting the power of committers to submit changes without review. Should Nixpkgs committers sign their commits? Probably, yea.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#172

Post by __MatrixMan__ » Tue, Oct 31, 2023, 6:12 AM UTC

lrvick wrote: post #167

__MatrixMan__ wrote: post #166

Earlier quoted context omitted.

Nix is two steps forward in these ways because they chose to focus on making things composable and repeatable instead of being curators of quality and trustworthiness. One magical thing about Nix is that there's a very small divide between managing to install software in the first place and creating an artifact that others can use for the same purpose. Because of this, practically every NixOS user has their configura…

My criticism is not so much on NixOS itself. If it wants to be a fun, composable, and easy to work on with minimal participation friction, fair enough. The yolo approach to contribution integrity has clearly resulted in fast development of ideas that might have been unable to grow in other distros. I suppose where my, perhaps misdirected, anxiety comes from is that I run a security consulting company and see NixOS be…

"A fork of nix" sounds awful drastic. Seems like you just need a curated list of packages which conform to some standard re:

- getting a dev sig when they get the code

- checking that sig

- adding a packager sig

And then you need to somehow inject a check before the user comes in contact with the outputs:

> are these signers in my trusted list?

Your users can then enable dev-sig-mode and point their sig checker at the list of keys. Hopefully that's less than whatever a fork of the whole OS entails.

Rather than framing it as a move towards signatures for all of NixOS, I'd frame it as you have a community of users who are willing to maintain a list of trusted keys and work with developers to standardize signature hand-off, and you want to add experimeny features to serve the little pocket of Nixdom that you're carving out for those users.

Avoid anything that smells like you're expecting NixOS act as an authority over which packages are trustworthy and for its devs do the political work of maintaining that list on behalf of the users. Many of us have landed at NixOS because we want less of that top-down nanny business, but we probably like the idea that users would themselves configure such a list.

I can maybe help come up with a standard flow for the signature hand-off, but the harder thing will be intercepting the myriad ways that a user might come in contact with derivation outputs and putting a sig check in their path. You may need another ally that knows the guts better than I do, not just the packaging side of things.

Hopefully we don't have to tamper with nix-env and nix-shell and `nix flake build` and `nix flake run` and `nix flake develop` all separately. Hopefully there's some common place where inbound bits can be checked regardless of how the user has asked for them.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#173

Post by lrvick » Tue, Oct 31, 2023, 7:08 AM UTC

__MatrixMan__ wrote: post #172

lrvick wrote: post #167

Earlier quoted context omitted.

My criticism is not so much on NixOS itself. If it wants to be a fun, composable, and easy to work on with minimal participation friction, fair enough. The yolo approach to contribution integrity has clearly resulted in fast development of ideas that might have been unable to grow in other distros. I suppose where my, perhaps misdirected, anxiety comes from is that I run a security consulting company and see NixOS be…

"A fork of nix" sounds awful drastic. Seems like you just need a curated list of packages which conform to some standard re: - getting a dev sig when they get the code - checking that sig - adding a packager sig And then you need to somehow inject a check before the user comes in contact with the outputs: > are these signers in my trusted list? Your users can then enable dev-sig-mode and point their sig checker at th…

I went to Nix liking everything about it except the fact there was no way to even -optionally- add signatures to NixOS core component contributions and package expressions, let alone verifying either of those as an end user.

I was informed the way to get changes into Nix was go ask the Nix Gods in the form of an RFC. I did that, and there was some interest, but ultimately it descended into bike shedding about optimal signing schemes and the idea was ultimately rejected by said Nix Gods. I felt very nannied to be honest, and felt I had to give up on nix.

I have had no viable alternatives but to use Arch and Debian for high security build systems across the industries I service and write all sorts of kludgy tools to hash pin packages to get deterministic results since that is what I can prove the integrity of back to the authors.

If you feel you have the social capital with nix leadership to try another RFC for an optional signature verifier hooked into nix user tools for both nix core and expressions, with optional signing from either authors or community members, that would be welcome progress and would certainly make me give the path of directly working with the nix community (vs forking) a second look.

My ideal would be to point nix at a repo of public keys, and nix would not download or execute anything whose supply chain did not start with those keys. This would be incredible for say docker build containers that only need small subsets of packages like compilers for a start... and then eventually a full bootable set of packages supported.

I would happily review any such proposals are capable of meeting threat models requiring high supply chain security.

Contact info on https://lance.dev

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#174

Post by JamesSwift » Tue, Oct 31, 2023, 1:46 PM UTC

chpatrick wrote: post #95

goodpoint wrote: post #76

Earlier quoted context omitted.

> unlike in mainstream distros Debian has been building in a clean sandbox with only required, tracked dependencies since decades. It's also building the large majority of packages reproducibly including the binary and whole installation packages (not just the sources like nixos)

> not just the sources like nixos Not sure what you mean by that, the Nix packages that are reproducible have reproducible binaries. In the Nixos world there isn't really a concept of a "binary/installation package" like in Debian or elsewhere. Everything can be rebuilt from source on any machine, but because everything is hashed, if the official binary caches have already built something with the same inputs, they c…

> In the Nixos world there isn't really a concept of a "binary/installation package" like in Debian or elsewhere. Everything can be rebuilt from source on any machine

Thats not actually the case. A derivation is just an abstract concept that combines a hash with a "thing". Here is an example [1] of a pre-compiled mono binary that gets downloaded and installed (after patching paths).

[1] - https://github.com/NixOS/nixpkgs/blob/0cbe9f69c234a7700596e9...

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#175

Post by __MatrixMan__ » Tue, Oct 31, 2023, 2:39 PM UTC

lrvick wrote: post #173

__MatrixMan__ wrote: post #172

Earlier quoted context omitted.

"A fork of nix" sounds awful drastic. Seems like you just need a curated list of packages which conform to some standard re: - getting a dev sig when they get the code - checking that sig - adding a packager sig And then you need to somehow inject a check before the user comes in contact with the outputs: > are these signers in my trusted list? Your users can then enable dev-sig-mode and point their sig checker at th…

I went to Nix liking everything about it except the fact there was no way to even -optionally- add signatures to NixOS core component contributions and package expressions, let alone verifying either of those as an end user. I was informed the way to get changes into Nix was go ask the Nix Gods in the form of an RFC. I did that, and there was some interest, but ultimately it descended into bike shedding about optimal…

I certainly don't have that social capital at right now, but I do intend to be hacking around in these areas in the future. If I get something useful working then I will probably be in a position to build that capital.

Because now that we have reframed it as something that puts users in explicit control of their trust graph and not something that would make NixOS like all the other distros, it's a feature that I'd like to use.

----

I'm working on a data annotation layer which I intend to apply as a filesystem and use nix and my test case. Edits are conceived of as annotations, so there's this sense that every file can be built by following a path of edits from the empty file. This makes for slow reads, but near-instant copies (you're just adding a new pointer at the same position in the edit graph as the file you're copying). It's a strange design choice, but it would solve some problems when using nix flakes with large repositories (everything gets copied into the nix store and it can take a while).

Signatures are exactly the sort of thing I was imagining living in an annotation. Ideally, the annotation adheres to patterns in the data and not the named file, so if the package applied a patch which invalidates the signature then the signature doubles as a link to the original, which can be diffed with the patched version and the user can be shown why it fails, not just that it fails. It's a long shot but that's the dream.

With a bit of luck I'll find a more elegant way to handle this without playing whack-a-mole with each user facing utility that builds a derivation. Maybe some kind of dashboard which shows you which files the system is rendering and whether they have associated signatures (or other metadata, known malicious, etc). The challenge, in the signature case, will be knowing which files are ok to be unsigned and which need to fail on read if not signed. Certainly we can't require a separate signature for every file we render.

It might be a long time coming though, this work proceeds on weekends and holidays and it's pretty far from a useful state. I'm still fiddling with tuning the rolling hash based fragmentation algorithm such that files are constructed out of right-sized fragments which end up being reused if the files are similar.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#176

Post by drdrey » Tue, Oct 31, 2023, 11:07 PM UTC

stcg wrote: post #103

edgyquant wrote: post #93

Earlier quoted context omitted.

That sounds like runtime differences not a difference between two binaries

The difference in binaries must be caused by some runtime difference of a compiler.

That's right, look at this thread for example:

https://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20...

The Global Value Numbering pass in LLVM was iterating over `DenseMap`, so the iteration order was dependent on the value of BasicBlock pointers. This could lead to the same source files and compiler producing different binaries.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#177

Post by k8svet » Thu, Nov 02, 2023, 9:47 PM UTC

lifeisstillgood wrote: post #155

k8svet wrote: post #88

Earlier quoted context omitted.

Except it's Docker, and like virtually all Dockerfiles, it immediately runs "apt-get update", tossing reproducibility out the window.

yes, hence the (planned) move to flake.nix and then compile to docker.

Music to my ears! God speed, good luck!

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#178

Post by rekado » Fri, Nov 10, 2023, 9:40 PM UTC

hardwaresofton wrote: post #165

onedognight wrote: post #7

Rebuilding the minimal ISO from source is an impressive milestone on the journey to a system that builds from source reproducibly. Guix had an orthogonal but equally impressive milestone on the same journey recently[0], bootstrapping a full compiler toolchain from a single reproducible 357 byte binary without any other binary compiler blobs. These two features may one day soon be combined to reproducibly build a full…

Classic HN, the top comment in a Nix post is about Guix. Nix has more packages and advocacy (even if the vast majority of people exposed to nix/guix will never actually use it), but Guix is a lot more interesting to me with the expressive power of scheme on offer. That said, there are some sharp edges[0] that seem a bit harder to figure out (is this just as inscrutable/difficult as nix?). Does anyone have some good l…

> Does anyone have some good links with people hacking/working with guix?

We've just had a conference about Guix in HPC: https://youtu.be/dT5S72x18R8

This is a recording of a stream for the second day with talks about large scale deployments of Guix System in HPC.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#179

Post by hardwaresofton » Fri, Nov 10, 2023, 10:43 PM UTC

rekado wrote: post #178

hardwaresofton wrote: post #165

Earlier quoted context omitted.

Classic HN, the top comment in a Nix post is about Guix. Nix has more packages and advocacy (even if the vast majority of people exposed to nix/guix will never actually use it), but Guix is a lot more interesting to me with the expressive power of scheme on offer. That said, there are some sharp edges[0] that seem a bit harder to figure out (is this just as inscrutable/difficult as nix?). Does anyone have some good l…

> Does anyone have some good links with people hacking/working with guix? We've just had a conference about Guix in HPC: https://youtu.be/dT5S72x18R8 This is a recording of a stream for the second day with talks about large scale deployments of Guix System in HPC.

Thanks for this! Going to give it a watch :)