NixOS Reproducible Builds: minimal ISO successfully independently rebuilt • Hacker News Board

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#161

Post by zimbatm » Mon, Oct 30, 2023, 11:47 AM UTC

acaloiar wrote: post #68

dwheeler wrote: post #41

Earlier quoted context omitted.

There are many specific causes, time stamps probably being the most common issue. You can see a list of common issues here: https://reproducible-builds.org/docs/ The main overall issue is that developers don't test to ensure they reproduce. Once it's part of the release tests it tends to stay reproducible.

I agree, although I wouldn't describe the overall issue as developers not testing to ensure reproducibility. The reason most builds aren't reproducible is that build reproducibility isn't a goal for most projects. It would be great if 100% of builds were reproducible, but I don't believe developers shouldn't be testing for reproducibility unless it's a defined goal. As generalized reproducible build tooling (guix, ni…

It's also because the cost of making things reproducible is still too high.

We have the tooling, but it still takes a bit of effort from the developer's side to integrate those into their CI pipeline.

Eventually we will get to a place where this will be the default. It will be integrated into day-to-day tooling like `cargo release`, `npm publish`, ...

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#162

Post by Foxboron » Mon, Oct 30, 2023, 12:22 PM UTC

chpatrick wrote: post #140

iopq wrote: post #131

Earlier quoted context omitted.

Sure there is, the NUR has a few thousand community packages that are not ready for release The nixpkgs are all official packages, it's just really easy to become a maintainer (you make a pull request adding the package you want to maintain)

I'm just saying that X% of arch official packages being reproducible isn't a complete statistic when many day to day things are in AUR, most of which are in nixpkgs not NUR.

AUR is unsupported, and the fact that nixpkgs decides to support everything is for them to decide.

Reaching for reproducible builds support in Arch is a more attainable goal than for nixpkgs. Properly maintaing 80k packages without regressions is going to be a lot more work in the long term.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#163

Post by preisschild » Mon, Oct 30, 2023, 1:10 PM UTC

londons_explore wrote: post #62

Smaug123 wrote: post #35

Earlier quoted context omitted.

Loads of things. Obvious ones where the decision is explicitly taken to be non-reproducible include timestamps and authorship information. There are also other places where reproducibility is implicitly broken by default: e.g. many runtimes don't define the order of entries in a hashmap, and then the compiler iterates over a hashmap to build the binary.

I can see why devs would want "This Software was built on 10/10/2007 by bob7 from git hash aaffaaff" to appear on the splash screen of software. How do you get similar behaviour while having a reproducible build? Can you, for example, have the final binary contain a reproducible part, and another section of the elf file for deliberately non-reproducible info?

You can still include the git hash or a git tag/release version info, since the reproducer has the same git repo anyway.

But including timestamp of build would necessitate “spoofing” the timestamp by the reproducer to be the same as the original.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#164

Post by preisschild » Mon, Oct 30, 2023, 1:15 PM UTC

mgaunard wrote: post #43

Don't you have to fake the system time to do this? The time often ends up inside the binaries one way or another.

You would just either not include the timestamp at all in all builds, or set 0, so the build date is 1970 everywhere.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#165

Post by hardwaresofton » Mon, Oct 30, 2023, 1:58 PM UTC

onedognight wrote: post #7

Rebuilding the minimal ISO from source is an impressive milestone on the journey to a system that builds from source reproducibly. Guix had an orthogonal but equally impressive milestone on the same journey recently[0], bootstrapping a full compiler toolchain from a single reproducible 357 byte binary without any other binary compiler blobs. These two features may one day soon be combined to reproducibly build a full…

Classic HN, the top comment in a Nix post is about Guix.

Nix has more packages and advocacy (even if the vast majority of people exposed to nix/guix will never actually use it), but Guix is a lot more interesting to me with the expressive power of scheme on offer.

That said, there are some sharp edges[0] that seem a bit harder to figure out (is this just as inscrutable/difficult as nix?).

Does anyone have some good links with people hacking/working with guix? Maybe some blogs to follow?

I care more about the server use-case and I'm a bit worried about the choice of shepherd over something more widely used like systemd and some of the other libre choices which make Guix what it is. Guix is fine doing what it does, but it seems rather hard to run a Guix-managed system with just a few non-libre parts, which is a non-starter.

Also, as mentioned elsewhere in this thread, the lack-of-package-signing-releases is kind of a problem for me. Being source and binary compatible is awesome, but I just don't have time to follow the source of every single dependency... At some point you have to trust people (and honestly organizations) -- the infrastructure for that is signatures.

Would love to hear of people using Guix in a production environment, but until then it seems like stuff like Fedora CoreOS/Flatcar Linux + going for reproducible containers & binaries is what makes the most sense for the average devops practitioner.

CoreOS/Flatcar are already "cutting edge" as far as I'm concerned for most ops teams (and arguably YAGNI), and it seems like Nix/Guix are just even farther afield.

[EDIT] Nevermind, Guix has a fix for the signature problem, called authorizations![1]

[0]: https://unix.stackexchange.com/questions/698811/in-guix-how-...

[1]: https://guix.gnu.org/manual/devel/en/html_node/Specifying-Ch...

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#166

Post by __MatrixMan__ » Mon, Oct 30, 2023, 2:54 PM UTC

lrvick wrote: post #156

__MatrixMan__ wrote: post #148

Earlier quoted context omitted.

You're talking sense, but this is due diligence for a developer, not for an operating system or a package manager. You're free to map your package definitions to the commits they contain and verify any signatures that you find there, but that process will have nothing to do with whether other people have signed the instructions that your machine follows to fetch and build the contents of that commit.

Sure I could write a lot of tooling to try and do this sort of basic verification manually in Nix every single time i pin a new package, but at that point why am I using nix over distros that have native support for maintainer level supply chain integrity? Compare to the Arch model where all official packages must be signed with keys belonging to a reasonably well vetted web of trust. Developers outside the web of tr…

Nix is two steps forward in these ways because they chose to focus on making things composable and repeatable instead of being curators of quality and trustworthiness.

One magical thing about Nix is that there's a very small divide between managing to install software in the first place and creating an artifact that others can use for the same purpose.

Because of this, practically every NixOS user has their configuration in source control--we're basically each building our own Linux distro with only packages that we trust. Some of us probably sign those commits too.

The ecosystem is useful because it encourages this kind if participation. Having a list of privileged maintainers would interrupt this.

It's unsurprising that a group of people who has worked quite hard to make this possible would be uninterested in creating a scheme whereby they are now responsible for determining which of their users creations is legitimate. Nix attracts users who are interested in that sort of thing, so let it be a userspace problem.

If you want to curate a list of trustworthy packages and work with their developers to set up a chain of trust that starts in a yubikey extends to a signature in a flake output, then I'll help because that sounds like useful work, but I wish you would stop criticizing a brick for not already being a house.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#167

Post by lrvick » Mon, Oct 30, 2023, 10:09 PM UTC

__MatrixMan__ wrote: post #166

lrvick wrote: post #156

Earlier quoted context omitted.

Sure I could write a lot of tooling to try and do this sort of basic verification manually in Nix every single time i pin a new package, but at that point why am I using nix over distros that have native support for maintainer level supply chain integrity? Compare to the Arch model where all official packages must be signed with keys belonging to a reasonably well vetted web of trust. Developers outside the web of tr…

Nix is two steps forward in these ways because they chose to focus on making things composable and repeatable instead of being curators of quality and trustworthiness. One magical thing about Nix is that there's a very small divide between managing to install software in the first place and creating an artifact that others can use for the same purpose. Because of this, practically every NixOS user has their configura…

My criticism is not so much on NixOS itself. If it wants to be a fun, composable, and easy to work on with minimal participation friction, fair enough. The yolo approach to contribution integrity has clearly resulted in fast development of ideas that might have been unable to grow in other distros.

I suppose where my, perhaps misdirected, anxiety comes from is that I run a security consulting company and see NixOS being used as-is in high risk applications, to compile binaries responsible for protecting peoples property or safety, and on the workstations of production engineering teams. Places where it has no business being because supply chain integrity is a non goal.

Maybe you are right though, and the answer is not trying to add supply chain security practices onto a community that does not want them, but to create a security focused fork of that distro that can inherit all that great community work and be a drop-in replacement for NixOS in environments where supply chain security is of critical importance.

I tried and failed to get buy-in for even -optional- expression integrity support back in 2018 and gave up on nix after that. https://github.com/NixOS/rfcs/pull/34

I did prototype a git multisig solution in the years following that though: https://git.distrust.co/public/supsig https://git.distrust.co/public/git-sig

It is already being used in production by some as nothing better exists atm.

A security focused Nix fork that imports, reviews, and git-sig (or similar) signs commits from NixOS and has signature verification built into the package manager, is probably the only way forward, and I would be willing to ally with others interested in this.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#168

Post by lrvick » Mon, Oct 30, 2023, 11:03 PM UTC

__MatrixMan__ wrote: post #146

lrvick wrote: post #138

Earlier quoted context omitted.

"Expecting to use their signatures for anything meaningful strikes me as a bit like expecting product support from a delivery driver." And yet most of the packages from most major linux distributions are signed. If you are going to spend hours maintaining a package, it takes only an extra half a second to tap a yubikey to prevent someone from impersonating you. Package maintainers from say Arch and Debian go through…

> The developers github account was compromised due to a sim swap on their email account while they were on vacation, and someone pushed a fake commit as that person. ...which is why it's irresponsible to sign packages unless you have a strong enough relationship with the developer (or are that developer) such that you would notice the malicious release. Signing packages without that level of trust creates a false se…

If a Github account was compromised, the attacker would still be unable to sign with the key that user has historically used. If all that was done was pin keys of developers and sound alarm bells if the key changes or changes revert to being unsigned, then this is still of significant value as a first step.

I agree most of the value of signing happens if we establish a web of trust with signed keys of known developers with a history of behaving non maliciously -and- also have signed code review by people in the same trust network to limit risk of coercion, or malware stealing yubikey taps on an endpoint.

Also, saying they are lazy about security is unfair. They just invested in areas of security most distros ignored, but sadly at the expense of the security most distros get right. The regression feels irresponsible, but as I said in my other post maybe we need to separate concerns between the AUR-style rapid development expression repohsitory and a new separate one select packages graduate to that have web of trust with well known maintainer/reviewer keys.

I could get behind that.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#169

Post by lrvick » Mon, Oct 30, 2023, 11:14 PM UTC

amarshall wrote: post #145

lrvick wrote: post #133

Now if only they would have maintainers sign packages like almost every other linux distribution has done since the 90s, so we have any idea if the code everyone is building is the same code submitted and reviewed by known individuals. Until signing is standardized, it is hard to imagine using nix in any production use case that protects anything of value.

What distros do this? As far as I can tell Debian doesn’t anymore (the build system signs builds), Fedora doesn’t, and Arch…I can’t tell but I don’t think it does. But maybe I’m wrong. The NixOS build system signs all build outputs with its key and the signature is verified upon download. If you’re paranoid, Nix at least makes it easy to just build everything from source.

Yes packages are reproduced and signed. I get that and this is fantastic. That is not the big gap I am talking about which is making sure the expressions themselves even use the correct sources and are not tampered with. Making sure the central signing key is not compromised is pretty critical too, but that can be mitigated with independent reproduction servers that also sign, requiring a minimum of 3 before a new package is trusted.

As for Debian, maintainers must still all be part of the web of trust with an application process and a registered key in the web of trust to contribute packages. https://wiki.debian.org/DebianMaintainer#step_5_:_Keyring_up...

Arch also still has each maintainer sign the packages they maintain, and there is a similar approval process to earn enough trust in your keys to become a maintainer, so there is a decentralized web of trust. Then you add reproducible builds on top of that to almost complete the story.

Signed reviews would remove the remaining SPOFs in debian and arch but sadly no one does this. Nix however does not even allow -optional- key pinning or verify signatures of any authors that choose to sign expressions or their commits, so NixOS is way behind Arch and Debian in this respect.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#170

Post by flkiwi » Tue, Oct 31, 2023, 12:44 AM UTC

__MatrixMan__ wrote: post #150

flkiwi wrote: post #51

Earlier quoted context omitted.

Another good option: https://github.com/Misterio77/nix-starter-configs I started with this one, the minimal version, then moved on to something more like the standard version, and now I'm moving on to something based on his much more complicated and flexible build in a different repo. I had been flailing, then this repo made it click.

Same here, that repo is fantastic.

His documentation has gotten better too! I actually just rebuilt my entire config based on his updated "standard" version. I want to use his "non-starter" config too, bc it seems remarkably powerful, but ... I need more time and brain to do that.

And, just to be clear, I rebuilt my entire config in less than an hour, copying and pasting where necessary, then it took about 20 minutes (!!) to download and setup all my software. Only one error, so I consider that a significant My NixOS Journey accomplishment! Granted, because it's NixOS, I have NO IDEA AT ALL what the error was, lol.