Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

discourse.nixos.org

Post reply on HN
Print view View original

41–50 of 179 posts

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#41

Post by dwheeler »

ahmedfromtunis wrote: post #32

Stupid question as I never worked on something like this before: why isn't reproducibility the default behavior? I mean if 2 copies of a piece of software were compiled from the same source, what stops them from being identical each and every time? I know there are so many moving parts, but I still can't understand how discrepancies can manifest themselves.

There are many specific causes, time stamps probably being the most common issue. You can see a list of common issues here:

https://reproducible-builds.org/docs/

The main overall issue is that developers don't test to ensure they reproduce. Once it's part of the release tests it tends to stay reproducible.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#42

Post by dataflow »

ahmedfromtunis wrote: post #32

Stupid question as I never worked on something like this before: why isn't reproducibility the default behavior? I mean if 2 copies of a piece of software were compiled from the same source, what stops them from being identical each and every time? I know there are so many moving parts, but I still can't understand how discrepancies can manifest themselves.

Sometimes it's randomized algorithms, sometimes it's performance (e.g. it might be faster not to sort something), sometimes it's time or environment-dependent metadata, sometimes it's thread interleaving, etc.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#44

Post by forkerenok »

15155 wrote: post #18
onedognight wrote: post #7

Rebuilding the minimal ISO from source is an impressive milestone on the journey to a system that builds from source reproducibly. Guix had an orthogonal but equally impressive milestone on the same journey recently[0], bootstrapping a full compiler toolchain from a single reproducible 357 byte binary without any other binary compiler blobs. These two features may one day soon be combined to reproducibly build a full…

At 357 bytes, do you need a reproducible binary at all? I'd think one could hand-document all 357 bytes of machine code and have them be intelligible.

Or tattooed on oneself! Or etched on a dog tag!

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#45

Post by mseepgood »

ahmedfromtunis wrote: post #32

Stupid question as I never worked on something like this before: why isn't reproducibility the default behavior? I mean if 2 copies of a piece of software were compiled from the same source, what stops them from being identical each and every time? I know there are so many moving parts, but I still can't understand how discrepancies can manifest themselves.

Laziness and carelessness of compiler developers.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#46

Post by Foxboron »

mauricioc wrote: post #39
Reventlov wrote: post #4

For those wondering : it should be remembered that the reproducibility of Nix / NixOS / Nixpkgs is only a reproducibility of the sources: if the sources change, one is warned, but it is not a question of the reproducibility of the binaries (which can change at each build). This binary reproducibility of Nix / NixOS / Nixpkgs is indeed not really tested, at least not systematically. Guix, Archlinux, Debian do the bina…

To emphasize chpatrick's point below, there are two definitions of "reproducibility" in this context: * Input reproducibility, meaning "perfect cache invalidation for inputs". Nix and Guix do this perfectly by design (which sometimes leads to too many rebuilds). This is not on the radar for Debian and Arch Linux, which handle the rebuild problem ("which packages should I rebuild if a particular source file is updated…

> Your comment can be misinterpreted as saying "Nix does not do binary reproducibility very well, just input reproducibility", which is false.

It's only "false" as nobody has actually tried to rebuild the entire package repository of nixpkgs, which to my knowledge is an open problem nobody has really worked on.

The current result is "only" ~800 packages and the set has regular regressions.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#47

Post by dathinab »

Reventlov wrote: post #4

For those wondering : it should be remembered that the reproducibility of Nix / NixOS / Nixpkgs is only a reproducibility of the sources: if the sources change, one is warned, but it is not a question of the reproducibility of the binaries (which can change at each build). This binary reproducibility of Nix / NixOS / Nixpkgs is indeed not really tested, at least not systematically. Guix, Archlinux, Debian do the bina…

I think you might want to read the article.

it's about binary bit by bit reproducibility of not just the binaries but also how they get packed into an iso (i.e. r13y.com is outdated, the missing now when it comes to packages beyond the core iso things become complicated to compare due to the subtle but in this regard significant different ways they handle packages, e.g. a bunch of packages you would find on arch in aur you find as normal packages in nix and most of the -bin upstream packages are simply not needed with nix

in general nix makes it easier to create reproducible builds but (independent of nix) this doesn't mean that it's always possible and often needs patching which often but not always is done if you combine this with the default package repository of nix being much larger (>80k) then e.g. arch (through one very common misconception is that the hash in the nix store path is based on the build output, but it's instead based on all sources (weather binary not) used for building the binary in an isolated environment

this means it has not quite the security benefit some people might think it has, but in turn is necessary as it means nix can use software which is non reproducible buildable in a way which still produces reasonable reproducable deplyments (as in not necessary all bits the same but all functionality, compiler-cfgs, dependencies versions, users, configurations etc. being the same

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#48

Post by jonhohle »

mseepgood wrote: post #45
ahmedfromtunis wrote: post #32

Stupid question as I never worked on something like this before: why isn't reproducibility the default behavior? I mean if 2 copies of a piece of software were compiled from the same source, what stops them from being identical each and every time? I know there are so many moving parts, but I still can't understand how discrepancies can manifest themselves.

Laziness and carelessness of compiler developers.

As others have mentioned, there’s sorting issues (are directory entries created in the same order for a project that compiled everything in a directory?), timestamps (archive files and many other formats embed timestamps), and things that you really want to be random (tmpdir on Linux [at least in the past] would create directories of varying length).

I’ve successfully built tools to compare Java JARs that required getting around two of those and other test tools that required the third. I’m sure there are more.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#49

Post by spicybright »

forkerenok wrote: post #44
15155 wrote: post #18

Earlier quoted context omitted.

At 357 bytes, do you need a reproducible binary at all? I'd think one could hand-document all 357 bytes of machine code and have them be intelligible.

Or tattooed on oneself! Or etched on a dog tag!

Nix is a great dog name
Post reply on HN