Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

discourse.nixos.org

Post reply on HN
Print view View original

71–80 of 179 posts

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#71

Post by [deleted] »

londons_explore wrote: post #62
Smaug123 wrote: post #35

Earlier quoted context omitted.

Loads of things. Obvious ones where the decision is explicitly taken to be non-reproducible include timestamps and authorship information. There are also other places where reproducibility is implicitly broken by default: e.g. many runtimes don't define the order of entries in a hashmap, and then the compiler iterates over a hashmap to build the binary.

I can see why devs would want "This Software was built on 10/10/2007 by bob7 from git hash aaffaaff" to appear on the splash screen of software. How do you get similar behaviour while having a reproducible build? Can you, for example, have the final binary contain a reproducible part, and another section of the elf file for deliberately non-reproducible info?

[deleted]

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#72

Post by mbakke »

oever wrote: post #66
mbakke wrote: post #59

Earlier quoted context omitted.

Typically part of a "version string": $ python3 Python 3.10.7 (main, Jan 1 1970, 00:00:01) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> Perhaps a relic from when software had to be manually updated?

On NixOS, I think the release time or commit time is used: $ python3 Python 3.10.11 (main, Apr 4 2023, 22:10:32) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> That is more useful than the build time.

How is that possible? Is nixpkgs an input to the Python derivation? Or do packagers "hard code" a value every time they modify the Python build code? Automated tooling that sets it after pull requests? Something else? :-)

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#73

Post by __MatrixMan__ »

dataflow wrote: post #70
pharmakom wrote: post #64

Earlier quoted context omitted.

With caching, just the time to download the artefact.

Doesn't caching completely defeat the point of bootstrapping? How do you know the cached artifact is correct? You have to build it manually to verify that, at which point you're still building manually...

1. hash it

2. rebuild it without the cache

3. hash that

4. compare

Or, trust somebody who has. Inconvenient, but is there any other way to establish trust in the correspondence between code and a binary?

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#74

Post by [deleted] »

chpatrick wrote: post #56
dicytea wrote: post #9

Earlier quoted context omitted.

> Guix, Archlinux, Debian do the binary reproducibility better than Nix / NixOS / Nixpkgs. Huh, didn't know that Arch Linux tests reproducibility. It's apparently 85.6% reproducible: https://reproducible.archlinux.org I wonder how much work would be needed for NixOS, considering it has more than 80k packages in the official repository.

I think that's also a bit of an unfair comparison given the number of AUR packages you usually use on Arch. With nixpkgs there isn't a distinction between official and community packages.

[deleted]

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#75

Post by 6581 »

rssoconnor wrote: post #57
Thrir94994i wrote: post #11

Earlier quoted context omitted.

357 bytes for bootstrap compiler binary is VERY impressive!

To be fair, it is 357 bytes ... plus a POSIX operating system. Still, that POSIX operating system bit is also being worked on.

Isn't that what builder-hex0 does?

https://github.com/ironmeld/builder-hex0

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#76

Post by goodpoint »

chpatrick wrote: post #23
mihalycsaba wrote: post #3

Sorry for being dense, but I thought one of the main reason for nixos's existence is reproducibilty. I thought they have these kinds of things solved already. I have only ~2 hours experience with Nixos, wanted to try hyprland, I thought it would be easier on Nixos since hyprland needs a bit of setup and maybe it's easier to use someone else's config on nixos, than on some other distro. Finding a config was hard too,…

> Sorry for being dense, but I thought one of the main reason for nixos's existence is reproducibilty. I thought they have these kinds of things solved already. Nixos has the advantage that everything is built in its own sandbox with only its explicitly declared (and hashed) dependencies available, unlike in mainstream distros where it's the full system environment, so in many cases you already get the same binary ev…

> unlike in mainstream distros

Debian has been building in a clean sandbox with only required, tracked dependencies since decades.

It's also building the large majority of packages reproducibly including the binary and whole installation packages (not just the sources like nixos)

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#77

Post by mbakke »

dataflow wrote: post #70
pharmakom wrote: post #64

Earlier quoted context omitted.

With caching, just the time to download the artefact.

Doesn't caching completely defeat the point of bootstrapping? How do you know the cached artifact is correct? You have to build it manually to verify that, at which point you're still building manually...

Guix has tooling to verify binaries:

https://guix.gnu.org/en/manual/en/html_node/Invoking-guix-ch...

"guix build --no-grafts --no-substitutes --check foo" will force a local rebuild of package foo and fail if the result is not bit-identical. "guix challenge" will compare your local binaries against multiple cache servers.

I build everything locally and compare my results with the official substitute servers from time to time.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#78

Post by raboof »

mbakke wrote: post #27

Very impressive milestone, congrats to those who made this possible! > [...] actually rebuilding the ISO still introduced differences. This was due to some remaining problems in the hydra cache and the way the ISO was created. Can anyone shed some light on the fix for "how the ISO was created"? I attempted making a reproducible ISO a while back but could not make the file system create extents in a deterministic fash…

For NixOS, it's in the 'how did we reproduce' section of the article: the last step of that process produces the iso in the ./result/iso directory.

It sounds like what you're looking for is the commands that that build invoked, but I'm not sure what step you're looking for. For example, the xorriso invocations are at https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-...

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#79

Post by reactordev »

ralferoo wrote: post #69
jowea wrote: post #30

Earlier quoted context omitted.

This[0] is basically the hand-documentation of those bytes then. Handwritten ELF header and assembly code. [0] https://github.com/oriansj/bootstrap-seeds/blob/master/POSIX...

Just had a read of this to see what it did... And I must admit, I don't understand what purpose this is supposed to serve. All it seems to do is convert hex into binary and dump it to a file. Not sure how that's any more useful than just copying the binary for next stage directly, after all this binary had to get on the system somehow.

It’s the first stage. Likely piped. Hence the hex out. The context on how it’s called is key: https://github.com/oriansj/stage0-posix-x86/blob/e86bf7d304b...

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#80

Post by beisner »

londons_explore wrote: post #62
Smaug123 wrote: post #35

Earlier quoted context omitted.

Loads of things. Obvious ones where the decision is explicitly taken to be non-reproducible include timestamps and authorship information. There are also other places where reproducibility is implicitly broken by default: e.g. many runtimes don't define the order of entries in a hashmap, and then the compiler iterates over a hashmap to build the binary.

I can see why devs would want "This Software was built on 10/10/2007 by bob7 from git hash aaffaaff" to appear on the splash screen of software. How do you get similar behaviour while having a reproducible build? Can you, for example, have the final binary contain a reproducible part, and another section of the elf file for deliberately non-reproducible info?

if you have a reproducible build, then the notion of "software was built on date by user" is kind of useless information, no? Because it does not matter - if you can verify that a specific git hash of a codebase results in a particular binary through reproducible builds, a malicious adversary could have built it yesterday and given it to me and i can be almost surely confident (barring hash-collisions...) it's identical to a known trusted team member building it.

Having information about which git has was used, as well as the time it was published, is part of the source distribution so an output can contain references to these inputs and still be deterministic w.r.t. those inputs.

If you REALLY want to know when/who built something, you could add in an auxiliary source file which contains that information, which is required to build. Which is essentially what compilers which leverage current time do anyway, it's just implicit.

Post reply on HN