Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

discourse.nixos.org

Post reply on HN
Print view View original

111–120 of 179 posts

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#111

Post by tripdout »

londons_explore wrote: post #101
beisner wrote: post #80

Earlier quoted context omitted.

if you have a reproducible build, then the notion of "software was built on date by user " is kind of useless information, no? Because it does not matter - if you can verify that a specific git hash of a codebase results in a particular binary through reproducible builds, a malicious adversary could have built it yesterday and given it to me and i can be almost surely confident (barring hash-collisions...) it's ident…

The usecase is: user wants an easy way to know, from the GUI of some running software, exactly what build/version/git commit/branch/date they're running - perhaps to file a bug report for example. The actual build date doesn't matter if the software is reproducible - but its a proxy for 'how out of date is this software'.

In that case, you can report the Git SHA and still be reproducible.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#112

Post by someplaceguy »

dataflow wrote: post #107
someplaceguy wrote: post #105

Earlier quoted context omitted.

Languages such as Standard ML and others (Scheme? Lisp? Not sure...) have implementations that can save the current state of the heap into a binary. This is used in theorem provers, for example, so that you don't have to verify proofs of theorems over and over again (which can be very slow). Instead, you verify them once, save the state of the heap to disk (as a binary ELF, for instance) and then you can run the bina…

Thanks, yeah. So I guess the concrete example I would cite here is that the most natural (and most efficient?) way of persisting std::map would introduce pointer ordering into the output.

Just like the most natural (and most efficient?) way of persisting any std::unordered_map can result in a completely randomly-ordered output, due to a DoS mitigation that some commonly-used language runtimes have.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#113

Post by mgaunard »

amelius wrote: post #55
mgaunard wrote: post #43

Don't you have to fake the system time to do this? The time often ends up inside the binaries one way or another.

Could you name an example of how (and for what reason) this might happen?

GCC embeds timestamps in o/gcno/gcda files to check they match.

It's mostly annoying as gcov will actively prevent you from using gcda files from a different but equivalent binary than what generated the gcno.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#114

Post by kaba0 »

edgyquant wrote: post #92
kaba0 wrote: post #34

Earlier quoted context omitted.

Parallelism. There might be actions that are not order-independent, and the state of the CPU might result in slightly different binaries, but all are correct.

Why does this matter though? Why does order of compilation result in a different binary?

Just some random, made up example: say you want to compile an OOP PL that has interfaces and implementations of that. You discover reachable implementations through static analysis, which is multi-threaded. You might discover implementations A,B,C in any order — but they will get their methods placed in the jump table based on this order. This will trivially result in semantically equivalent, but not binary-equivalent executables.

Of course there would have been better designs for this toy example, but binary reproducibility is/was usually not of the highest priority historically in most compiler infrastructures, and in some cases it might be a relatively big performance regression to fix, or simply just a too big refactor.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#115

Post by prateem_ »

Foxboron wrote: post #46
mauricioc wrote: post #39

Earlier quoted context omitted.

To emphasize chpatrick's point below, there are two definitions of "reproducibility" in this context: * Input reproducibility, meaning "perfect cache invalidation for inputs". Nix and Guix do this perfectly by design (which sometimes leads to too many rebuilds). This is not on the radar for Debian and Arch Linux, which handle the rebuild problem ("which packages should I rebuild if a particular source file is updated…

> Your comment can be misinterpreted as saying "Nix does not do binary reproducibility very well, just input reproducibility", which is false. It's only "false" as nobody has actually tried to rebuild the entire package repository of nixpkgs, which to my knowledge is an open problem nobody has really worked on. The current result is "only" ~800 packages and the set has regular regressions.

I am probably misunderstanding your point BUT I have actually depended on Nix for "reproducible docker images" for confidential compute usecase so that all parties can independently verify the workload image hash. Rarely (actually only once) it did fail to produce bit identical images every other time it successfully produced bit identical images on very different machine setups. Granted this is not ISO but docker images, but I would say Nix does produce reproducible builds for many real world complex uses.

Ref: [1] https://gitlab.com/prateem/turning-polyglot-solutions-into-t... [2] https://discourse.nixos.org/t/docker-image-produced-by-docke...

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#116

Post by Ericson2314 »

rgoulter wrote: post #36
candiddevmike wrote: post #24

Earlier quoted context omitted.

Nix is a declarative OS, where you describe what the OS should look like, instead of Ansible where you give the OS steps to follow. Silverblue and Nix are orthogonal aside from being Linux distributions--Silverblue is attempting to change how software is delivered using only containers on an immutable host. If you're interested in an Ansible alternative that uses Jsonnet and state tracking to somewhat mimic Nix, chec…

> Nix is a declarative OS I think precision is important. "Nix" refers to the package manager (and the language the package manager uses). Whereas it's "NixOS" that's the OS which makes use of Nix to manage the system configuration.

Thank you. This is important. Too bad our website doesn't make it clear at all.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#117

Post by Foxboron »

prateem_ wrote: post #115
Foxboron wrote: post #46

Earlier quoted context omitted.

> Your comment can be misinterpreted as saying "Nix does not do binary reproducibility very well, just input reproducibility", which is false. It's only "false" as nobody has actually tried to rebuild the entire package repository of nixpkgs, which to my knowledge is an open problem nobody has really worked on. The current result is "only" ~800 packages and the set has regular regressions.

I am probably misunderstanding your point BUT I have actually depended on Nix for "reproducible docker images" for confidential compute usecase so that all parties can independently verify the workload image hash. Rarely (actually only once) it did fail to produce bit identical images every other time it successfully produced bit identical images on very different machine setups. Granted this is not ISO but docker im…

I'm very sure you are actually just rebuilding the container images themselves, not the package tree you are depending on. Building reproducible ISOs, or container images, with a package repository as a base isn't particularly hard these days.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#118

Post by pharmakom »

dataflow wrote: post #70
pharmakom wrote: post #64

Earlier quoted context omitted.

With caching, just the time to download the artefact.

Doesn't caching completely defeat the point of bootstrapping? How do you know the cached artifact is correct? You have to build it manually to verify that, at which point you're still building manually...

You have a hash that n trusted parties agree on. This is enabled by reproducible builds.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#119

Post by acaloiar »

acaloiar wrote: post #68
dwheeler wrote: post #41

Earlier quoted context omitted.

There are many specific causes, time stamps probably being the most common issue. You can see a list of common issues here: https://reproducible-builds.org/docs/ The main overall issue is that developers don't test to ensure they reproduce. Once it's part of the release tests it tends to stay reproducible.

I agree, although I wouldn't describe the overall issue as developers not testing to ensure reproducibility. The reason most builds aren't reproducible is that build reproducibility isn't a goal for most projects. It would be great if 100% of builds were reproducible, but I don't believe developers shouldn't be testing for reproducibility unless it's a defined goal. As generalized reproducible build tooling (guix, ni…

Typo: I don't believe developers shouldn't be -> I don't believe developers should be

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#120

Post by TacticalCoder »

onedognight wrote: post #7

Rebuilding the minimal ISO from source is an impressive milestone on the journey to a system that builds from source reproducibly. Guix had an orthogonal but equally impressive milestone on the same journey recently[0], bootstrapping a full compiler toolchain from a single reproducible 357 byte binary without any other binary compiler blobs. These two features may one day soon be combined to reproducibly build a full…

That is amazing and it is great to see there are people out there fighting the good fight (while others ask: "but where's the benefit!? if there's a backdoor, everybody is still going to get the backdoor!").

> it gives us a reliable way to verify the binaries we ship are faithful to their sources

That's the thing many don't understand: it's not about proving that the result is 100% trustable. It's about proving it's 100% faithful to the source. Which means that should monkey business be detected (like a sneaky backdoor), it can be recreated deterministically 100% of the time.

In other words for the bad guys: nowhere to run, nowhere to hide.

Post reply on HN