Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

discourse.nixos.org

Post reply on HN
Print view View original

81–90 of 179 posts

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#81

Post by drdrey »

dataflow wrote: post #42
ahmedfromtunis wrote: post #32

Stupid question as I never worked on something like this before: why isn't reproducibility the default behavior? I mean if 2 copies of a piece of software were compiled from the same source, what stops them from being identical each and every time? I know there are so many moving parts, but I still can't understand how discrepancies can manifest themselves.

Sometimes it's randomized algorithms, sometimes it's performance (e.g. it might be faster not to sort something), sometimes it's time or environment-dependent metadata, sometimes it's thread interleaving, etc.

a very common one is pointer values being different from run to run and across different operating systems. Any code that intentionally or accidentally relies on pointer values will be non-deterministic

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#82

Post by Smaug123 »

mbakke wrote: post #72
oever wrote: post #66

Earlier quoted context omitted.

On NixOS, I think the release time or commit time is used: $ python3 Python 3.10.11 (main, Apr 4 2023, 22:10:32) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> That is more useful than the build time.

How is that possible? Is nixpkgs an input to the Python derivation? Or do packagers "hard code" a value every time they modify the Python build code? Automated tooling that sets it after pull requests? Something else? :-)

GCC respects SOURCE_DATE_EPOCH, and Nixpkgs has specific support for setting that environment variable: https://github.com/NixOS/nixpkgs/blob/92fdbd284c262f3e478033... (although I haven't proved that this is actually how it works for cpython's build).

Irrelevant spelunking details follow:

That string is output by cpython to contain the contents of the __DATE__ C macro (https://github.com/python/cpython/blob/fa35b9e89b2e207fc8bae... which calls to https://github.com/python/cpython/blob/fa35b9e89b2e207fc8bae... which uses the __DATE__ macro at https://github.com/python/cpython/blob/fa35b9e89b2e207fc8bae... ).

Cpython is defined in nixpkgs at https://github.com/NixOS/nixpkgs/blob/92fdbd284c262f3e478033... which I imagine (but haven't proved) uses GCC.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#83

Post by Smaug123 »

londons_explore wrote: post #62
Smaug123 wrote: post #35

Earlier quoted context omitted.

Loads of things. Obvious ones where the decision is explicitly taken to be non-reproducible include timestamps and authorship information. There are also other places where reproducibility is implicitly broken by default: e.g. many runtimes don't define the order of entries in a hashmap, and then the compiler iterates over a hashmap to build the binary.

I can see why devs would want "This Software was built on 10/10/2007 by bob7 from git hash aaffaaff" to appear on the splash screen of software. How do you get similar behaviour while having a reproducible build? Can you, for example, have the final binary contain a reproducible part, and another section of the elf file for deliberately non-reproducible info?

Yeah, "who built this" information belongs in a signing certificate that accompanies the build artefact, not in the artefact itself. The Git hash can certainly appear in the binary (it's a reproducible part of the build input), and the date can instead be e.g. the commit date, which is probably more relevant to a user anyway.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#84

Post by dataflow »

drdrey wrote: post #81
dataflow wrote: post #42

Earlier quoted context omitted.

Sometimes it's randomized algorithms, sometimes it's performance (e.g. it might be faster not to sort something), sometimes it's time or environment-dependent metadata, sometimes it's thread interleaving, etc.

a very common one is pointer values being different from run to run and across different operating systems. Any code that intentionally or accidentally relies on pointer values will be non-deterministic

Would be nice if you could explain how/why this happens, given that normally, pointers aren't persisted.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#85

Post by ndriscoll »

beisner wrote: post #80
londons_explore wrote: post #62

Earlier quoted context omitted.

I can see why devs would want "This Software was built on 10/10/2007 by bob7 from git hash aaffaaff" to appear on the splash screen of software. How do you get similar behaviour while having a reproducible build? Can you, for example, have the final binary contain a reproducible part, and another section of the elf file for deliberately non-reproducible info?

if you have a reproducible build, then the notion of "software was built on date by user " is kind of useless information, no? Because it does not matter - if you can verify that a specific git hash of a codebase results in a particular binary through reproducible builds, a malicious adversary could have built it yesterday and given it to me and i can be almost surely confident (barring hash-collisions...) it's ident…

Your source would also have to have a reference to which exact version of which compiler to use, which versions of which external headers to use, etc. and now you're inventing Nix.

Conceivably there could be a standard for a sidecar file to specify how something was built (e.g. nixpkgs commit hash, or all of the parameters that went into the build). Or content address the inputs, i.e. invent Nix again.

So we could solve this problem by having everyone standardize on using Nix.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#86

Post by oever »

mbakke wrote: post #72
oever wrote: post #66

Earlier quoted context omitted.

On NixOS, I think the release time or commit time is used: $ python3 Python 3.10.11 (main, Apr 4 2023, 22:10:32) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> That is more useful than the build time.

How is that possible? Is nixpkgs an input to the Python derivation? Or do packagers "hard code" a value every time they modify the Python build code? Automated tooling that sets it after pull requests? Something else? :-)

2023-04-04T22:10:32 is the timestamp of Python-3.10.11/Misc/NEWS from https://www.python.org/ftp/python/3.10.11/Python-3.10.11.tar...

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#87

Post by raboof »

mbakke wrote: post #72
oever wrote: post #66

Earlier quoted context omitted.

On NixOS, I think the release time or commit time is used: $ python3 Python 3.10.11 (main, Apr 4 2023, 22:10:32) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> That is more useful than the build time.

How is that possible? Is nixpkgs an input to the Python derivation? Or do packagers "hard code" a value every time they modify the Python build code? Automated tooling that sets it after pull requests? Something else? :-)

The source for the cpython build is the release tarball (https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen...).

In that case, NixOS sets SOURCE_DATE_EPOCH (which I suspect will be picked up by the python build) to the latest timestamp found in that archive (https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-supp...)

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#88

Post by k8svet »

lifeisstillgood wrote: post #19
mihalycsaba wrote: post #12

Earlier quoted context omitted.

I don't remember, some of them needed some other tools installed(like flakes whatever it is), I looked for configs, that looked like they don't need a few more hours to learn and to setup some other tools for them to work. I just wanted to take a quick look at hyprland, I imagined I just use an existing config, I never thought it would need hours of research. Later I installed an arch vm and managed to install hyprla…

I am on a similar journey I built https://github.com/mikadosoftware/workstation (hey nearly 500 stars!) as the idea of defining a reproducible laptop build. I don't think docker is the right level - so my next project when i have free time (!) is to do a box build that then might compile to docker I think there is a sensible point of being able to define via nix both developer workstations and servers

Except it's Docker, and like virtually all Dockerfiles, it immediately runs "apt-get update", tossing reproducibility out the window.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#89

Post by traxys »

dataflow wrote: post #84
drdrey wrote: post #81

Earlier quoted context omitted.

a very common one is pointer values being different from run to run and across different operating systems. Any code that intentionally or accidentally relies on pointer values will be non-deterministic

Would be nice if you could explain how/why this happens, given that normally, pointers aren't persisted.

I think they meant if you cast a pointer to an integer, do some math on that and then store that. Then you will a stored result that will likely differ from run to run

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#90

Post by mbakke »

Smaug123 wrote: post #82
mbakke wrote: post #72

Earlier quoted context omitted.

How is that possible? Is nixpkgs an input to the Python derivation? Or do packagers "hard code" a value every time they modify the Python build code? Automated tooling that sets it after pull requests? Something else? :-)

GCC respects SOURCE_DATE_EPOCH, and Nixpkgs has specific support for setting that environment variable: https://github.com/NixOS/nixpkgs/blob/92fdbd284c262f3e478033... (although I haven't proved that this is actually how it works for cpython's build). Irrelevant spelunking details follow: That string is output by cpython to contain the contents of the __DATE__ C macro ( https://github.com/python/cpython/blob/fa35b9e8…

Thank you! Setting SOURCE_DATE_EPOCH to the most recent file timestamp found in the source input is a clever hack.
Post reply on HN