Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

discourse.nixos.org

Post reply on HN
Print view View original

61–70 of 179 posts

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#61

Post by colordrops »

mihalycsaba wrote: post #3

Sorry for being dense, but I thought one of the main reason for nixos's existence is reproducibilty. I thought they have these kinds of things solved already. I have only ~2 hours experience with Nixos, wanted to try hyprland, I thought it would be easier on Nixos since hyprland needs a bit of setup and maybe it's easier to use someone else's config on nixos, than on some other distro. Finding a config was hard too,…

Nix is reproducteable in tbe environment sense, meaning you can get the exact same setup every time, but not in the bit-for-bit sense, meaning that the compiled binaries will be identical.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#62

Post by londons_explore »

Smaug123 wrote: post #35
ahmedfromtunis wrote: post #32

Stupid question as I never worked on something like this before: why isn't reproducibility the default behavior? I mean if 2 copies of a piece of software were compiled from the same source, what stops them from being identical each and every time? I know there are so many moving parts, but I still can't understand how discrepancies can manifest themselves.

Loads of things. Obvious ones where the decision is explicitly taken to be non-reproducible include timestamps and authorship information. There are also other places where reproducibility is implicitly broken by default: e.g. many runtimes don't define the order of entries in a hashmap, and then the compiler iterates over a hashmap to build the binary.

I can see why devs would want "This Software was built on 10/10/2007 by bob7 from git hash aaffaaff" to appear on the splash screen of software.

How do you get similar behaviour while having a reproducible build?

Can you, for example, have the final binary contain a reproducible part, and another section of the elf file for deliberately non-reproducible info?

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#63

Post by rgoulter »

t0astbread wrote: post #53
ParetoOptimal wrote: post #14

Earlier quoted context omitted.

> I just wanted to take a quick look at hyprland, I imagined I just use an existing config, I never thought it would need hours of research. It shouldn't. You'd want a simple flake to start with that has home-manager (for higher chance of finding declarative vest practice configs and modules) and to add small things to that. I imagine you tried grabbing someone's complex config, modifying it, and ran into issues?

Flakes will hopefully be that soon but I wouldn't recommend starting with flakes when learning Nix in 2023. They're experimental and you still need to learn most of flake-less Nix (except channels and NIX_PATH) anyways. When I started learning/using NixOS about two years ago I found it useful to start out with just Nixpkgs (i.e. what you get out of the box) and only add libraries when I felt they would help me. My fi…

> Flakes will hopefully be that soon but I wouldn't recommend starting with flakes when learning Nix in 2023.

That Flakes provide a consistent entrypoint (and a consistent schema for such) into a codebase would have deferred a significant amount of confusion I had when getting started with Nix.

> They're experimental

The functionality as-is hasn't been changed. The 'experimental' flag itself hasn't been a practical problem.

However, flakes still have some rough edges & design problems to them, and there's some disagreement in the community over how flakes were rolled out.

I'd say for an end user, the benefits far outweigh the costs.

> ... and you still need to learn most of flake-less Nix (except channels and NIX_PATH) anyways.

I think the phrase "flake-less Nix" paints the wrong idea. I'd instead put it: Most of what you need to learn about Nix is unrelated whether the Nix evaluation started from a Flake or not.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#64

Post by pharmakom »

dataflow wrote: post #40
onedognight wrote: post #7

Rebuilding the minimal ISO from source is an impressive milestone on the journey to a system that builds from source reproducibly. Guix had an orthogonal but equally impressive milestone on the same journey recently[0], bootstrapping a full compiler toolchain from a single reproducible 357 byte binary without any other binary compiler blobs. These two features may one day soon be combined to reproducibly build a full…

How long does a fully bootstrapped build take?

With caching, just the time to download the artefact.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#65

Post by mbakke »

dataflow wrote: post #40
onedognight wrote: post #7

Rebuilding the minimal ISO from source is an impressive milestone on the journey to a system that builds from source reproducibly. Guix had an orthogonal but equally impressive milestone on the same journey recently[0], bootstrapping a full compiler toolchain from a single reproducible 357 byte binary without any other binary compiler blobs. These two features may one day soon be combined to reproducibly build a full…

How long does a fully bootstrapped build take?

It obviously depends on the hardware, but IIRC for me maybe 3-4 hours building from the 357 byte seed to the latest GCC.

The early binaries are not very optimized :-)

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#66

Post by oever »

mbakke wrote: post #59
amelius wrote: post #55

Earlier quoted context omitted.

Could you name an example of how (and for what reason) this might happen?

Typically part of a "version string": $ python3 Python 3.10.7 (main, Jan 1 1970, 00:00:01) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> Perhaps a relic from when software had to be manually updated?

On NixOS, I think the release time or commit time is used:

    $ python3
    Python 3.10.11 (main, Apr  4 2023, 22:10:32) [GCC 12.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>>
That is more useful than the build time.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#67

Post by tracnar »

onedognight wrote: post #7

Rebuilding the minimal ISO from source is an impressive milestone on the journey to a system that builds from source reproducibly. Guix had an orthogonal but equally impressive milestone on the same journey recently[0], bootstrapping a full compiler toolchain from a single reproducible 357 byte binary without any other binary compiler blobs. These two features may one day soon be combined to reproducibly build a full…

It's not yet as far as the Guix stage0, but there was an interesting talk about bootstrapping nix from TinyCC at NixCon: https://media.ccc.de/v/nixcon-2023-34402-bootstrapping-nix-a...

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#68

Post by acaloiar »

dwheeler wrote: post #41
ahmedfromtunis wrote: post #32

Stupid question as I never worked on something like this before: why isn't reproducibility the default behavior? I mean if 2 copies of a piece of software were compiled from the same source, what stops them from being identical each and every time? I know there are so many moving parts, but I still can't understand how discrepancies can manifest themselves.

There are many specific causes, time stamps probably being the most common issue. You can see a list of common issues here: https://reproducible-builds.org/docs/ The main overall issue is that developers don't test to ensure they reproduce. Once it's part of the release tests it tends to stay reproducible.

I agree, although I wouldn't describe the overall issue as developers not testing to ensure reproducibility. The reason most builds aren't reproducible is that build reproducibility isn't a goal for most projects.

It would be great if 100% of builds were reproducible, but I don't believe developers shouldn't be testing for reproducibility unless it's a defined goal.

As generalized reproducible build tooling (guix, nix, etc.) becomes more mainstream, I imagine we'll see more reproducible builds as adoption grows and reproducibility is no longer something developers have to "check for", but simply rely upon from their tooling.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#69

Post by ralferoo »

jowea wrote: post #30
15155 wrote: post #18

Earlier quoted context omitted.

At 357 bytes, do you need a reproducible binary at all? I'd think one could hand-document all 357 bytes of machine code and have them be intelligible.

This[0] is basically the hand-documentation of those bytes then. Handwritten ELF header and assembly code. [0] https://github.com/oriansj/bootstrap-seeds/blob/master/POSIX...

Just had a read of this to see what it did... And I must admit, I don't understand what purpose this is supposed to serve.

All it seems to do is convert hex into binary and dump it to a file. Not sure how that's any more useful than just copying the binary for next stage directly, after all this binary had to get on the system somehow.

Re: NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

#70

Post by dataflow »

pharmakom wrote: post #64
dataflow wrote: post #40

Earlier quoted context omitted.

How long does a fully bootstrapped build take?

With caching, just the time to download the artefact.

Doesn't caching completely defeat the point of bootstrapping? How do you know the cached artifact is correct? You have to build it manually to verify that, at which point you're still building manually...
Post reply on HN