Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nixos-unstable’s ISO_minimal.x86_64-Linux is 100% reproducible

discourse.nixos.org

Post reply on HN
Print view View original

171–180 of 232 posts

Re: Nixos-unstable’s ISO_minimal.x86_64-Linux is 100% reproducible

#171

Post by eru »

User23 wrote: post #95
mjg59 wrote: post #53

Earlier quoted context omitted.

Diverse double compilation should allow a demonstration that the toolchain is trustworthy.

Really? How does that accomplish more than proving the build is a fixed point? An attacker may well be aware of the fixed point combinator after all. Edit: I think that tone may have come off as snarky, but I meant it as an honest question. If any expert can answer I'd really appreciate it.

Fixed points don't come in here at all, unless you specifically want to talk about compiling compilers.

Diverse double compilation is useful for run-of-the mill programs, too.

Re: Nixos-unstable’s ISO_minimal.x86_64-Linux is 100% reproducible

#172

Post by eru »

takeda wrote: post #168
cookiengineer wrote: post #144

Earlier quoted context omitted.

Actually, being able to build projects much easier from GitHub is the sole reason why I'm currently using Arch as my main OS. Building a project is just a shell script with a couple of defined functions. Quite literally. I really admire NixOS's philosophy of pushing the boundaries as a distro where everything, including configurations and modifications, can be done in a reproducible manner. They're basically trying t…

This is slightly different thing. Nix and NixOS are trying to solve multiple things, and that's what it might be a bit confusing. Many people don't realize that, but if you get for example mentioned project from github and I do and we compile it on our machines we get a different file (it'll work the same but it won't be exactly the same). Say we use the same dependencies, we still will get a different files, because…

A rich source of non-reproducibility is non-determinism introduced by parallel building.

Preserving parallel execution, but arriving at deterministic outputs, is an interesting and ongoing challenge. With a rich mathematical structure, too.

Re: Nixos-unstable’s ISO_minimal.x86_64-Linux is 100% reproducible

#173

Post by eru »

zamadatix wrote: post #38
jnxx wrote: post #32

Earlier quoted context omitted.

No, it is not always the maintainer. Imagine you download a binary software package via HTTPS. In theory, the integrity of the download is protected by the server certificate. However, it is possible that certificates get hacked, get stolen, or that nation states force CAs to give out back doors. In that case, your download could have been changed on the fly with arbitrary alterations. Reproducible builds make it pos…

Same as when you download the source instead of the binary and see it reproducibly builds the backdoored binary. And at this point we're back to "Build from source. This will always be a deeply niche thing to do. It's slow, inconvenient, and inaccessible except to nerds." anyways. It's not that reproducible builds provide 0 value it's that they don't truly solve the trust problem as initially stated. They also have n…

Reproducible builds still help a lot with security. For example, they let you shift build latency around.

Eg suppose you have a software package X, available both as a binary and in source.

With reproducible builds, you can start distributing the binary to your fleet of computers, while at the same time you are kicking off the build process yourself.

If the result of your own build is the same as the binary you got, you can give the command to start using it. (Otherwise, you quarantine the downloaded binary, and ring some alarm bells.)

Similarly, you can re-build some random sample of packages locally, just to double-check, and report.

If most debian users were to do something like that, any tempering with the debian repositories would be quickly detected.

(Having a few malicious users wouldn't hurt this strategy much, they can only insert noise in the system, but not give you false answers that you trust.)

Re: Nixos-unstable’s ISO_minimal.x86_64-Linux is 100% reproducible

#174

Post by eru »

IgorPartola wrote: post #136
pabs3 wrote: post #122

Earlier quoted context omitted.

The crev folks are working on a co-ordination system for incremental distributed code review: https://github.com/crev-dev/

I actually wonder if it’s possible to write code at such a macro level as to obfuscate, say, a keylogger in a huge codebase such that reviewing just a single module/unit would not reveal that something bad is going on.

Depends on how complicated the project itself is. A simple structure with the bare minimum of side-effects (think, functional programming) would make this effort harder.

For something like C, all bets are off: http://www.underhanded-c.org/ or https://en.wikipedia.org/wiki/Underhanded_C_Contest

Re: Nixos-unstable’s ISO_minimal.x86_64-Linux is 100% reproducible

#175

Post by eru »

taviso wrote: post #170
franga2000 wrote: post #166

Earlier quoted context omitted.

> Reproducibility is technically pointless, because you still have to trust the developer, and they can still add backdoors. Builder != developer - and with reproducible builds, you no longer beed to trust the builder. CI is commonly used for the final distributable builds and you can't always trust the CI server. Even if you do, many rely on third party thingd like docker images - if the base build image gets compro…

> As a developer, it would be quite reassuring to build my binary (which I already do for testing) and compare the hash with the one from the CI server to confirm nothing has been tampered with. That makes sense! However, this is not a good argument for reproducible builds, because you can already do that today. You already have to build a trusted binary locally for testing right? You're dreaming of being able to com…

With the reproducible build, you can start using the untrusted binary while you are still building your trusted one.

You can also have ten people on the internet verify the untrusted binary. With signatures, adding more people doesn't help.

Re: Nixos-unstable’s ISO_minimal.x86_64-Linux is 100% reproducible

#176

Post by taviso »

eru wrote: post #175
taviso wrote: post #170

Earlier quoted context omitted.

> As a developer, it would be quite reassuring to build my binary (which I already do for testing) and compare the hash with the one from the CI server to confirm nothing has been tampered with. That makes sense! However, this is not a good argument for reproducible builds, because you can already do that today. You already have to build a trusted binary locally for testing right? You're dreaming of being able to com…

With the reproducible build, you can start using the untrusted binary while you are still building your trusted one. You can also have ten people on the internet verify the untrusted binary. With signatures, adding more people doesn't help.

> With the reproducible build, you can start using the untrusted binary while you are still building your trusted one.

That's not how it works, you have to reproduce it before it becomes trusted.

> You can also have ten people on the internet verify the untrusted binary.

Sure, then we have to build a complex consensus system that introduces a bunch of unsolved problems. My opinion is that this just isn't worth it, there is practically nothing to gain and it's really really hard.

Re: Nixos-unstable’s ISO_minimal.x86_64-Linux is 100% reproducible

#177

Post by siraben »

delroth wrote: post #29
jnxx wrote: post #23

A good sign that the friendly competition by Guix has a positive influence :) https://guix.gnu.org/manual/en/html_node/Bootstrapping.html https://guix.gnu.org/en/blog/2020/guix-further-reduces-boots...

This smaller bootstrap seed thing is a different problem from reproducible builds. nixpkgs does still have a pretty big initial TCB (aka. stage0) compared to Guix. But as far as I can tell NixOS has the upper hand in terms of how much can be built reproducibly (aka. the output hash matches across separate builds).

There's an issue for this[0]. Currently Nixpkgs relies on a 130 MB (!) uncompressed tarball, which is pretty big compared to Guix. It would be amazing to get it down to something like less than 1 KB with live-bootstrap.

Also, due to the way Nixpkgs is architectured, it also lets us experiment with more unusual ideas like a uutils-based stdenv[1] instead of GNU coreutils.

[0] https://github.com/NixOS/nixpkgs/issues/123095

[1] https://github.com/NixOS/nixpkgs/pull/116274

Re: Nixos-unstable’s ISO_minimal.x86_64-Linux is 100% reproducible

#178

Post by Foxboron »

catern wrote: post #149
esjeon wrote: post #96

Earlier quoted context omitted.

> I'm grateful to the nixos team for being beating a trail thru the jungle here. Retrofitting reproducibility onto a big software project that grew without it, is hard work. Actually, it's Debian guys who pushed reproducible build hard in the early days. They upstreamed necessary changes and also spread the concept itself. This is a two-decade long community effort. In turn, NixOS is mostly just wrapping those projec…

That's somewhat uncharitable. patchelf, for example, is one tool developed by NixOS which is widely used for reproducible build efforts. (although I don't know concretely if Debian uses it today)

patchelf is not really widely used for solving reproducible builds issues. It's made for rewriting RPATHs which is essential for NixOS, but not something you would be seeing in other distributions except for when someone need to work around poor upstream decisions.

Re: Nixos-unstable’s ISO_minimal.x86_64-Linux is 100% reproducible

#179

Post by stavros »

kzrdude wrote: post #169
raziel2p wrote: post #163

Earlier quoted context omitted.

The ratio of reproducible to non-reproducible packages doesn't seem to have changed that much in the last 5 years.

They have new challenges with new packages. In the last 5 years there entered a lot of rust packages for example, a new compiler to tackle reproducibility with (and not trivial, even if upstream has worked on it a lot).

In my experience, rustc builds are reproducible if you build on the same path. They come out byte for byte identical.

Re: Nixos-unstable’s ISO_minimal.x86_64-Linux is 100% reproducible

#180

Post by microtonal »

taviso wrote: post #77

I don't see a single comment doubting the value of reproducibility, so I'll be the resident skeptic :) I think build reproducibility is a cargo cult. The website says reproducibility can reduce the risk of developers being threatened or bribed to backdoor their software, but that is just ridiculous. Developers have a perfect method for making their own software malicious: bugdoors. A bugdoor (bug + backdoor) is a del…

I think build reproducibility is a cargo cult.

Most people here are debating you on the security angle, but in the case of Nix (and Guix) there is another important angle - reproducible builds make a content-addressed store possible.

In Nix, the store is traditionally addressed by the hash of the derivation (the recipe that builds the package). For example, lr96h... in the path

    /nix/store/lr96h3dlny8aiba9p3rmxcxfda0ijj08-coreutils-8.32
is the hash of the (normalized) derivation that was used to build coreutils. Since the derivation includes build inputs, either changing the derivation for coreutils itself or one of its inputs (dependencies) results in a different hash and a rebuild of coreutils.

This also means that if somebody changes the derivation of coreutils every package that depends on coreutils will be rebuilt, even if this change does not result in a different output path (compiled package).

This is being addressed by the new work on the content-addressed Nix store (although content-addressing aws already discussed in Eelco Dolstra's PhD thesis about Nix). In the content-addressed store, the hash in the path, such as the on above is a hash of the output path (the built package), rather than a hash of the normalized derivation. This means that if the derivation of coreutils is changed in such a way that it does not change the output path, none of the packages that depend on coreutils are rebuilt.

However, this only works reliably with reproducible builds, because if there is non-determinism in the build, how do you know whether a change in the output path is changed as a result of changing a derivation or as a result of uninteresting non-determinisms (the output hash would change in both cases).

Post reply on HN