Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You don’t need reproducible builds

blog.cmpxchg8b.com

Post reply on HN
Print view View original

171–179 of 179 posts

Re: You don’t need reproducible builds

#171

Post by rramdin »

pests wrote: post #139
R0b0t1 wrote: post #99

Earlier quoted context omitted.

The holy grail of reproducible builds is achieving the same binary via different compilers. This was, at least when I started looking into reproducible builds, why I wanted to do it and why others wanted to do it. The other benefits are kind of side benefits.

This can't be correct. What would be the point of different compilers be then? There's no way that every compiler would produce the same exact insrtuctions for each respected input. There would be no point to using an optimizing compiler or one with better intrinsic support.

Perhaps you're comparing a multithreaded version with a single threaded version, or a single-host build vs distributed build.

When adding new features to a compiler, you might want to verify that the old and new versions have the same output given the same input. If your compilers produced non-deterministic output, this exercise would not be possible.

Re: You don’t need reproducible builds

#172

Post by pests »

R0b0t1 wrote: post #166
pests wrote: post #139

Earlier quoted context omitted.

This can't be correct. What would be the point of different compilers be then? There's no way that every compiler would produce the same exact insrtuctions for each respected input. There would be no point to using an optimizing compiler or one with better intrinsic support.

Obtaining evidence a compiler probably isn't backdoored is the point. >There's no way that every compiler would produce the same exact insrtuctions for each respected input. You don't know this.

I do, because I can make a compiler that chooses non standard instructions.

Re: You don’t need reproducible builds

#173

Post by pests »

Willamin wrote: post #149
pests wrote: post #139

Earlier quoted context omitted.

This can't be correct. What would be the point of different compilers be then? There's no way that every compiler would produce the same exact insrtuctions for each respected input. There would be no point to using an optimizing compiler or one with better intrinsic support.

> What would be the point of different compilers be then? When given the same input and expecting the same output, there remains only one thing: compile speed.

Well, and memory usage too etc. But what's the point then? Why would Intel have their optimizing compiler? Why GCC vs llvm?

Re: You don’t need reproducible builds

#174

Post by pests »

rramdin wrote: post #171
pests wrote: post #139

Earlier quoted context omitted.

This can't be correct. What would be the point of different compilers be then? There's no way that every compiler would produce the same exact insrtuctions for each respected input. There would be no point to using an optimizing compiler or one with better intrinsic support.

Perhaps you're comparing a multithreaded version with a single threaded version, or a single-host build vs distributed build. When adding new features to a compiler, you might want to verify that the old and new versions have the same output given the same input. If your compilers produced non-deterministic output, this exercise would not be possible.

I took the statement "different compilers" to be completely different projects and codebases. Of course reproducable builds make sense in that situation. But why in the world would you ever expect gcc to 100% always match the output of llvm? That doesn't make sense.

Re: You don’t need reproducible builds

#175

Post by giantrobot »

nix23 wrote: post #168
giantrobot wrote: post #165

Earlier quoted context omitted.

By default you're not guaranteed the exact same output in two compiled binaries. There's a lot of variable bits[1] that make into binaries from C and C++. Different languages/compilers have different levels of variable bits. [1] https://blog.conan.io/2019/09/02/Deterministic-builds-with-C...

>Different languages/compilers have different levels of variable bits. No that's the point! Exact same environment same binary, that's reproducibility.

Yes that would be reproducibility iff the environment is identical. However "identical environment" is a complicated issue.

Differing file paths, timestamps, and host date/time can all easily make their way into a binary through macros in several languages without explicit compiler/linker flags. If compiled artifacts are bundled into a container (like a jar file) their metadata need to deterministically set or else the container as an artifact won't be deterministic.

So yes doing all the work to make build deterministic enables reproducibility but it's not free or automatic. Then doing the work to ensure the build environment is deterministic is an additional task that's not free or automatic.

Re: You don’t need reproducible builds

#176

Post by nix23 »

giantrobot wrote: post #175
nix23 wrote: post #168

Earlier quoted context omitted.

>Different languages/compilers have different levels of variable bits. No that's the point! Exact same environment same binary, that's reproducibility.

Yes that would be reproducibility iff the environment is identical. However "identical environment" is a complicated issue. Differing file paths, timestamps, and host date/time can all easily make their way into a binary through macros in several languages without explicit compiler/linker flags. If compiled artifacts are bundled into a container (like a jar file) their metadata need to deterministically set or else t…

>make build deterministic enables reproducibility but it's not free or automatic

No one said that, Debian had a really hard time:

https://wiki.debian.org/ReproducibleBuilds/About

Re: You don’t need reproducible builds

#177

Post by deknos »

tavis assumes there's an complete organization which is either good or bad. but what about people with in the organization with restricted or accountable access to sources or binaries?

i think reproducible builds are about malicious parties in an bigger organization.

Re: You don’t need reproducible builds

#178

Post by R0b0t1 »

pests wrote: post #172
R0b0t1 wrote: post #166

Earlier quoted context omitted.

Obtaining evidence a compiler probably isn't backdoored is the point. >There's no way that every compiler would produce the same exact insrtuctions for each respected input. You don't know this.

I do, because I can make a compiler that chooses non standard instructions.

Then nobody will use it for reproducible builds. What is your point? Performant compilers for an arch are likely to produce similar code.

Re: You don’t need reproducible builds

#179

Post by GauntletWizard »

zeroimpl wrote: post #154
GauntletWizard wrote: post #106

Earlier quoted context omitted.

You're semantically right, but also missing the point. The expense of getting to deterministic builds is large - You have to take great care in your build infrastructure and scripts. The benefits are also large, and worth it. Once you've gotten to deterministic builds, the expense of getting to reproducible builds is small; Typically days worth of work as opposed to months. The benefits are very different, but far fr…

It really depends. If I’m building a Java project, I’m pretty sure I’ve got a deterministic build just by running javac pointed at a source directory. If I want a reproducible build, I probably need to do a lot more: - ensure timestamps of all files embedded in jar files is consistent - ensure there is no BuildTime/BuildHost/BuildNumber variable of any kind being captured - ensure the exact version of compiler is doc…

There's an interesting pattern I've found in this: Don't ensure that there's no BuildTime/BuildHost/BuildNumber embedded. Ensure that all variables that are part of the build are captured and embedded. That is - It's okay for your build to include the Build Time, but that's an assertion at build time. Include it as a build output. Binaries should include all of the mutable environment used to build them as an embed. As in, their --version output should include them.

    # bazel version
    Build label: 2.0.0
    Build target: bazel-out/darwin- opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
    Build time: Thu Dec 19 12:33:30 2019 (1576758810)
    Build timestamp: 1576758810
    Build timestamp as int: 1576758810
Post reply on HN