Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reproducible C++ builds by logging Git hashes

jgarby.uk

Post reply on HN
Print view View original

21–30 of 45 posts

Re: Reproducible C++ builds by logging Git hashes

#21

Post by peterldowns »

chuckadams wrote: post #13
peterldowns wrote: post #12

Earlier quoted context omitted.

Agreed. Recently started a new gig and set up Mise (previously had used nix for this) in our primary repos so that we can all share dependencies, scripts, etc. The new monorepo mode is great. Basically no one has complained and it's made everyone's lives a lot easier. Can't imagine working any other way — having the same tools everywhere is really great. I'll also say I have absolutely 0 regrets about moving from Nix…

Mise is pretty nice, I'd recommend it over all the other gazillion version-manager things out there, but it's not without its own weak spots: I tried mise for a php project, neither of the backends available for php had a binary for macos, and both of them failed to build it. I now use a flake.nix, along with direnv and `use flake`. The nix language definitely makes for some baffling boilerplate around the dependenci…

The beauty of mise is that as long as someone is hosting a precompiled binary for you, it's easy to get it. I just repro'd and yeah, `mise use php` fails for me on my machine because I don't have any dev headers. But looks like there's an easy workaround using the `ubi` downloader:

https://github.com/jdx/mise/discussions/4720#discussioncomme...

or see the first comment on this thread to see a way to explicitly specify where to find the binaries for each platform:

https://github.com/jdx/mise/discussions/4720#discussioncomme...

Having these kind of "eject" options is one of the reasons I really appreciate Mise. Not sure this would work for you but I'd rather be able to do this than have to manage/support everyone on my dev team installing and maintaining Nix.

Re: Reproducible C++ builds by logging Git hashes

#22

Post by matrss »

WalterBright wrote: post #20
halayli wrote: post #11

That barely scratches the surface when it comes to reproducible c and c++ builds. In fact the topic of reproducible builds assumes your sources are the same, as in that's really not the problem here. You need to control every single library header version you are using outside your source like stdlibs, os headers, third party, and have a strategy to deal with rand/datetime variables that can be part of the binary.

Also the compiler/linker used to build it.

As well as the toolchain used to compile your toolchain, through multiple levels, and all compiler flags along the path, and so on, down to some "seed" from which everything is build.

Guix' full-source bootstrap is pretty enlightening on that topic: https://guix.gnu.org/manual/devel/en/html_node/Full_002dSour...

Re: Reproducible C++ builds by logging Git hashes

#23

Post by corysama »

MomsAVoxell wrote: post #19
YayaScript wrote: post #18

Earlier quoted context omitted.

How would you even start solving these?

Virtualization, imho. Every build gets its own virtual machine, and once the build is released to the public, the VM gets cloned for continued development and the released VM gets archived. I do this git tags thing with my projects - it helps immensely if the end user can hover over the company logo and get a tooltip with the current version, git tag and hash, and any other relevant information to the build. Then, if…

Back in the PS2 era of game development, we didn't have much of virtual machines to work with. And, making a shippable build involved wacky custom hardware that wouldn't work in a VM anyway. So, instead we had The Build Machine.

The Build Machine would be used to make The Gold Master Disc. A physical DVD that would be shipped to the publisher to be reproduced hopefully millions of times. Getting The Gold Master Disc to a shippable state would usually take weeks because it involved burning a custom disc format for each build and there was usually no way to debug other than watching what happened on the game screen.

When The Gold Master Disc was finally finalized, The Build Machine would be powered down, unplugged, labeled "This is the machine that made The Gold Master Disc for Game XYZ. DO NOT DISCARD. Do not power on without express permission from the CTO." and archived in the basement forever. Or, until the company shut down. Then, who knows what happens to it.

But, there was always a chance that the publisher or Sony would come back and request to make a change for 1.0.1 version because of some subtle issue that was found later. You don't want to take any chances starting the build process over on a different machine. You make the minimal changes possible on The Build Machine and you get The Gold Master Disc 1.0.1 out ASAP.

Re: Reproducible C++ builds by logging Git hashes

#24

Post by syncsynchalt »

YayaScript wrote: post #18
halayli wrote: post #11

That barely scratches the surface when it comes to reproducible c and c++ builds. In fact the topic of reproducible builds assumes your sources are the same, as in that's really not the problem here. You need to control every single library header version you are using outside your source like stdlibs, os headers, third party, and have a strategy to deal with rand/datetime variables that can be part of the binary.

How would you even start solving these?

Take a look at the decade+ long effort that Debian has put into this problem: https://wiki.debian.org/ReproducibleBuilds

Here's a talk from 2024: https://debconf24.debconf.org/talks/18-reproducible-builds-t...

Several distros are above the 90% mark of all packages being byte-for-byte reproducible, and one or two have hit the 99% mark.

Re: Reproducible C++ builds by logging Git hashes

#25

Post by chuckadams »

bigfishrunning wrote: post #14
chuckadams wrote: post #6

Earlier quoted context omitted.

We'd have been a lot further along if tools like make had ever adopted hashes for freshness checking rather than timestamps. We'd have ccache built in to make, make could hash entire targets, and now we're halfway to derivations. Of course that's handwaving over the tricky problem of making sure targets build reproducibly, but perhaps compiler toolchains would have taken more care to ensure it.

Timestamps have the property of being easily comparable; you can always tell if one file is older then the other. If you were to use hashes for the same purpose, you'd have to keep a database of expected hashes, and comparing them would be a less trivial task, etc. It's doable, but it would be a very differently designed (and much more computationally expensive) program then make.

I bet we could get pretty far with symlinks, but then again even those were an exotic feature on some of make's supported platforms. Nowadays, may as well use sqlite.

Re: Reproducible C++ builds by logging Git hashes

#26

Post by matrss »

YayaScript wrote: post #18
halayli wrote: post #11

That barely scratches the surface when it comes to reproducible c and c++ builds. In fact the topic of reproducible builds assumes your sources are the same, as in that's really not the problem here. You need to control every single library header version you are using outside your source like stdlibs, os headers, third party, and have a strategy to deal with rand/datetime variables that can be part of the binary.

How would you even start solving these?

A good package manager, e.g. GNU Guix, let's you define a reproducible environment of all of your dependencies. This accounts for all of those external headers and shared libraries, which will be made available in an isolated build environment that only contains them and nothing else.

Eliminating nondeterminism from your builds might require some thinking, there are a number of places this can creep in (timestamps, random numbers, nondeterministic execution, ...). A good package manager can at least give you tooling to validate that you have eliminated nondeterminism (e.g. `guix build --check ...`).

Once you control the entire environment and your build is reproducible in principal, you might still encounter some fun issues, like "time traps". Guix has a great blog post about some of these issues and how they mitigate them: https://guix.gnu.org/en/blog/2024/adventures-on-the-quest-fo...

Re: Reproducible C++ builds by logging Git hashes

#27

Post by hogehoge51 »

halayli wrote: post #11

That barely scratches the surface when it comes to reproducible c and c++ builds. In fact the topic of reproducible builds assumes your sources are the same, as in that's really not the problem here. You need to control every single library header version you are using outside your source like stdlibs, os headers, third party, and have a strategy to deal with rand/datetime variables that can be part of the binary.

You also need to capture the version of the toolchain etc etc. Should also have a traceable link to the version of your specifications.

Just use ClearCase/ClearMake, it's been doing all of this software configuration auditing stuff for you since the 1990s.

Re: Reproducible C++ builds by logging Git hashes

#28

Post by hogehoge51 »

YayaScript wrote: post #18
halayli wrote: post #11

That barely scratches the surface when it comes to reproducible c and c++ builds. In fact the topic of reproducible builds assumes your sources are the same, as in that's really not the problem here. You need to control every single library header version you are using outside your source like stdlibs, os headers, third party, and have a strategy to deal with rand/datetime variables that can be part of the binary.

How would you even start solving these?

AFAIK ClearMake intercepted file system access and recorded the version of everything touched during your build.

Re: Reproducible C++ builds by logging Git hashes

#29

Post by ignoramous »

syncsynchalt wrote: post #24
YayaScript wrote: post #18

Earlier quoted context omitted.

How would you even start solving these?

Take a look at the decade+ long effort that Debian has put into this problem: https://wiki.debian.org/ReproducibleBuilds Here's a talk from 2024: https://debconf24.debconf.org/talks/18-reproducible-builds-t... Several distros are above the 90% mark of all packages being byte-for-byte reproducible, and one or two have hit the 99% mark.

> Several distros are above the 90% mark of all packages being byte-for-byte reproducible, and one or two have hit the 99% mark.

Simply incredible.

Explains F-Droid's recent success with Reproducible Builds (as some F-Droid maintainers are also active in the Debian scene): https://f-droid.org/en/2025/05/21/making-reproducible-builds...

Re: Reproducible C++ builds by logging Git hashes

#30

Post by kazinator »

Git hashes have nothing whatsoever to do with whether you can do a clean build of the same tree twice with the same results, bit for bit.

Git hashes or tags can help identify what was built: the inputs.

You only need to know that for traceability: when you hold the released outputs, but do not hold (or are not sure you hold) the matching inputs.

If builds are reproducible, the traceability becomes more meaningful.

In the TXR project, have a ./configure option called --build-id. This sets an ID that is appended to the version, which is in the executable. It is nothing by default; not used. It is meant to be useful for people who interact with the code, so they can check what they are running (things can get confusing when you are going back and forth among versions, or making local changes).

If you set the build ID it to the word "git", then it is calculated using:

  git describe --tags --dirty
that's probably what this author should be using. It gives you a meaningful ID that is related to the most recent release tag, and whether the repo was dirty.

  $ git describe --tags --dirty
  txr-302-20-g77c99b74e-dirty
We are (sadly, only) 20 commits after 302, at a commit whose short hash is 77c99b74e, and the repo is in a modified state.

I have it rigged in the Makefile that it actually keeps track of the most recent build ID in a little .build_id file. If the build ID changes relative to what is in that file, the Makefile will force a rebuild of the .o files which incorporate the build ID.

Also, there is no need to be generating dynamic #include material just for this. A simple -Dsymbol=var option in the CFLAGS will define a preprocessor symbol:

  CFLAGS += -DMY_BUILD_ID=\"$(my_build_id)\"
Post reply on HN