Live data from Hacker News

Reproducible C++ builds by logging Git hashes

jgarby.uk

11–20 of 45 posts

Re: Reproducible C++ builds by logging Git hashes

#11
That barely scratches the surface when it comes to reproducible c and c++ builds. In fact the topic of reproducible builds assumes your sources are the same, as in that's really not the problem here.

You need to control every single library header version you are using outside your source like stdlibs, os headers, third party, and have a strategy to deal with rand/datetime variables that can be part of the binary.

Re: Reproducible C++ builds by logging Git hashes

#12
post #5

Give Nix a look sometime, it takes this to a whole new level by including all of the build dependencies in the hash, and their build dependencies and so on. The standard flake workflow even includes the warning about having uncommitted files.

It's quite odd to me that Nix or something similar like Mise isn't completely ubiquitous in software. I feel like I went from having issues with build dependencies to having that aspect of software development completely solved as soon as I adopted Nix. I absolutely can't imagine not using some kind of tool like this. Feels as vital as VCS to me now.

Agreed. Recently started a new gig and set up Mise (previously had used nix for this) in our primary repos so that we can all share dependencies, scripts, etc. The new monorepo mode is great. Basically no one has complained and it's made everyone's lives a lot easier. Can't imagine working any other way — having the same tools everywhere is really great.

I'll also say I have absolutely 0 regrets about moving from Nix to Mise. All the common tools we want are available, it's especially easy to install tools from pip or npm and have the environments automanaged. The docs are infinity times better. And the speed of install and shell sourcing is, you guessed it, much better. Initial setup and install is also fantastically easier. I understand the ideology behind Nix, and if I were working on projects where some of our tools weren't pre-packageable or had weird conflicting runtime lib problems I'd get it, but basically everything these days has prebuilt static binaries available.

Re: Reproducible C++ builds by logging Git hashes

#13
post #5

Earlier quoted context omitted.

It's quite odd to me that Nix or something similar like Mise isn't completely ubiquitous in software. I feel like I went from having issues with build dependencies to having that aspect of software development completely solved as soon as I adopted Nix. I absolutely can't imagine not using some kind of tool like this. Feels as vital as VCS to me now.

Agreed. Recently started a new gig and set up Mise (previously had used nix for this) in our primary repos so that we can all share dependencies, scripts, etc. The new monorepo mode is great. Basically no one has complained and it's made everyone's lives a lot easier. Can't imagine working any other way — having the same tools everywhere is really great. I'll also say I have absolutely 0 regrets about moving from Nix…

Mise is pretty nice, I'd recommend it over all the other gazillion version-manager things out there, but it's not without its own weak spots: I tried mise for a php project, neither of the backends available for php had a binary for macos, and both of them failed to build it. I now use a flake.nix, along with direnv and `use flake`. The nix language definitely makes for some baffling boilerplate around the dependencies list, but devs unfamiliar with nix can ignore it and just paste in the package name from nixpkgs search.

There's also jbadeau/mise-nix that lets you use flakes in mise, but I figured at that point I may as well just use flake.nix.

Re: Reproducible C++ builds by logging Git hashes

#14
post #5

Earlier quoted context omitted.

It's quite odd to me that Nix or something similar like Mise isn't completely ubiquitous in software. I feel like I went from having issues with build dependencies to having that aspect of software development completely solved as soon as I adopted Nix. I absolutely can't imagine not using some kind of tool like this. Feels as vital as VCS to me now.

We'd have been a lot further along if tools like make had ever adopted hashes for freshness checking rather than timestamps. We'd have ccache built in to make, make could hash entire targets, and now we're halfway to derivations. Of course that's handwaving over the tricky problem of making sure targets build reproducibly, but perhaps compiler toolchains would have taken more care to ensure it.

Timestamps have the property of being easily comparable; you can always tell if one file is older then the other. If you were to use hashes for the same purpose, you'd have to keep a database of expected hashes, and comparing them would be a less trivial task, etc. It's doable, but it would be a very differently designed (and much more computationally expensive) program then make.

Re: Reproducible C++ builds by logging Git hashes

#15
post #4

A simpler way to do this, especially if you do tagging in your repositories, is to use `git describe`. For example: $ git describe --dirty v1.4.1-1-gde18fe90-dirty The format is - -g - . If the repo isn't dirty, then the hash you get excludes that part: $ git describe --dirty v1.4.1-1-gde18fe90 If you're using lightweight tags (the default) and not annotated tags (with messages and signatures and etc) you may want to…

The one caveat to this is that you must perform a sufficiently-deep clone that you can actually reach the tag.

Re: Reproducible C++ builds by logging Git hashes

#16
post #5

Give Nix a look sometime, it takes this to a whole new level by including all of the build dependencies in the hash, and their build dependencies and so on. The standard flake workflow even includes the warning about having uncommitted files.

It's quite odd to me that Nix or something similar like Mise isn't completely ubiquitous in software. I feel like I went from having issues with build dependencies to having that aspect of software development completely solved as soon as I adopted Nix. I absolutely can't imagine not using some kind of tool like this. Feels as vital as VCS to me now.

I think bazel is the tool lot of people are converging towards, but turns out that maintaining complex build setups is a lot of work.

Re: Reproducible C++ builds by logging Git hashes

#17
post #5

Earlier quoted context omitted.

It's quite odd to me that Nix or something similar like Mise isn't completely ubiquitous in software. I feel like I went from having issues with build dependencies to having that aspect of software development completely solved as soon as I adopted Nix. I absolutely can't imagine not using some kind of tool like this. Feels as vital as VCS to me now.

We'd have been a lot further along if tools like make had ever adopted hashes for freshness checking rather than timestamps. We'd have ccache built in to make, make could hash entire targets, and now we're halfway to derivations. Of course that's handwaving over the tricky problem of making sure targets build reproducibly, but perhaps compiler toolchains would have taken more care to ensure it.

I'd say the sad part is that nix really works well when the toolchain does caching transparently. But to deliver good DX outside of nix, you kind of want great porcelain tooling that handles everything behind the scenes - downloading of libraries, building said libraries, linking everything together. Sometimes people choose to just embed a whole programming language to make their build system work e.g. gradle. Cargo just does everything. Nix then can't really granularly build everything piece by piece when building rust crates with Cargo - you just get to rebuild every dependency any time the derivation is built and any one input changed. I wonder how much less time would've been wasted if newer languages chose to build on top of nix. Of course, nix would need to become slightly more compatible with Windows and other OSes for this to be practical.

Re: Reproducible C++ builds by logging Git hashes

#18
post #11

That barely scratches the surface when it comes to reproducible c and c++ builds. In fact the topic of reproducible builds assumes your sources are the same, as in that's really not the problem here. You need to control every single library header version you are using outside your source like stdlibs, os headers, third party, and have a strategy to deal with rand/datetime variables that can be part of the binary.

How would you even start solving these?

Re: Reproducible C++ builds by logging Git hashes

#19
post #11

That barely scratches the surface when it comes to reproducible c and c++ builds. In fact the topic of reproducible builds assumes your sources are the same, as in that's really not the problem here. You need to control every single library header version you are using outside your source like stdlibs, os headers, third party, and have a strategy to deal with rand/datetime variables that can be part of the binary.

How would you even start solving these?

Virtualization, imho. Every build gets its own virtual machine, and once the build is released to the public, the VM gets cloned for continued development and the released VM gets archived.

I do this git tags thing with my projects - it helps immensely if the end user can hover over the company logo and get a tooltip with the current version, git tag and hash, and any other relevant information to the build.

Then, if I need to triage something specific, I un-archive the virtualized build environment, and everything that was there in the original build is still there.

This is a very handy method for keeping large code bases under control, and has been very effective over the years in going back to triage new bugs found, fixing them, and so on.

Re: Reproducible C++ builds by logging Git hashes

#20
post #11

That barely scratches the surface when it comes to reproducible c and c++ builds. In fact the topic of reproducible builds assumes your sources are the same, as in that's really not the problem here. You need to control every single library header version you are using outside your source like stdlibs, os headers, third party, and have a strategy to deal with rand/datetime variables that can be part of the binary.

Also the compiler/linker used to build it.
Post reply on HN