Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lots of progress for Debian's reproducible builds

lwn.net

Post reply on HN
Print view View original

31–35 of 35 posts

Re: Lots of progress for Debian's reproducible builds

#31

Post by agwa »

jzwinck wrote: post #20
agwa wrote: post #16

Earlier quoted context omitted.

The biggest offender is timestamps - compiled binaries, documentation, archives, etc. often contain the time at which the file was built. Other problems include non-deterministic filesystem order, randomized hash algorithms, and even the fact that Markdown processors mangle email addresses randomly. A highly vexing problem that I'm currently trying to solve is that libxslt implements the XSLT generate-id() function b…

Maybe you could fix the libxslt problem by using a pool allocator for the nodes. The pool would contain only the nodes, and you could use the offset in the pool as their ID, rather than the full virtual address. Just some food for thought.

That's a very interesting idea. It's possible to tell libxml2 to use a custom allocator, but unfortunately it will use that allocator for all objects, not just nodes, and thus would probably be affected by hash table randomization. The architecture dependence is also a problem.

My current solution is a patch that uses a hash table to map the memory address of a node to a counter that increments in a deterministic order. It's a massive hack and I hate it. I'm currently trying to assess what its performance impact is.

The proper solution would be for libxml2 to maintain a counter and assign every node a deterministic ID, but that would require extending the _xmlNode struct, which would break ABI compatibility because the struct is not opaque.

Re: Lots of progress for Debian's reproducible builds

#32

Post by VLM »

Alupis wrote: post #14

Can anyone comment on why all builds are not currently "reproducible"? I mean, if a package is compiled on the same system, with the same compiler, with the same build script -- should it not produce the same output?

From memory windows binary format supposedly contains a compilation timestamp, but I've never developed for windows. And I know ELF doesn't contain a timestamp.

You can blame the compiler random number seed for much non-determinism. I googled for this and found at good explanation at:

http://blog.mindfab.net/2013/12/on-way-to-deterministic-bina...

The summary is anonymous namespaces actually have a name, that being a big integer, and they're all distinct so each has a different one and gcc just picks a psuedorandom number for each anon namespace. They're anonymous to you, but not to the compiler LOL crazy but true. Some folklore that once every 50 trillion years (maybe less often) linking two compiled files will fail at runtime not compile time because two namespaces randomly were assigned the same number so they "crossed the streams".

Supposedly everyone knows some versions of GCC sometimes pick specific optimizations pseudorandomly. This is one of those "everyone knows and nobody has evidence" things. I'd welcome a link to actual evidence, like to actual code that sometimes compiles differently based on phase of moon or whatever. If I were more bored I'd make a github project specifically to manipulate gcc for fun, sounds amusing.

If you're compiling up static libraries, AR is just a generic, although crude and ancient, file archiver and therefore contains timestamps as explained in the link below. I don't think that's necessary for static library operation its just an artifact of the file format.

http://en.wikipedia.org/wiki/Ar_%28Unix%29

There is also a funny failure mode where two systems with somewhat different libraries (perhaps one upgraded after the other, or a security patch, or got owned), when compiling a statically linked binary with the same code and compiler will obviously have different static linked result. This is of the same class as optimizations getting a little too personal for your specific CPU family such that two generic amd64 intel boxes are not quite so identical as you'd think in both optimizations and runtimes. (oh edited to add what if your virtualization is funky such that GCC didn't think a certain instruction set was present in the virtual image, so bare metal compiles would have different optimizations than images running on the same bare metal, although this is totally theoretical and probably doesn't exist in reality)

Re: Lots of progress for Debian's reproducible builds

#33

Post by chubot »

csirac2 wrote: post #30
chubot wrote: post #22

Earlier quoted context omitted.

Unfortunately I haven't published it yet, but I can describe what I did. I have like 3000 lines of shell script to make containers, and maybe 800-1000 lines are related to debootstrap. I have a cron job running daily on multiple machines, doing a deterministic debootstrap of Debian Wheezy on i386 and amd64. It basically wraps debootstrap, strips down the image a bit, and stamps out the nondeterminism. Every day it ha…

Thanks for sharing, that sounds great. Do you have a GitHub/bit bucket/etc. account I could follow? I've found myself automating Debian/kernel/boot loader/libs/app stack packages for some ARM hardware for which I'm an app developer. This is due to the outrageous situation of COM module vendors (and/or SoC manufacturers) thinking that kernel 3.0 and horribly ancient/unsupported/unpatched userlands are acceptable in 20…

I didn't have anything related to this on my Github account, but I just uploaded the data files in case that is useful to anyone.

https://github.com/andychu/debian-wheezy-metadata

(NOTE: The repo takes up 300+ MB on my local disk)

I chatted on #debian IRC about this a little. It seemed like one person thought doing all the archs would take up too much space. So far doing it daily for over a year for two archs has been manageable. I think you simply have one repo per arch, instead of having multiple archs in one repo like I have here.

I will try to get the code up... if you think you will use it you can ping me on github. I think this should be in Debian itself, and actually supported by debootstrap, but so far I am just wrapping it rather than patching it. (I started off trying to patch it.)

Re: Lots of progress for Debian's reproducible builds

#34

Post by [deleted] »

SwellJoe wrote: post #28
jml7c5 wrote: post #17

Will this provide a guaranteed method for reproducible builds, or will it still be technically possible to create build scripts that produce different results (e.g., by pulling from /dev/random, or grabbing timing information from various sources, or by writing a multithreaded program whose threads all write to a single file)?

How could anyone or anything prevent you from building something nonreproducible? This is about making build processes that are intentionally reproducible... You seem to be asking if one could continue to do things as they currently are done, which given that the entire system is open source, of course you can. This can't possibly force all users to only make repeatable builds. This seems like such an odd question th…

[deleted]

Re: Lots of progress for Debian's reproducible builds

#35

Post by csirac2 »

stsp wrote: post #24
csirac2 wrote: post #4

It can be surprisingly difficult. Funnily enough moving from svn git in one project I know of probably did a lot of the necessary work to achieve this, by having to remove reliance on $SVN tags and pre/post-"build commits" which used to be a part of the release process. It's an interesting use-case for Docker as well: you can ship the build environment (or its Dockerfile describing it) for people to run builds under…

Can you elaborate on what these pre/post-"build commits" did and why they were needed? If these commits were used to adjust version numbers in source files, the following trick should eliminate them: Check out the release branch into a working copy, adjust the version numbers in the working copy, and then copy the working copy to the tag's URL (as in: cd working-copy; svn copy . ^/tags/1.0) If this was a maven projec…

It was a bespoke build system but this aspect of it was similarly "wrong". IIRC pre-build commit, yes, was version-number related. It also gathered issues fixed in the release being built and updated change logs/release notes/upgrade info documentation automatically. IIRC the post-build commit helped confirm in the commit history that a particular build for release x.y.z was successful and the version number can now be incremented (occasionally it took multiple attempts for the release manager to build successfully). Of course, one could go through the tags but most of the developers liked seeing release management stuff in trunk/release branches.

I didn't mean to criticize SVN as being inherently incapable of reproducible builds (if anything that's harder to achieve with git, especially with hacks like I've listed above), but the act of cleaning up the SVN repos and preparing for migration to git where a lot of our old SVN (and RCS!) habits would be problematic, also seem like the same kind of housecleaning you'd need to prepare for reproducible builds.

Post reply on HN