Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lots of progress for Debian's reproducible builds

lwn.net

Post reply on HN
Print view View original

21–30 of 35 posts

Re: Lots of progress for Debian's reproducible builds

#21

Post by chubot »

xai3luGi wrote: post #12
chubot wrote: post #5

I actually did some work making debootstrap reproducible. So even if the 100 or so .deb builds it depends on are reproducible, then the chroot image resulting from debootstrap will not be reproducible byte-for-byte, due to the debootstrap shell script itself and the tools it calls. Offhand, I remember that /etc/{passwd,group} are copied from the host machine by design. There is also a random seed file, to save entrop…

Debian appears to be doing some work on that too: https://wiki.debian.org/ReproducibleInstalls

Interesting, didn't know that. They mentioned logs like bootstrap.log and dpkg.log, which I noticed, but looking at my shell scripts now there is also nondeterminism/host influence in:

  etc/resolv.conf
  var/cache/ldconfig/aux-cache
  var/lib/urandom/random-seed
  etc/init.d/.depend.{stop,start,boot}
  etc/shadow
  etc/passwd- and family (with trailing hyphen)
  etc/apt/trustdb.gpg and other keys
(not exhaustive)

Re: Lots of progress for Debian's reproducible builds

#22

Post by chubot »

csirac2 wrote: post #11
chubot wrote: post #5

I actually did some work making debootstrap reproducible. So even if the 100 or so .deb builds it depends on are reproducible, then the chroot image resulting from debootstrap will not be reproducible byte-for-byte, due to the debootstrap shell script itself and the tools it calls. Offhand, I remember that /etc/{passwd,group} are copied from the host machine by design. There is also a random seed file, to save entrop…

I'm interested! I actually have had to work around the /etc/{passwd,group} shenanigans for other reasons, interested to see what you did there.

Unfortunately I haven't published it yet, but I can describe what I did. I have like 3000 lines of shell script to make containers, and maybe 800-1000 lines are related to debootstrap.

I have a cron job running daily on multiple machines, doing a deterministic debootstrap of Debian Wheezy on i386 and amd64.

It basically wraps debootstrap, strips down the image a bit, and stamps out the nondeterminism. Every day it has gives a checksum for i386 and one for amd64, which holds across multiple host machines (one is Debian Wheezy, the other is Ubuntu Trusty). So it is free from host influence.

The /etc/apt/sources.list just has the "wheezy" repo now (i.e. not wheezy-updates). So Debian 7.7 had one pair of checksums, and then on Jan 10 2015, I noticed Debian 7.8 was released. They changed that day, and have been stable/reproducible every day since.

Part of this is also mirroring the Release/Packages metadata daily and storing version history in Git. One nice thing I found out about Debian through doing this is that the Release file completely describes the input, since it's hashes all the way down (a "Merkle tree", basically like Git itself.) My scripts also make it so you can store versioned metadata in one tree, while keeping data immutable in "pool" (all this really requires is symlinks and file:// URLs for the repo).

What project were you working with in this area? I'm basically doing this to make reproducible builds of containers. I'm kind of surprised that Docker completely punts on this problem.

Re: Lots of progress for Debian's reproducible builds

#24

Post by stsp »

csirac2 wrote: post #4

It can be surprisingly difficult. Funnily enough moving from svn git in one project I know of probably did a lot of the necessary work to achieve this, by having to remove reliance on $SVN tags and pre/post-"build commits" which used to be a part of the release process. It's an interesting use-case for Docker as well: you can ship the build environment (or its Dockerfile describing it) for people to run builds under…

Can you elaborate on what these pre/post-"build commits" did and why they were needed?

If these commits were used to adjust version numbers in source files, the following trick should eliminate them: Check out the release branch into a working copy, adjust the version numbers in the working copy, and then copy the working copy to the tag's URL (as in: cd working-copy; svn copy . ^/tags/1.0)

If this was a maven project: The maven release plugin is doing this wrong and performs 3 commits to an SVN repository per release...

Re: Lots of progress for Debian's reproducible builds

#25

Post by aplanas »

I love this kind of projects, and I think that for Debian is one of the best things that can happens.

Also openSUSE have reproducible builds/packages since ages via OBS (http://build.opensuse.org) and now Factory/Tumbleweed have reproducible packages + automatic CI (using openQA: https://openqa.opensuse.org) Quite an achievement for a rolling distribution.

Re: Lots of progress for Debian's reproducible builds

#26

Post by csmeu »

jzwinck wrote: post #20
agwa wrote: post #16

Earlier quoted context omitted.

The biggest offender is timestamps - compiled binaries, documentation, archives, etc. often contain the time at which the file was built. Other problems include non-deterministic filesystem order, randomized hash algorithms, and even the fact that Markdown processors mangle email addresses randomly. A highly vexing problem that I'm currently trying to solve is that libxslt implements the XSLT generate-id() function b…

Maybe you could fix the libxslt problem by using a pool allocator for the nodes. The pool would contain only the nodes, and you could use the offset in the pool as their ID, rather than the full virtual address. Just some food for thought.

That probably isn't architecture independent.

Re: Lots of progress for Debian's reproducible builds

#27

Post by dochtman »

agwa wrote: post #16
Alupis wrote: post #14

Can anyone comment on why all builds are not currently "reproducible"? I mean, if a package is compiled on the same system, with the same compiler, with the same build script -- should it not produce the same output?

The biggest offender is timestamps - compiled binaries, documentation, archives, etc. often contain the time at which the file was built. Other problems include non-deterministic filesystem order, randomized hash algorithms, and even the fact that Markdown processors mangle email addresses randomly. A highly vexing problem that I'm currently trying to solve is that libxslt implements the XSLT generate-id() function b…

Thanks for your work!

I've been wondering -- why do compiled binaries actually include timestamps? For human-readable stuff, I sort of understand, but for binaries I don't see we'd want to include timestamps.

Re: Lots of progress for Debian's reproducible builds

#28

Post by SwellJoe »

jml7c5 wrote: post #17

Will this provide a guaranteed method for reproducible builds, or will it still be technically possible to create build scripts that produce different results (e.g., by pulling from /dev/random, or grabbing timing information from various sources, or by writing a multithreaded program whose threads all write to a single file)?

How could anyone or anything prevent you from building something nonreproducible? This is about making build processes that are intentionally reproducible... You seem to be asking if one could continue to do things as they currently are done, which given that the entire system is open source, of course you can. This can't possibly force all users to only make repeatable builds. This seems like such an odd question that I think I must be misunderstanding you.

Re: Lots of progress for Debian's reproducible builds

#29

Post by revelation »

dochtman wrote: post #27
agwa wrote: post #16

Earlier quoted context omitted.

The biggest offender is timestamps - compiled binaries, documentation, archives, etc. often contain the time at which the file was built. Other problems include non-deterministic filesystem order, randomized hash algorithms, and even the fact that Markdown processors mangle email addresses randomly. A highly vexing problem that I'm currently trying to solve is that libxslt implements the XSLT generate-id() function b…

Thanks for your work! I've been wondering -- why do compiled binaries actually include timestamps? For human-readable stuff, I sort of understand, but for binaries I don't see we'd want to include timestamps.

I guess it's related to debugging purposes. If you know when a binary was built, you know that a source file X with a changetime later than the build is different to the source file used for building the binary.

Re: Lots of progress for Debian's reproducible builds

#30

Post by csirac2 »

chubot wrote: post #22
csirac2 wrote: post #11

Earlier quoted context omitted.

I'm interested! I actually have had to work around the /etc/{passwd,group} shenanigans for other reasons, interested to see what you did there.

Unfortunately I haven't published it yet, but I can describe what I did. I have like 3000 lines of shell script to make containers, and maybe 800-1000 lines are related to debootstrap. I have a cron job running daily on multiple machines, doing a deterministic debootstrap of Debian Wheezy on i386 and amd64. It basically wraps debootstrap, strips down the image a bit, and stamps out the nondeterminism. Every day it ha…

Thanks for sharing, that sounds great. Do you have a GitHub/bit bucket/etc. account I could follow?

I've found myself automating Debian/kernel/boot loader/libs/app stack packages for some ARM hardware for which I'm an app developer. This is due to the outrageous situation of COM module vendors (and/or SoC manufacturers) thinking that kernel 3.0 and horribly ancient/unsupported/unpatched userlands are acceptable in 2014/2015.

I run our .deb builds (and final debootstrap) in a bunch of docker containers, given that xapt/debcross toolchain has some stuff that, although these are awesome in themselves, still have some quirks when building stuff with awkward/complex dependencies and require a lot of hand-holding normally. So dockerizing the build env makes my life easier (some of it still has to be done from arm native chroot via qemu+binfmts). I use Jenkins to automate and integrate with the rest of our build tools but make files work too.

The /etc/passed et. al. copying from host is definitely the wrong thing to do in my circumstances as well; it's curious the various failure modes chroot builds can have.

Post reply on HN