Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building small Docker images faster

sgt.hootr.club

Post reply on HN
Print view View original

11–20 of 28 posts

Re: Building small Docker images faster

#11

Post by abound »

lrvick wrote: post #10

For even smaller images that are always deterministic/reproducible with a multi-party signed supply chain, check out https://stagex.tools

Might want to disclose that you built it.

Also, I took a quick look and I don't understand how your tool could possibly produce "even smaller images". The article is using multi-stage builds to produce a final Docker image that is quite literally just the target binary in question (based on the scratch image), whereas your tool appears be a whole Linux distribution.

Re: Building small Docker images faster

#12

Post by yjftsjthsd-h »

zerotolerance wrote: post #2

I always like finding people advocating for older sage knowledge and bringing it forward for new audiences. That said, as someone who wrote a book about Docker and has lived the full container journey I tend to skip the containerized build all together. Docker makes for great packaging. But containerizing ever step of the build process or even just doing it in one big container is a bit extra. Positioning it as a bui…

I quite strongly disagree; a Dockerfile is a fairly good way to describe builds, a uniform approach across ecosystems, and the self contained nature is especially useful for building software without cluttering the host with build dependencies or clashing with other things you want to build. I like it so much that I've started building binaries in docker even for programs that will actually run on the host!

Re: Building small Docker images faster

#15

Post by solatic »

zerotolerance wrote: post #2

I always like finding people advocating for older sage knowledge and bringing it forward for new audiences. That said, as someone who wrote a book about Docker and has lived the full container journey I tend to skip the containerized build all together. Docker makes for great packaging. But containerizing ever step of the build process or even just doing it in one big container is a bit extra. Positioning it as a bui…

Agree, and I would go another step to suggest dropping Docker altogether for building the final container image. It's quite sad that Docker requires root to run, and all the other rootless solutions seem to require overcomplicated setups. Rootless is important because, unless you're providing CI as a public service and you're really concerned about malicious attackers, you will get way, way, way more value out of semi-permanent CI workers that can maintain persistent local caches compared to the overhead of VM enforced isolation. You just need an easy way to wipe the caches remotely, and a best-effort at otherwise isolating CI builds.

A lot of teams should think long and hard about just taking build artifacts, throwing them into their expected places in a directory taking the place of chroot, generating a manifest JSON, and wrapping everything in a tar, which is indeed a container.

Re: Building small Docker images faster

#16

Post by solatic »

yjftsjthsd-h wrote: post #12
zerotolerance wrote: post #2

I always like finding people advocating for older sage knowledge and bringing it forward for new audiences. That said, as someone who wrote a book about Docker and has lived the full container journey I tend to skip the containerized build all together. Docker makes for great packaging. But containerizing ever step of the build process or even just doing it in one big container is a bit extra. Positioning it as a bui…

I quite strongly disagree; a Dockerfile is a fairly good way to describe builds, a uniform approach across ecosystems, and the self contained nature is especially useful for building software without cluttering the host with build dependencies or clashing with other things you want to build. I like it so much that I've started building binaries in docker even for programs that will actually run on the host!

It can indeed be uniform across ecosystems, but it's slow. There's a very serious difference between being on a team where CI takes ~1 minute to run, vs. being on a team where CI takes a half hour or even, gasp, longer. A large part of that is the testing story, sure, but when you're really trying to optimize CI times, then every second counts.

Re: Building small Docker images faster

#17

Post by maccard »

solatic wrote: post #16
yjftsjthsd-h wrote: post #12

Earlier quoted context omitted.

I quite strongly disagree; a Dockerfile is a fairly good way to describe builds, a uniform approach across ecosystems, and the self contained nature is especially useful for building software without cluttering the host with build dependencies or clashing with other things you want to build. I like it so much that I've started building binaries in docker even for programs that will actually run on the host!

It can indeed be uniform across ecosystems, but it's slow . There's a very serious difference between being on a team where CI takes ~1 minute to run, vs. being on a team where CI takes a half hour or even, gasp, longer. A large part of that is the testing story, sure, but when you're really trying to optimize CI times, then every second counts.

You can have fast pipelines in containers - I’ve worked in quick containerised build environments and agonisingly slow non-containerised places, the difference is whether anyone actually cares and if there’s a culture of paying attention to this stuff.

Re: Building small Docker images faster

#18

Post by yjftsjthsd-h »

solatic wrote: post #16
yjftsjthsd-h wrote: post #12

Earlier quoted context omitted.

I quite strongly disagree; a Dockerfile is a fairly good way to describe builds, a uniform approach across ecosystems, and the self contained nature is especially useful for building software without cluttering the host with build dependencies or clashing with other things you want to build. I like it so much that I've started building binaries in docker even for programs that will actually run on the host!

It can indeed be uniform across ecosystems, but it's slow . There's a very serious difference between being on a team where CI takes ~1 minute to run, vs. being on a team where CI takes a half hour or even, gasp, longer. A large part of that is the testing story, sure, but when you're really trying to optimize CI times, then every second counts.

If the difference is 30 minutes, containers (per se) are not the problem. If I was guessing blindly, it sounds like you're not caching/reusing layers, effectively throwing out a super easy way to cache intermediate artifacts and trashing performance for no good reason. And in fact, this is also a place where I think docker - when used correctly - is quite good, because if you (re)use layers sensibly it's trivial to get build caching without having to figure out a per-(language|build system|project) caching system.

Re: Building small Docker images faster

#19

Post by miladyincontrol »

solatic wrote: post #15
zerotolerance wrote: post #2

I always like finding people advocating for older sage knowledge and bringing it forward for new audiences. That said, as someone who wrote a book about Docker and has lived the full container journey I tend to skip the containerized build all together. Docker makes for great packaging. But containerizing ever step of the build process or even just doing it in one big container is a bit extra. Positioning it as a bui…

Agree, and I would go another step to suggest dropping Docker altogether for building the final container image. It's quite sad that Docker requires root to run, and all the other rootless solutions seem to require overcomplicated setups. Rootless is important because, unless you're providing CI as a public service and you're really concerned about malicious attackers, you will get way, way, way more value out of sem…

I mean personally I find nspawn to be a pretty simple way of doing rootless containers. Replace manifest JSON with a systemd service file and you've got a rootless container that can run on most linux systems without any non-systemd dependencies or strange configuration required. Dont even need to extract the tarball.

Re: Building small Docker images faster

#20

Post by lrvick »

abound wrote: post #11
lrvick wrote: post #10

For even smaller images that are always deterministic/reproducible with a multi-party signed supply chain, check out https://stagex.tools

Might want to disclose that you built it. Also, I took a quick look and I don't understand how your tool could possibly produce "even smaller images". The article is using multi-stage builds to produce a final Docker image that is quite literally just the target binary in question (based on the scratch image), whereas your tool appears be a whole Linux distribution.

I am one of the maintainers at this point, fair.

This would be a much smaller drop in replacement for the base images used in the post to give full source bootstrapped final binaries.

You can still from scratch for the final layer though of course and that would be unlikely to change size much though, to your point.

Post reply on HN