Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building small Docker images faster

sgt.hootr.club

Post reply on HN
Print view View original

21–28 of 28 posts

Re: Building small Docker images faster

#21

Post by maccard »

cogman10 wrote: post #8
maccard wrote: post #3

Earlier quoted context omitted.

I’m inclined to agree with you about not building containers. That said, I find myself going around in circles. We have an app that uses a specific toolchain version, how do we install that version on a build machine without requiring an SRE ticket to update our toolchain? Containers nicely solve this problem. Then your builds get a little slow, so you want to cache things. Now your docker file looks like this. You w…

You can mount the current directory into docker and run an image of your tool. Here's an example of that from the docker maven. `docker run -it --rm --name my-maven-project -v "$(pwd)":/usr/src/mymaven -w /usr/src/mymaven maven:3.3-jdk-8 mvn clean install` You can get as fancy as you like with things like your `.m2` directory, this just gives you the basics of how you'd do that.

Thanks - this is an interesting idea I had never considered. I do like the layer based caching of dockerfiles, which you give up entirely for this but it allows for things like running containerised builds cached SCM checkouts (our repository is 300GB…)

Re: Building small Docker images faster

#22

Post by maccard »

pstuart wrote: post #5
maccard wrote: post #3

Earlier quoted context omitted.

I’m inclined to agree with you about not building containers. That said, I find myself going around in circles. We have an app that uses a specific toolchain version, how do we install that version on a build machine without requiring an SRE ticket to update our toolchain? Containers nicely solve this problem. Then your builds get a little slow, so you want to cache things. Now your docker file looks like this. You w…

Depending on how the container is structured, you could have the original container as a baseline default, and then have "enhanced" containers that use it as a base and overlay the caching and other errata to serve that specialized need.

I’ve tried this in the past, but it pushes the dependency management pf the layers into whatever is orchestrating the container build, as opposed to multi stage builds which will parallelise!

Not dismissing, but it’s just caveats every which way. I think in an ideal world I just want Bazel or Nixos without the baggage that comes with them - docker comes so close but yet falls so short of the finish line.

Re: Building small Docker images faster

#23

Post by solatic »

yjftsjthsd-h wrote: post #18
solatic wrote: post #16

Earlier quoted context omitted.

It can indeed be uniform across ecosystems, but it's slow . There's a very serious difference between being on a team where CI takes ~1 minute to run, vs. being on a team where CI takes a half hour or even, gasp, longer. A large part of that is the testing story, sure, but when you're really trying to optimize CI times, then every second counts.

If the difference is 30 minutes, containers (per se) are not the problem. If I was guessing blindly, it sounds like you're not caching/reusing layers, effectively throwing out a super easy way to cache intermediate artifacts and trashing performance for no good reason. And in fact, this is also a place where I think docker - when used correctly - is quite good, because if you (re)use layers sensibly it's trivial to g…

I'm exaggerating somewhat. But I'm familiar with Docker's multi-stage builds and how to attempt to optimize cache layers. The first problem that you run into, with ephemeral runners, is where the Docker cache is supposed to be downloaded from, and it's often not faster at all compared to re-downloading artifacts (network calls are network calls, and files are files after all). This is fundamentally different from per-language caching systems where libraries are known to be a dumb mirror of upstream, often hash-addressed for modern packaging, and thus are safe to share between builds, which means that it is safe to keep them on the CI runner and not be forced to download the cache for a build before starting it.

> without having to figure out a per-language caching systems

But most companies, even large ones, tend to standardize on no more than a handful of languages. Typescript, Python, Go, Java... I don't need something that'll handle caching for PHP or Erlang or Nix (not that you can really work easily with Nix inside a container...) or OCaml or Haskell... Yeah I do think there's a lot of room for companies to say, this is the standardized supported stack, and we put in some time to optimize the shit out of it because the DX dividends are incredible.

Re: Building small Docker images faster

#24

Post by OptionOfT »

solatic wrote: post #15
zerotolerance wrote: post #2

I always like finding people advocating for older sage knowledge and bringing it forward for new audiences. That said, as someone who wrote a book about Docker and has lived the full container journey I tend to skip the containerized build all together. Docker makes for great packaging. But containerizing ever step of the build process or even just doing it in one big container is a bit extra. Positioning it as a bui…

Agree, and I would go another step to suggest dropping Docker altogether for building the final container image. It's quite sad that Docker requires root to run, and all the other rootless solutions seem to require overcomplicated setups. Rootless is important because, unless you're providing CI as a public service and you're really concerned about malicious attackers, you will get way, way, way more value out of sem…

I like to build my stuff inside of Docker because it is my moat against changes of the environment.

We have our base images, and in there we install dependencies by version. That package then is the base for our code build. (as apt seemingly doesn't have any lock file support?).

In the subsequent built EVERYTHING is versioned, which allows us to establish provenance all the way up to the base image.

And next to that when we promote images from PR -> main we don't even rebuild the code. It's the same image that gets retagged. All in the name of preserving provenance.

Re: Building small Docker images faster

#25

Post by cogman10 »

maccard wrote: post #21
cogman10 wrote: post #8

Earlier quoted context omitted.

You can mount the current directory into docker and run an image of your tool. Here's an example of that from the docker maven. `docker run -it --rm --name my-maven-project -v "$(pwd)":/usr/src/mymaven -w /usr/src/mymaven maven:3.3-jdk-8 mvn clean install` You can get as fancy as you like with things like your `.m2` directory, this just gives you the basics of how you'd do that.

Thanks - this is an interesting idea I had never considered. I do like the layer based caching of dockerfiles, which you give up entirely for this but it allows for things like running containerised builds cached SCM checkouts (our repository is 300GB…)

Yeah, it's basically tradeoffs all around.

The benefit of this approach is it's a lot easier to make sure dependencies end up on the build node so you aren't redownloading and caching the same dependency for multiple artifacts. But then you don't get to take advantage of docker build caching to speed up things when something doesn't change.

That's the part about docker I don't love. I get why it's this way, but I wish there was a better way to have it reuse files between images. The best you can do is a cache mount. But that can run into size issues as time goes on which is annoying.

Re: Building small Docker images faster

#26

Post by solatic »

OptionOfT wrote: post #24
solatic wrote: post #15

Earlier quoted context omitted.

Agree, and I would go another step to suggest dropping Docker altogether for building the final container image. It's quite sad that Docker requires root to run, and all the other rootless solutions seem to require overcomplicated setups. Rootless is important because, unless you're providing CI as a public service and you're really concerned about malicious attackers, you will get way, way, way more value out of sem…

I like to build my stuff inside of Docker because it is my moat against changes of the environment. We have our base images, and in there we install dependencies by version. That package then is the base for our code build. (as apt seemingly doesn't have any lock file support?). In the subsequent built EVERYTHING is versioned, which allows us to establish provenance all the way up to the base image. And next to that…

You can still use a base image; you download it from the registry, extract the tar, then add your build artifacts before re-generating a manifest and re-tarring. If you specify a base image digest, you can also use a hash-addressed cache and share it between builds safely without re-downloading.

Once you have your container image, how you decide to promote it is a piece of cake, skopeo doesn't require root and often doesn't require re-pulling the full tar. Containerization is great, I'm specifically trying to point out that there are alternatives to Docker.

Re: Building small Docker images faster

#27

Post by yjftsjthsd-h »

solatic wrote: post #23
yjftsjthsd-h wrote: post #18

Earlier quoted context omitted.

If the difference is 30 minutes, containers (per se) are not the problem. If I was guessing blindly, it sounds like you're not caching/reusing layers, effectively throwing out a super easy way to cache intermediate artifacts and trashing performance for no good reason. And in fact, this is also a place where I think docker - when used correctly - is quite good, because if you (re)use layers sensibly it's trivial to g…

I'm exaggerating somewhat. But I'm familiar with Docker's multi-stage builds and how to attempt to optimize cache layers. The first problem that you run into, with ephemeral runners, is where the Docker cache is supposed to be downloaded from , and it's often not faster at all compared to re-downloading artifacts (network calls are network calls, and files are files after all). This is fundamentally different from pe…

I really don't see how that's different at all, certainly not fundamentally. You can download flat files over the network, and you can download OCI image layers over the network. I'm pretty sure those image layers are hash-addressed and safe to share between builds, too, and you should make every effort to keep them on the CI runner and reuse them.
Post reply on HN