Live data from Hacker News

Cache is King: A guide for Docker layer caching in GitHub Actions

blacksmith.sh

61–70 of 111 posts

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#61
post #44

The trick to Docker (well OCI) images is never under any circumstance use `docker build` or anything based on it. Dockerfile is your enemy. Use tools like Bazel + rules_oci or Gradle + jib and never spend time thinking about image builds taking time at all.

+1 to this, migrating our build setup to Nix + nix2container decreased our pipeline duration for incremental changes by a lot, thanks to Nix's granular caching abilities.

Yeah I really need to actually sit down and learn Nix, seems like it can solve this in a more general way for cases where the thing you want to run is packaged for Nix already.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#62

Docker layer caching is complicated! CircleCI has an implementation that used to use a detachable disk, but that had issues with concurrency It’s since been replaced with an approach that uses a docker plugin under the hood to store layers in object storage https://circleci.com/docs/docker-layer-caching/

Is it any better than buildx cache then, that also stores caches in object storage (via OCI registry)?

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#63
post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

Additionally, docker build refuses to cause any side effects to the host system. This makes any kind of caching difficult by design. IMO, if possible, consider doing your build outside of docker and just copying it into a scratch container...

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#64
post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

GitHub really need to invest in their CI. It is a second-class feature in the platform, but should be the beating heart of every SaaS team.

GitLab CI is leaps and bounds ahead.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#65
post #8

This is wild. I've spent the last three weeks working on this stuff for two separate clients. Important note if you're taking advice: cache-from and cache-to both accept multiple values. Cache to just ouputs the cache data to all the ones specified. cache-from looks for cache hits in the sources in-order. You can do some clever stuff to maximize cache hits with the least amount of downloading using the right combinat…

That’s a great idea.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#66
post #44

The trick to Docker (well OCI) images is never under any circumstance use `docker build` or anything based on it. Dockerfile is your enemy. Use tools like Bazel + rules_oci or Gradle + jib and never spend time thinking about image builds taking time at all.

Please no! Do not use Bazel unless you have a platform team with multiple people who know how to use it - e.g. large Google-like teams.

We had “the Bazel guy” in our mid-sized company that Bazelified so many build processes, then left.

It has been an absolute nightmare to maintain because no normal person has any experience with this tooling. It’s very esoteric. People in our company have reluctantly had to pick up Bazel tech debt tasks, like how the rules_docker package got randomly deprecated and replaced with rules_oci with a different API, which meant we could no longer update our Golang services to new versions of Go.

In the process we’ve broken CI, builds on Mac, had production outages, and all kinds of peculiarities and rollbacks needed that have been introduced because of an over-engineered esoteric build system that no one really cares about or wanted.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#67
post #44

The trick to Docker (well OCI) images is never under any circumstance use `docker build` or anything based on it. Dockerfile is your enemy. Use tools like Bazel + rules_oci or Gradle + jib and never spend time thinking about image builds taking time at all.

Please no! Do not use Bazel unless you have a platform team with multiple people who know how to use it - e.g. large Google-like teams. We had “the Bazel guy” in our mid-sized company that Bazelified so many build processes, then left. It has been an absolute nightmare to maintain because no normal person has any experience with this tooling. It’s very esoteric. People in our company have reluctantly had to pick up B…

Bazel isn't for everyone which is why I suggested using any similar tool, jib, Nix, etc. Just not Dockerfile (or if you are going to use Dockerfile only use ADD).

Also just because you don't have experience with something doesn't make it a bad choice. I would recommending understanding it first, why your coworker chose it and how other tools would actually do in the same role, grass is often greener on the other side until you get there.

Personally I went through a bit of an adventure with Bazel. My first exposure to it was similar to yours, was used in a place I didn't understand for reasons I didn't understand, broke in ways I didn't understand and (regretfully) didn't want to spend time understanding.

The reality was once I sat down to use it properly and understood the concepts a lot of things made sense and a whole bunch of very very difficult things became tractable.

That last bit is super important. Bazel raises the baseline effort to do something with the build system, which annoys people that don't want to invest time in understanding a build system. However it drastically reduces the complexity of extremely difficult things like fully byte for byte reproducible builds, extremely fast incremental builds and massive build step parallelization through remote build execution.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#68
post #53

I took a detailed look at Docker's caching mechanism (actually: BuildKit) in this article https://www.augmentedmind.de/2023/11/19/advanced-buildkit-ca... There I also explain that IF you use a registry cache import/export, you should use the same registry to which you are also pushing your actual image, and use the "image-manifest=true" option (especially if you are targeting GHCR - on DockerHub "image-manifest=true"…

Thanks, this is a very thorough explanation.

Is there really no way to cache the 'cachemount' directories?

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#69
post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

GitHub really need to invest in their CI. It is a second-class feature in the platform, but should be the beating heart of every SaaS team. GitLab CI is leaps and bounds ahead.

Linked in is playing with twitch like video.

Zoom is adding in email.

Years ago I worked for a bank. You know what happens if you set up bill pay with a bank? You're unlikely to end that relationship. Because who the fuck wants to do all that work to move.

Your labor, your suffering (cause setting up bill pay sucks) is an egress fee.

If you have GitHub acting as anything other than your public facing code repo you're locking yourself into the platform. Bug tracking, code review, CI pipelines, GitHub features that are going to keep you from moving quickly if you need to change providers.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#70
post #67

Earlier quoted context omitted.

Please no! Do not use Bazel unless you have a platform team with multiple people who know how to use it - e.g. large Google-like teams. We had “the Bazel guy” in our mid-sized company that Bazelified so many build processes, then left. It has been an absolute nightmare to maintain because no normal person has any experience with this tooling. It’s very esoteric. People in our company have reluctantly had to pick up B…

Bazel isn't for everyone which is why I suggested using any similar tool, jib, Nix, etc. Just not Dockerfile (or if you are going to use Dockerfile only use ADD). Also just because you don't have experience with something doesn't make it a bad choice. I would recommending understanding it first, why your coworker chose it and how other tools would actually do in the same role, grass is often greener on the other side…

If you're saying to use a proper dependency management system (package manager or monorepo build system) and keep Docker to mostly dumb installs, I agree.

Though I also think Nix and Bazel are typically not the right starting points for most projects. If you're not committed to having at least four experts on those complex tools in perpetuity, find something simpler.

To be clear, inventing your own, better system is typically as bad. Language specific ecosystems can be too, but it's often hard to avoid both Maven and gradle if you're a Java shop, for instance.

Post reply on HN