Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cache is King: A guide for Docker layer caching in GitHub Actions

blacksmith.sh

Post reply on HN
Print view View original

41–50 of 111 posts

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#41

Post by kylegalbraith »

solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

Thanks for the shout-out regarding Depot, really appreciate it. We came to the same conclusion regarding Docker layer cache and thus why we created Depot in the first place. The limitations and performance surrounding GitHub Actions cache leaves a lot to be desired.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#42

Post by kylegalbraith »

boronine wrote: post #13

I've spent days trying all of these solution at my company. All of these solutions suck, they are slow and only successful builds get their layers cached. This is a dead end. The only workable solution is to have a self-hosted runner with a big disk.

This is definitely a direction to try. But if its faster Docker image builds and a layer caching system that actually works, you should definitely try out Depot. We automatically persist layer cache to persistent NVMe devices and orchestrate that to be immediately available across builds.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#43

Post by krainboltgreene »

aayushshah15 wrote: post #29
damianh wrote: post #27

Earlier quoted context omitted.

The fact that GitHub don't provide a better solution here has to be actually costing them money with the network usage and extra agent time consumed. Right?

GitHub has perverse incentives to not fix this problem because they charge customers based on usage (by the minute), so they make more money by providing slower builds to end-users.

They've also just completely refocused to AI in the last two years thanks to the microsoft/ChatGPT situation.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#44

Post by jpgvm »

The trick to Docker (well OCI) images is never under any circumstance use `docker build` or anything based on it. Dockerfile is your enemy.

Use tools like Bazel + rules_oci or Gradle + jib and never spend time thinking about image builds taking time at all.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#45

Post by matsemann »

solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

> as soon as you have more advanced needs

If there's one thing I've learned over the years, is that we really seldom have advanced needs. Mostly we just want things to work a certain way, and will fight systems to make it behave so. It's easier to just leave it be. Like maven vs gradle; yes, gradle can do everything, but if you need that it's worth taking a step back and assess why the normal maven flow won't work. What's so special with our app compared to the millions working just fine out of the box?

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#46

Post by kylegalbraith »

maxmcd wrote: post #3

I weep for this period of time where we don't have sticky disks readily available for builds. Uploading the layer cache each time is such a coarse and time-consuming way to cache things. Maybe building from scratch all the time is a good correctness decision? Maybe stale values in disks is a tricky enough issue to want to avoid entirely? If you keep a stack of disks around and grab a free one when the job starts you'…

Couldn't agree more. Somewhere, we lost the concept of disks in CI unless you run it yourself, and a lot of build tools could benefit from having them.

We came to the same conclusion and built Depot around this exact workflow for the Docker image build problem. We're now bringing that same tech into GitHub Actions workflows.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#47

Post by kylegalbraith »

notnmeyer wrote: post #2

this is pretty neat—it’s been a while since i’ve tried caching layers with gha. it used to be quite frustrating. my previous experience was that in nearly all situations the time spent sending and retrieving cache layers over the network wound up making a shorter build step moot. ultimately we said “fuck it” and focused on making builds faster without (docker layer) caching.

Yup! We observed the same thing back before we built Depot. The act of saving/loading cache over a GHA network pretty much negated any performance gain from layer caching. So, we created a solution to persist cache to NVMe disks and orchestrate that across builds so it's immediately available on the next build. All the performance of layer caching without any network transfer.

The registry cache idea is a neat idea, but in practice suffers the same problem.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#48

Post by glenjamin »

Docker layer caching is complicated!

CircleCI has an implementation that used to use a detachable disk, but that had issues with concurrency

It’s since been replaced with an approach that uses a docker plugin under the hood to store layers in object storage

https://circleci.com/docs/docker-layer-caching/

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#49

Post by ownagefool »

remdoWater wrote: post #24
damianh wrote: post #17

Earlier quoted context omitted.

Selh-hosted runners can be ephemeral too. With such either mount the cache as a disk or bake docker layers/images into the runner image.

This requires a lot of work from a dev inf team, though. Not as straightforward for an average team.

It's actually pretty easy.

Setup GitHub app. Install the arc helm charts. Install a buildkitd statefulset.

Update parans on build to use buildkitd.

That's not to say there aren't better caching strategies, but a really basic ephemeral setup is right there.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#50

Post by ValtteriL »

adityamaru wrote: post #32
ValtteriL wrote: post #31

Docker layer caching is one of the reasons I moved to Jenkins 2 years ago and have been very happy with it for the most part. I only need to install utils once and all build time goes to building my software. It even integrates nicely with Github. Result: 50% faster feedback. However, it needs a bit initial housekeeping and discipline to use correctly. For example using Jenkinsfiles is a must and using containers as…

do you self host your jenkins deployment in your AWS account?

Self host
Post reply on HN