Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cache is King: A guide for Docker layer caching in GitHub Actions

blacksmith.sh

Post reply on HN
Print view View original

91–100 of 111 posts

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#91

Post by adamgordonbell »

oftenwrong wrote: post #79
manx wrote: post #30

Earthly solves this really well: https://earthly.dev They rethink Dockerfiles with really good caching support.

The caching support is mostly the same. Both Earthly and Dockerfile are BuildKit frontends. BuildKit provides the caching mechanisms. A possible exception is the "auto skip" feature for Earthly Cloud, since I do not know how that is implemented.

Also CACHE keyword, for cache mounts. Makes incremental tools like compilers work well in the context of dockerfiles and layer caches.

That can extend beyond just producing docker iamges as well. Under the covers the CACHE keyword is how lib/rust in Earthly makes building Rust artifacts in CI faster.

https://github.com/earthly/earthly/issues/1399

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#92

Post by vladaionescu »

oftenwrong wrote: post #79
manx wrote: post #30

Earthly solves this really well: https://earthly.dev They rethink Dockerfiles with really good caching support.

The caching support is mostly the same. Both Earthly and Dockerfile are BuildKit frontends. BuildKit provides the caching mechanisms. A possible exception is the "auto skip" feature for Earthly Cloud, since I do not know how that is implemented.

I would add that 1. Earthly is meant for full CI/CD use-cases, not just for image building. We've forked buildkit to make that possible. And 2. remote caching is pretty slow overall because of the limited amount of data you can push/pull before it becomes performance-prohibitive. We have a comparison in our docs between remote runner (e.g. Earthly Satellites) vs remote cache [1].

[1]: https://docs.earthly.dev/docs/caching#sharing-cache

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#93

Post by joe0 »

tkiolp4 wrote: post #59

Docker has been among us for years. Why isn’t efficient caching already implemented out of the box? It’s a leaking abstraction that users have to deal with. Annoying at best.

They actually have recently, but it’s a separate (payed) product offering: https://www.docker.com/products/build-cloud/

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#94

Post by stackskipton »

matsemann wrote: post #45
solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

> as soon as you have more advanced needs If there's one thing I've learned over the years, is that we really seldom have advanced needs. Mostly we just want things to work a certain way, and will fight systems to make it behave so. It's easier to just leave it be. Like maven vs gradle; yes, gradle can do everything, but if you need that it's worth taking a step back and assess why the normal maven flow won't work. W…

I'm sad as DevOps Engineer I only have one upvote to give. YAGNI needs to be every team motto.

We tried caching at several companies. Outside node builds, it was never worth it. Horray, our .Net builds took 15 seconds instead of 4 Minutes. Eventually you realized no one cared since we averaged deployments every 4 days outside of outages and time being burned by it just wasn't there.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#95

Post by yeswecatan »

pas wrote: post #57
bushbaba wrote: post #38

Earlier quoted context omitted.

Can’t you use s3 + mountpoint for most distributed CI cache needs?

if you want fast builds it's worth spinning up a buildkit server on a beefy dedicated server. docker/nerdctl only transfers the context, everything else is cached on the builder. it's very useful for monorepos (where you usually want to build and tag images for every tested commit) and the builder directly pushes the images/tags/layers to the registry. (which can be just a new tag for already existing layer.) a noop…

i haven't looked into setting up a buildkit server. would it be easier to just attach an ebs volume?

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#96

Post by yeswecatan »

cqqxo4zV46cp wrote: post #56
solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

I got it working, with intermediate layers, too. All to find that I didn’t see that material a performance benefit after taking into account how long it takes to pull from and push to the cache.

how large is your cache and how long does the pull/push take?

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#98

Post by crote »

solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

> GitHub Actions is a CI system that makes it easy to get started

It's not even that! Coming from GitLab I was quite surprised at how poor the "getting started" experience was. Rather than a simple "on push, run command X" you first have to do a deep dive into actions/events/workflows/jobs/runs, and then figure out what kind of weird tooling is used for trivial things like checking out your code, or storing artifacts.

And then you try to unify your pipeline across several projects because that's what Github is heavily promoting with the whole "uses: actions/checkout" reuse thing - but it turns out to be a huge hassle to get it working because nothing works the way you'd expect it to work.

In the end I did get GHA to do what I was already doing in GitLAb, but it took me ten times as long as it did originally setting it up. I believe GHA is flexible and powerful enough to be well-suited for medium-sized companies, but it's neither easy enough for small companies, nor powerful enough for large companies. It's one of the few Github features I genuinely dislike using.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#99

Post by DanielHB »

solatic wrote: post #36
aayushshah15 wrote: post #22

Earlier quoted context omitted.

Did you consider using a local (in the same VPC) docker registry mirror perhaps? https://docs.docker.com/docker-hub/mirror/

It's not the pulls that are the problem, it's caching intermediate layers from the build that is the problem. As soon as you introduce a networked registry, the time it takes to pull layers from the registry cache and push them back to the registry cache are frequently not much better than simply rebuilding the layers, not to mention the additional compute/storage cost of running the registry cache itself. It's just…

Yeah I had the exact same problem and came to the same conclusion.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#100

Post by _cenw »

Arbortheus wrote: post #64
solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

GitHub really need to invest in their CI. It is a second-class feature in the platform, but should be the beating heart of every SaaS team. GitLab CI is leaps and bounds ahead.

Always love to shock more people with the random fact that GitHub Actions is Azure DevOps Pipelines in a trenchcoat (and Azure Pipelines is seemingly abandoned / in maintenance mode now).

The runner code is on GitHub, and it's not pretty. In fact last time I ran it it had trouble generating stable exit codes.

Post reply on HN