Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cache is King: A guide for Docker layer caching in GitHub Actions

blacksmith.sh

Post reply on HN
Print view View original

81–90 of 111 posts

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#81

Post by kbolino »

matsemann wrote: post #45
solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

> as soon as you have more advanced needs If there's one thing I've learned over the years, is that we really seldom have advanced needs. Mostly we just want things to work a certain way, and will fight systems to make it behave so. It's easier to just leave it be. Like maven vs gradle; yes, gradle can do everything, but if you need that it's worth taking a step back and assess why the normal maven flow won't work. W…

It has been a few years, but last I recall, the key advantage of Gradle over Maven was not power so much as brevity. Doing many things in Maven required a dozen nested XML tags, while doing the same thing in Gradle was often a one-liner.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#82

Post by kbolino »

dboreham wrote: post #35

No! So much time spent debunking such broken "caching" solutions. Computers are very fast now. Use proper package/versioning systems (part of the problem here is that those are often also broken/badly designed).

"Computers are very fast now" is largely because of caching. The CPU has a cache, the disk drive has a cache, the OS has a cache, the HTTP client has a cache, the CDN serving the content has a cache, etc. There may be better ways to cache than at the level of Docker image layers, but no caching is the same as a cache miss on every request, which can be dozens, hundreds, or even thousands of times slower than a cache hit.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#83

Post by cpuguy83 »

candiddevmike wrote: post #63
solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

Additionally, docker build refuses to cause any side effects to the host system. This makes any kind of caching difficult by design. IMO, if possible, consider doing your build outside of docker and just copying it into a scratch container...

I'm not sure what you mean here. "RUN --mount type=cache,dest=/foo" is exactly for keeping a persistent cache on the host.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#84

Post by cpuguy83 »

cqqxo4zV46cp wrote: post #56
solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

I got it working, with intermediate layers, too. All to find that I didn’t see that material a performance benefit after taking into account how long it takes to pull from and push to the cache.

You might want to try the actions cache "--cache-to=gha --cache-from=gha", but still it needs to pull that stuff down, just that locality is likely better here.

There's also an action out there "GitHub cache dance" that will stick your whole buildkit state dir into the gha cache.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#85

Post by gigatexal »

Arbortheus wrote: post #64
solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

GitHub really need to invest in their CI. It is a second-class feature in the platform, but should be the beating heart of every SaaS team. GitLab CI is leaps and bounds ahead.

Yeah that’s my thought as well: this is something GitHub is supposed to do. Keep it simple on the users and leave the hard stuff to the creators/runners of the tool

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#87

Post by fireflash38 »

kylegalbraith wrote: post #41
solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

Thanks for the shout-out regarding Depot, really appreciate it. We came to the same conclusion regarding Docker layer cache and thus why we created Depot in the first place. The limitations and performance surrounding GitHub Actions cache leaves a lot to be desired.

Quick glance showed no, but is there no purely local for depot? It's all cloud based?

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#88

Post by intelVISA »

damianh wrote: post #25
remdoWater wrote: post #24

Earlier quoted context omitted.

This requires a lot of work from a dev inf team, though. Not as straightforward for an average team.

I won't disagree. It should be easier imo. I guess this is why a cottage industry has sprung up addressing such e.g. https://news.ycombinator.com/item?id=39930908

Now I've seen everything...

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#89

Post by unshavedyak »

CSSer wrote: post #72
zer00eyz wrote: post #69

Earlier quoted context omitted.

Linked in is playing with twitch like video. Zoom is adding in email. Years ago I worked for a bank. You know what happens if you set up bill pay with a bank? You're unlikely to end that relationship. Because who the fuck wants to do all that work to move. Your labor, your suffering (cause setting up bill pay sucks) is an egress fee. If you have GitHub acting as anything other than your public facing code repo you're…

The funny thing about this is that as far as most software engineers are concerned these things are generic competencies. As long as the price isn’t egregious and the feature-set is rich, we really don’t and shouldn’t care if we’re locked in for this. Some tools do belong together, and most people’s job in this sector shouldn’t be to spend half their time fiddling with devOps/project management tools, it should be to…

Tangent, boy i love that site's design. Simple, elegant, animations feel like they layer on-top of the primary UX (ie they add to the text. Rather than the text being delayed for the purpose of showing some fancy animation).

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#90

Post by LtWorf »

zer00eyz wrote: post #69
Arbortheus wrote: post #64

Earlier quoted context omitted.

GitHub really need to invest in their CI. It is a second-class feature in the platform, but should be the beating heart of every SaaS team. GitLab CI is leaps and bounds ahead.

Linked in is playing with twitch like video. Zoom is adding in email. Years ago I worked for a bank. You know what happens if you set up bill pay with a bank? You're unlikely to end that relationship. Because who the fuck wants to do all that work to move. Your labor, your suffering (cause setting up bill pay sucks) is an egress fee. If you have GitHub acting as anything other than your public facing code repo you're…

Well before github had a CI everybody used travis for free from it. Then they killed the free tier and people just started to switch.

It's trivial to switch from github to codeberg for example… So I don't think it's that bad to be honest.

Post reply on HN