Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cache is King: A guide for Docker layer caching in GitHub Actions

blacksmith.sh

Post reply on HN
Print view View original

21–30 of 111 posts

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#21

Post by adityamaru »

solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

totally agree, github actions has done an excellent job at this lowest layer of the build pipeline today but is woefully inadequate the minute your org hits north of 50 engineers

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#22

Post by aayushshah15 »

solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

Did you consider using a local (in the same VPC) docker registry mirror perhaps? https://docs.docker.com/docker-hub/mirror/

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#23

Post by withinboredom »

maxmcd wrote: post #3

I weep for this period of time where we don't have sticky disks readily available for builds. Uploading the layer cache each time is such a coarse and time-consuming way to cache things. Maybe building from scratch all the time is a good correctness decision? Maybe stale values in disks is a tricky enough issue to want to avoid entirely? If you keep a stack of disks around and grab a free one when the job starts you'…

Interesting.

I remember working on a project where the first clean build would always fail, and only incremental builds could succeed. I was a junior at the time, so this was 15-20 years ago. I remember spending some time trying to get it to succeed from a clean build and my lead pulling me aside: he said it was an easy fix, but if we fixed it, the ops guys would insist on building from scratch for every build. So please, stop.

Personally, unless you have an exotic build env, it’s usually faster and easier to simply build in the runner. If you need a docker image at the end, build a dockerfile that simply copies the artifacts from disk.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#24

Post by remdoWater »

damianh wrote: post #17
aayushshah15 wrote: post #14

Earlier quoted context omitted.

how do you ensure isolation between runs on a self hosted runner that way?

Selh-hosted runners can be ephemeral too. With such either mount the cache as a disk or bake docker layers/images into the runner image.

This requires a lot of work from a dev inf team, though. Not as straightforward for an average team.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#25

Post by damianh »

remdoWater wrote: post #24
damianh wrote: post #17

Earlier quoted context omitted.

Selh-hosted runners can be ephemeral too. With such either mount the cache as a disk or bake docker layers/images into the runner image.

This requires a lot of work from a dev inf team, though. Not as straightforward for an average team.

I won't disagree. It should be easier imo. I guess this is why a cottage industry has sprung up addressing such e.g. https://news.ycombinator.com/item?id=39930908

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#26

Post by damianh »

airspeedjangle wrote: post #16
boronine wrote: post #15

Earlier quoted context omitted.

What kind of isolation do you need? We are building our own code so I don't see the need for isolation beyond a clear directory.

Shared runner infrastructure in a big company. It's pretty common to treat these situations as multi-tenant low trust environments.

Plenty of marketplace actions will install things and/or mutate the runner. It's a matter of time before someone does something or there's a build that doesn't cleannup after itself (e.g. leaving test processes running) that ruins the day for everyone else.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#27

Post by damianh »

aayushshah15 wrote: post #11

(blacksmith co-founder here) it's unfortunate the amount of expertise / tinkering required to get "incrementalism" in docker builds in github actions. we're hoping to solve this with some of the stuff we have in the pipeline in the near future.

The fact that GitHub don't provide a better solution here has to be actually costing them money with the network usage and extra agent time consumed. Right?

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#28

Post by boundlessdreamz »

remdoWater wrote: post #24
damianh wrote: post #17

Earlier quoted context omitted.

Selh-hosted runners can be ephemeral too. With such either mount the cache as a disk or bake docker layers/images into the runner image.

This requires a lot of work from a dev inf team, though. Not as straightforward for an average team.

runs-ons supports custom images - https://runs-on.com/features/byoi/ and caching to S3 - https://runs-on.com/reference/caching/

I haven't used it yet but these two features make it the clear favourite for me in alternate github action runners

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#29

Post by aayushshah15 »

damianh wrote: post #27
aayushshah15 wrote: post #11

(blacksmith co-founder here) it's unfortunate the amount of expertise / tinkering required to get "incrementalism" in docker builds in github actions. we're hoping to solve this with some of the stuff we have in the pipeline in the near future.

The fact that GitHub don't provide a better solution here has to be actually costing them money with the network usage and extra agent time consumed. Right?

GitHub has perverse incentives to not fix this problem because they charge customers based on usage (by the minute), so they make more money by providing slower builds to end-users.
Post reply on HN