Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cache is King: A guide for Docker layer caching in GitHub Actions

blacksmith.sh

Post reply on HN
Print view View original

71–80 of 111 posts

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#71

Post by jpgvm »

humanrebar wrote: post #70
jpgvm wrote: post #67

Earlier quoted context omitted.

Bazel isn't for everyone which is why I suggested using any similar tool, jib, Nix, etc. Just not Dockerfile (or if you are going to use Dockerfile only use ADD). Also just because you don't have experience with something doesn't make it a bad choice. I would recommending understanding it first, why your coworker chose it and how other tools would actually do in the same role, grass is often greener on the other side…

If you're saying to use a proper dependency management system (package manager or monorepo build system) and keep Docker to mostly dumb installs, I agree. Though I also think Nix and Bazel are typically not the right starting points for most projects. If you're not committed to having at least four experts on those complex tools in perpetuity, find something simpler. To be clear, inventing your own, better system is…

Yeah it's a conundrum. The easiest time to adopt something like Bazel/Buck/etc is at the start. However that is when tools like that provide the least value which given their additional friction isn't a good trade-off.

I recently started a side project and decided to do the whole thing using Gradle instead of Bazel. Essentially committing to writing absolutely everything in Kotlin instead of going for a multi-language setup with a more complex build tool. However Kotlin is a bit special in that regard because with multi-platform you can build for server/desktop/mobile/web all in one language with one build system.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#72

Post by CSSer »

zer00eyz wrote: post #69
Arbortheus wrote: post #64

Earlier quoted context omitted.

GitHub really need to invest in their CI. It is a second-class feature in the platform, but should be the beating heart of every SaaS team. GitLab CI is leaps and bounds ahead.

Linked in is playing with twitch like video. Zoom is adding in email. Years ago I worked for a bank. You know what happens if you set up bill pay with a bank? You're unlikely to end that relationship. Because who the fuck wants to do all that work to move. Your labor, your suffering (cause setting up bill pay sucks) is an egress fee. If you have GitHub acting as anything other than your public facing code repo you're…

The funny thing about this is that as far as most software engineers are concerned these things are generic competencies. As long as the price isn’t egregious and the feature-set is rich, we really don’t and shouldn’t care if we’re locked in for this. Some tools do belong together, and most people’s job in this sector shouldn’t be to spend half their time fiddling with devOps/project management tools, it should be to make/fix software. If you don’t believe me, consider that even in the scenario that you describe, any VCS platform is ultimately going to require a robust API to support integrations with other tools anyway, which will be orders of magnitude more difficult to accomplish than decent, built-in reasonable ops/pm features. This is coming from the person who typically agrees with you about lock-in. I’m afraid in this case your approach gets you JIRA and https://ifuckinghatejira.com/

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#73

Post by jonathanlydall »

jpgvm wrote: post #67
Arbortheus wrote: post #66

Earlier quoted context omitted.

Please no! Do not use Bazel unless you have a platform team with multiple people who know how to use it - e.g. large Google-like teams. We had “the Bazel guy” in our mid-sized company that Bazelified so many build processes, then left. It has been an absolute nightmare to maintain because no normal person has any experience with this tooling. It’s very esoteric. People in our company have reluctantly had to pick up B…

Bazel isn't for everyone which is why I suggested using any similar tool, jib, Nix, etc. Just not Dockerfile (or if you are going to use Dockerfile only use ADD). Also just because you don't have experience with something doesn't make it a bad choice. I would recommending understanding it first, why your coworker chose it and how other tools would actually do in the same role, grass is often greener on the other side…

Lack of experience is a perfectly valid and often very rationale reason for something being a bad choice, especially when considering upskilling costs and possible challenges in finding new hires proficient in the chosen technology.

The new technology needs to be sufficiently better than the existing to justify the investment or ongoing additional cost, and not just “has more features”, it should be solving problems which may otherwise not be reasonably solvable.

In a past job we had an incident where a dev had unilaterally decided to develop parts of a .NET project in F#, when the contract was for a C# project to be ultimately handed over to the client.

This was a run of mill back-end API, there were no interesting requirements that could possibly justify saddling the client with a need to hire for a relatively niche language.

The dev in question had this general view that F# was an underrated language and technically better than C# and if other devs would just give it a chance, they’d see for themselves.

What they totally ignored is that hiring C# devs is super easy here, F# though, not so much.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#75

Post by flatline »

zer00eyz wrote: post #69
Arbortheus wrote: post #64

Earlier quoted context omitted.

GitHub really need to invest in their CI. It is a second-class feature in the platform, but should be the beating heart of every SaaS team. GitLab CI is leaps and bounds ahead.

Linked in is playing with twitch like video. Zoom is adding in email. Years ago I worked for a bank. You know what happens if you set up bill pay with a bank? You're unlikely to end that relationship. Because who the fuck wants to do all that work to move. Your labor, your suffering (cause setting up bill pay sucks) is an egress fee. If you have GitHub acting as anything other than your public facing code repo you're…

I’ve migrated between devops platforms multiple times on multiple projects. The barrier is not really that high, and the cost of losing some data is relatively low. You can script most of it or pay a small fee for a user friendly plugin. There are lots of roughly equivalent options, some of them free. It’s nothing like, say, migrating between cloud providers.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#76

Post by notnmeyer »

adityamaru wrote: post #7
notnmeyer wrote: post #2

this is pretty neat—it’s been a while since i’ve tried caching layers with gha. it used to be quite frustrating. my previous experience was that in nearly all situations the time spent sending and retrieving cache layers over the network wound up making a shorter build step moot. ultimately we said “fuck it” and focused on making builds faster without (docker layer) caching.

Yeah that still holds true to some extent today with the GHA cache. Blacksmith colocates its cache with our CI runners, and ensures that they're in the same local network allowing us to saturate the NIC and provide much faster cache reads/writes. We're also thinking of clever ways to avoid downloading from a cache entirely and instead bind mount cache volumes over the network into the CI runner. Still early days, but…

there’s probably a cool consistent hashing solution where jobs are routed to a host that that is likely to have the cache stored locally already and can be mounted into the containers.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#77

Post by notnmeyer »

kylegalbraith wrote: post #47
notnmeyer wrote: post #2

this is pretty neat—it’s been a while since i’ve tried caching layers with gha. it used to be quite frustrating. my previous experience was that in nearly all situations the time spent sending and retrieving cache layers over the network wound up making a shorter build step moot. ultimately we said “fuck it” and focused on making builds faster without (docker layer) caching.

Yup! We observed the same thing back before we built Depot. The act of saving/loading cache over a GHA network pretty much negated any performance gain from layer caching. So, we created a solution to persist cache to NVMe disks and orchestrate that across builds so it's immediately available on the next build. All the performance of layer caching without any network transfer. The registry cache idea is a neat idea,…

totally, your approach is the right one and anything reasonable is going to focus on collocating the cache as close as possible to where the build runs.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#78

Post by mshekow »

remram wrote: post #68
mshekow wrote: post #53

I took a detailed look at Docker's caching mechanism (actually: BuildKit) in this article https://www.augmentedmind.de/2023/11/19/advanced-buildkit-ca... There I also explain that IF you use a registry cache import/export, you should use the same registry to which you are also pushing your actual image, and use the "image-manifest=true" option (especially if you are targeting GHCR - on DockerHub "image-manifest=true"…

Thanks, this is a very thorough explanation. Is there really no way to cache the 'cachemount' directories?

The only option I know is to use network shares/disks, but you need to make sure that each share/disk is only used by one BuildKit process at a time.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#79

Post by oftenwrong »

manx wrote: post #30

Earthly solves this really well: https://earthly.dev They rethink Dockerfiles with really good caching support.

The caching support is mostly the same. Both Earthly and Dockerfile are BuildKit frontends. BuildKit provides the caching mechanisms.

A possible exception is the "auto skip" feature for Earthly Cloud, since I do not know how that is implemented.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#80

Post by clintonb »

user- wrote: post #9

I feel like ive seen so many new companies just providing cheaper github actions.

They provide Actions _runners_ because GitHub runners are quite expensive (per CPU and GB of memory) compared to the underlying cost of a Kubernetes node on most cloud providers or bare metal. Of course, that assumes you’ve already paid the cost to setup a cluster, which is not free.
Post reply on HN