Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cache is King: A guide for Docker layer caching in GitHub Actions

blacksmith.sh

Post reply on HN
Print view View original

101–110 of 111 posts

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#101

Post by codethief »

Arbortheus wrote: post #64
solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

GitHub really need to invest in their CI. It is a second-class feature in the platform, but should be the beating heart of every SaaS team. GitLab CI is leaps and bounds ahead.

Wait, really? We have been using Gitlab CI for a few years and it's awful. I run into bugs and surprises (missing features etc.) on the daily.

Now I don't even want to imagine what Github Actions are like…

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#102

Post by pas »

yeswecatan wrote: post #95
pas wrote: post #57

Earlier quoted context omitted.

if you want fast builds it's worth spinning up a buildkit server on a beefy dedicated server. docker/nerdctl only transfers the context, everything else is cached on the builder. it's very useful for monorepos (where you usually want to build and tag images for every tested commit) and the builder directly pushes the images/tags/layers to the registry. (which can be just a new tag for already existing layer.) a noop…

i haven't looked into setting up a buildkit server. would it be easier to just attach an ebs volume?

you mean run an EC2 instance with EBS as buildkit server storage dir?

sure, it should work nicely. (I just prefer the local disk, it's just a cache after all.)

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#103

Post by deepsun »

Arbortheus wrote: post #64
solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

GitHub really need to invest in their CI. It is a second-class feature in the platform, but should be the beating heart of every SaaS team. GitLab CI is leaps and bounds ahead.

GitLab were the first to introduce built-in CI. GitHub followed their lead once it became a decision point for many.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#104

Post by lispisok »

solatic wrote: post #20

As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…

I use self-hosted runners. It wasnt even because we could have large disk for caching. Github pricing for their runners is so bad it was a no brainer to host our own.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#105

Post by paholg »

boronine wrote: post #13

I've spent days trying all of these solution at my company. All of these solutions suck, they are slow and only successful builds get their layers cached. This is a dead end. The only workable solution is to have a self-hosted runner with a big disk.

If you have the ability to choose the tooling, I recommend looking into building docker images with nix.

Yes, nix is complex. But its caching story is soooo much better than docker's, and all the other docker issues just disappear.

https://nix.dev/tutorials/nixos/building-and-running-docker-...

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#106

Post by spurin »

It would be possible to offload the caching to Docker Build Cloud transparently, it’s part of the Docker subscription service, every account gets free minutes - 50 free minutes a month so depending on usage, you may be able to get this at zero cost.

With this approach, you’d use buildx and remotely, they would manage and maintain cache amongst other benefits.

It does require a credit card signup (which takes $0 to mitigate fraud). Full transparency, I’m a Docker Captain and helped test this whilst it was called Hydrobuild.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#107

Post by SJC_Hacker »

Smaller images are also another way to go. At my last company, image sizes were like 2-3Gb. I was able to prune that down to ~1.5 GB. Boost and a custom clang/llvm build were particular major offenders here.

There's quite a bit of cruft that can be pruned.

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#108

Post by daulis »

mshekow wrote: post #53

I took a detailed look at Docker's caching mechanism (actually: BuildKit) in this article https://www.augmentedmind.de/2023/11/19/advanced-buildkit-ca... There I also explain that IF you use a registry cache import/export, you should use the same registry to which you are also pushing your actual image, and use the "image-manifest=true" option (especially if you are targeting GHCR - on DockerHub "image-manifest=true"…

After years of lurking, I made an account to reply to this

"image-manifest=true" was the magic parameter that I needed to make this work with a non-DockerHub registry (Artifactory). I spent a lot of time fighting this, and non-obvious error messages. Thank you!!

We use a multi-stage build for a DevContainer environment, and the final image is quite large (for various reasons), so a better caching strategy really helps in our use case (smaller incremental image updates, smaller downloads for developers, less storage in the repository, etc)

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#109

Post by mysza »

boronine wrote: post #13

I've spent days trying all of these solution at my company. All of these solutions suck, they are slow and only successful builds get their layers cached. This is a dead end. The only workable solution is to have a self-hosted runner with a big disk.

Can you share example of github actions? When i use docker/setup-buildx-action and local runner i can't make it use the cache. I think it's the "docker-container" runner's fault

Re: Cache is King: A guide for Docker layer caching in GitHub Actions

#110

Post by baobun »

jpgvm wrote: post #67
Arbortheus wrote: post #66

Earlier quoted context omitted.

Please no! Do not use Bazel unless you have a platform team with multiple people who know how to use it - e.g. large Google-like teams. We had “the Bazel guy” in our mid-sized company that Bazelified so many build processes, then left. It has been an absolute nightmare to maintain because no normal person has any experience with this tooling. It’s very esoteric. People in our company have reluctantly had to pick up B…

Bazel isn't for everyone which is why I suggested using any similar tool, jib, Nix, etc. Just not Dockerfile (or if you are going to use Dockerfile only use ADD). Also just because you don't have experience with something doesn't make it a bad choice. I would recommending understanding it first, why your coworker chose it and how other tools would actually do in the same role, grass is often greener on the other side…

> or if you are going to use Dockerfile only use ADD

So we should do

    ADD https://example.com/fetch-and-install.sh
    RUN sudo fetch-and-install.sh
instead of

    COPY fetch-and-install.sh .
    RUN sudo fetch-and-install.sh
Post reply on HN