31–40 of 111 posts
I only need to install utils once and all build time goes to building my software. It even integrates nicely with Github. Result: 50% faster feedback.
However, it needs a bit initial housekeeping and discipline to use correctly. For example using Jenkinsfiles is a must and using containers as agents is desirable.
Docker layer caching is one of the reasons I moved to Jenkins 2 years ago and have been very happy with it for the most part. I only need to install utils once and all build time goes to building my software. It even integrates nicely with Github. Result: 50% faster feedback. However, it needs a bit initial housekeeping and discipline to use correctly. For example using Jenkinsfiles is a must and using containers as…
Docker layer caching is one of the reasons I moved to Jenkins 2 years ago and have been very happy with it for the most part. I only need to install utils once and all build time goes to building my software. It even integrates nicely with Github. Result: 50% faster feedback. However, it needs a bit initial housekeeping and discipline to use correctly. For example using Jenkinsfiles is a must and using containers as…
I weep for this period of time where we don't have sticky disks readily available for builds. Uploading the layer cache each time is such a coarse and time-consuming way to cache things. Maybe building from scratch all the time is a good correctness decision? Maybe stale values in disks is a tricky enough issue to want to avoid entirely? If you keep a stack of disks around and grab a free one when the job starts you'…
So much time spent debunking such broken "caching" solutions.
Computers are very fast now. Use proper package/versioning systems (part of the problem here is that those are often also broken/badly designed).
As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…
Did you consider using a local (in the same VPC) docker registry mirror perhaps? https://docs.docker.com/docker-hub/mirror/
It's just a problem that requires big, local disks to solve.
But in the long run, as annoying as it is out build pipelines reduced but quite a few minutes per build.
As someone who spent way too much time chasing this rabbit, the real answer is Just Don't. GitHub Actions is a CI system that makes it easy to get started with simple CI needs but runs into hard problems as soon as you have more advanced needs. Docker caching is one of those advanced needs. If you have non-trivial Docker builds then you simply need on-disk local caching, period. Either use Depot or switch to self-hos…
No! So much time spent debunking such broken "caching" solutions. Computers are very fast now. Use proper package/versioning systems (part of the problem here is that those are often also broken/badly designed).
I have this set up in our pipeline, we also build the image early and use assets to move it between jobs. We've also just switched to self-hosted runners, so might look into shared disk. But in the long run, as annoying as it is out build pipelines reduced but quite a few minutes per build.
Glad it worked really well for you.
What made you switch to self-hosted runners?