Caching gets trickier when tasks spin up containers.
Reverse engineering GitHub Actions cache to make it fast
21–30 of 39 posts
Re: Reverse engineering GitHub Actions cache to make it fast
#22At this point this is considered a baseline feature of every good GitHub Actions third-party provider, but nice to see the write-up and solution they came up with!
Note that GitHub Actions Cache v2 is actually very good in terms of download/upload speed right now, when running from GitHub managed runners. The low speed Blacksmith was seeing before is just due to their slow (Hetzner?) network.
I benchmarked most providers (I maintain RunsOn) with regards to their cache performance here: https://runs-on.com/benchmarks/github-actions-cache-performa...
Re: Reverse engineering GitHub Actions cache to make it fast
#23I am struggling with justification for CI/CD pipelines that are so complex this kind of additional tooling becomes necessary. There are ways to refactor your technology so that you don't have to suffer so much at integration and deployment time. For example, the use of containers and hosted SQL where neither are required can instantly 10x+ the complexity of deploying your software. The last few B2B/SaaS projects I wo…
Here is my cynical take in ci. Firstly, testing is almost never valued by management which would rather close a deal on half finished promises than actually build a polished, reliable product (they can always scapegoat the eng team if things go wrong with the customer anyway). So, to begin with, testing is rarely prioritized. But most developer orgs eventually realize that centralized testing is necessary or else eve…
> they typically are going to sell you one of two things or both: support or cpu time
At what point does the line between CPU time in GH Actions and CPU time in the actual production environment lose all meaning? Why even bother moving to production? You could just create a new GH action called "Production" that gets invoked at the end of the pipeline and runs perpetually.
I think I may have identified a better canary here. If the CI/CD process takes so much CPU time that we are consciously aware of the resulting bill, there is definitely something going wrong.
Re: Reverse engineering GitHub Actions cache to make it fast
#24At this point this is considered a baseline feature of every good GitHub Actions third-party provider, but nice to see the write-up and solution they came up with! Note that GitHub Actions Cache v2 is actually very good in terms of download/upload speed right now, when running from GitHub managed runners. The low speed Blacksmith was seeing before is just due to their slow (Hetzner?) network. I benchmarked most provi…
Also note this open-source project that shows a way to implement this: https://github.com/falcondev-oss/github-actions-cache-server
Re: Reverse engineering GitHub Actions cache to make it fast
#25Related read: Cirrus Labs also wrote a drop-in replacement for GH Actions cache on their platform.
https://cirrus-runners.app/blog/2024/04/23/speeding-up-cachi...
Re: Reverse engineering GitHub Actions cache to make it fast
#26> iptables was already doing heavy lifting for other subsystems inside our environment, and with each VM adding or removing its own set of rules, things got messy fast, and extremely flakey We saw the same thing at Vercel. Back when we were still doing docker-as-a-service we used k8s for both internal services as well as user deployments. The latter lead to master deadlocks and all sorts of SRE nightmares (literally)…
[flagged]
What a delightfully arrogant comment :) This was about ten years ago. At the time, given the constraints, iptables did not have a public, stable programmatic interface, no. Perhaps you used other firewall systems/frameworks. We had to interop with Docker. I don't remember specifics; I do remember going very deep down that rabbit hole.
For what it's worth, the development team, at least in the early ZEIT days, were the most competent engineers I've ever worked with on a team.
Re: Reverse engineering GitHub Actions cache to make it fast
#27Related read: Cirrus Labs also wrote a drop-in replacement for GH Actions cache on their platform. https://cirrus-runners.app/blog/2024/04/23/speeding-up-cachi... https://github.com/cirruslabs/cache
It is not transparent though, so it doesn't work with all the other actions that use the cache toolkit, and you have to reference a specific action.
Re: Reverse engineering GitHub Actions cache to make it fast
#28I'm currently migrating some stuff from azdo to GHA, and have been putting past lessons to serious use:
* Perf: don't use "install X" (Node, .Net, Ruby, Python, etc.) tasks. Create a container image with all your deps and use that instead.
* Perf: related to the last, keep multiple utility container images around of varying degrees of complexity. For example, in our case, I decided on PowerShell because we have some devs with Windows and it's the easiest to get working across Linux+Windows - so my simplest container has pwsh and some really basic tools (git, curl, etc.). I build another container on that which has .Net deps. Then each .Net repo uses that to:
* Perf: don't use the cache action at all. Run a job nightly that pulls down your code into a container, restore/install to warm the cache, then delete the code. `RUN --mount` is a good way to avoid creating a layer with your code in it.
* Maintainability: don't write big scripts in your workflow file. Create scripts as files that can also be executed on your local machine. Keep the "glue code" between GHA and your script in the workflow file. I slightly lie here, I do source in a single utility script that reads in GHA envars and has functions to set CI variables and so forth (that does sensible things when run locally).
Our CI builds are stupid fast. Comparatively speaking.
For the OP (I just sent your pricing page to my manager ;) ): having a colocated container registry for these types of things would be super useful. I would say you don't need to expose it to the internet, but sometimes you do need to be able to `podman run` into an image for debug purposes.
[1]: https://docs.github.com/en/actions/how-tos/writing-workflows...
Re: Reverse engineering GitHub Actions cache to make it fast
#29> iptables was already doing heavy lifting for other subsystems inside our environment, and with each VM adding or removing its own set of rules, things got messy fast, and extremely flakey We saw the same thing at Vercel. Back when we were still doing docker-as-a-service we used k8s for both internal services as well as user deployments. The latter lead to master deadlocks and all sorts of SRE nightmares (literally)…
[flagged]
Please don't cross into being a jerk on HN. You may not feel you owe the person you're speaking with better, but you owe this community better if you're participating in it.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
Re: Reverse engineering GitHub Actions cache to make it fast
#30Earlier quoted context omitted.
I think iptables compiles BPF filters; you could write your own thing to compile BPF filters. In general, the whole Linux userspace interface (with few exceptions) is considered stable; if you go below any given userspace tool, you're likely to find a more stable, but less well documented, kernel interface. Since it's all OSS, you can even use iptables itself as a starting point to build your own thing.
Nowadays you would use nftables, which like most new-ish kernel infra uses netlink as an API, and supports at least atomic updates of multiple rules. That's not to say there's documentation for that; there isn't.
I spent a decade and a bit away from Linux programming and have recently come back to it, and I'm absolutely blown away at how poor the documentation has become.
Back in the day, one of the best things about Linux was actually how good the docs were. Comprehensive man pages, stable POSIX standards, projects and APIs that have been used since 1970 so every little quirk has been documented inside out.
Now it seems like the entire OS has been rewritten by freedesktop and if I'm lucky I might find some two year out of date information on the ArchLinux wiki. If I'm even luckier, that behaviour won't have been completely broken by a commit from @poettering in a minor point release.
I actually think a lot of the new stuff is really fantastic once I reverse engineer it enough to understand what it's doing. I will defend to the death that systemd is, in principle, a lot better than the adhoc mountain of distro-specific shell scripts it replaces. Pulseaudio does a lot of important things that weren't possible before, etc. But honestly it feels like nobody wants to write any docs because it's changing too frequently, but then everything just constantly breaks because it turns out changing complex systems rapidly without any documentation leads to weird bugs that nobody understands.