Reverse engineering GitHub Actions cache to make it fast
1–10 of 39 posts
Re: Reverse engineering GitHub Actions cache to make it fast
#2Anything for artifacts perhaps? ;) We use external runners(not blacksmith) and had to work around this manually. https://github.com/actions/download-artifact/issues/362#issu...
Re: Reverse engineering GitHub Actions cache to make it fast
#3Anything for artifacts perhaps? ;) We use external runners(not blacksmith) and had to work around this manually. https://github.com/actions/download-artifact/issues/362#issu...
[cofounder of blacksmith here]
This is on our radar! The primitives mentioned in this blog post are fairly general and allow us to support various types of artifact storage and caching protocols.
Re: Reverse engineering GitHub Actions cache to make it fast
#4Fun to see folks replicating what we’ve done with Depot for GitHub Actions [0]. Going as far as using a similar title :)
Forking the ecosystem of actions to plug in your cache backed isn’t a good long term solution.
Re: Reverse engineering GitHub Actions cache to make it fast
#5Anything for artifacts perhaps? ;) We use external runners(not blacksmith) and had to work around this manually. https://github.com/actions/download-artifact/issues/362#issu...
Also might be worth checking out our sticky disks for your use-case: https://github.com/useblacksmith/stickydisk. It can be a good option for persisting artifacts across jobs - especially when they're larger
Re: Reverse engineering GitHub Actions cache to make it fast
#6Anything for artifacts perhaps? ;) We use external runners(not blacksmith) and had to work around this manually. https://github.com/actions/download-artifact/issues/362#issu...
[cofounder of blacksmith here] This is on our radar! The primitives mentioned in this blog post are fairly general and allow us to support various types of artifact storage and caching protocols.
Small blacksmith.sh user here, have yous any plans on reducing billing from per minute to something smaller like per second?
Re: Reverse engineering GitHub Actions cache to make it fast
#7Earlier quoted context omitted.
[cofounder of blacksmith here] This is on our radar! The primitives mentioned in this blog post are fairly general and allow us to support various types of artifact storage and caching protocols.
Small blacksmith.sh user here, have yous any plans on reducing billing from per minute to something smaller like per second?
Hi! no plans for more resolution than per minute currently
Re: Reverse engineering GitHub Actions cache to make it fast
#8> iptables was already doing heavy lifting for other subsystems inside our environment, and with each VM adding or removing its own set of rules, things got messy fast, and extremely flakey
We saw the same thing at Vercel. Back when we were still doing docker-as-a-service we used k8s for both internal services as well as user deployments. The latter lead to master deadlocks and all sorts of SRE nightmares (literally).
So I was tasked to write a service scheduler from scratch that replaced k8s. When it got to the manhandling of IP address allocations, deep into the rabbit hole, we had already written our own redis-backed DHCP implementation and needed to insert those IPs into the firewall tables ourselves, since Docker couldn't really do much at all concurrently.
Iptables was VERY fragile. Aside from the fact it didn't even have a stable programmatic interface, it was also a race condition nightmare, rules were strictly ordered, had no composition or destruction-free system (name spacing, layering, etc), and was just all around the worst tool for the job.
Unfortunately not much else existed at the time, and given that we didn't have time to spend on implementing our own kernel modules for this system, and that Docker itself had a slew of ridiculous behavior, we ended up scratching the project.
Learned a lot though! We were almost done, until we weren't :)
Re: Reverse engineering GitHub Actions cache to make it fast
#9It's pretty amazing to see what Blacksmith, Depot, Actuated, etc. have been able to build on top of GitHub Actions. At RWX we got a bit tired of constantly trying to work around the limitations of the platform with self-hosted runners, so we just built an entirely new CI platform on a brand new execution model with support for things like lightning-fast caching out of the box. Plus, there are some fundamental limitations that are impossible to work around, like the retry behavior [0]. Still, I have a huge appreciation for the patience of the Blacksmith team to actually dig in and improve what they can with GHA.
[0] https://www.rwx.com/blog/retry-failures-while-run-in-progres...
Re: Reverse engineering GitHub Actions cache to make it fast
#10> iptables was already doing heavy lifting for other subsystems inside our environment, and with each VM adding or removing its own set of rules, things got messy fast, and extremely flakey We saw the same thing at Vercel. Back when we were still doing docker-as-a-service we used k8s for both internal services as well as user deployments. The latter lead to master deadlocks and all sorts of SRE nightmares (literally)…
yeah our findings were similar. the issues we saw with iptables rules, especially at scale with ephemeral workloads, was starting to cause us a lot of operational toil. nftables ftw