I'm currently using blacksmith for my arm64 Docker builds. Unfortunately my workflow currently requires invoking a custom bash script which executes the Docker commands. Does this mean, I can now utilize Docker image caching without needing to migrate to useblacksmith/build-push-action?
Reverse engineering GitHub Actions cache to make it fast
31–39 of 39 posts
Re: Reverse engineering GitHub Actions cache to make it fast
#32Earlier quoted context omitted.
Shame, I am not sure how others use actions but I like really small granular ones to make it easy to see what's wrong at a glance. E.g. formatting checks per language per project in my monorepo etc. Each check is like 10 seconds and I have like 70+, so the per minute is biting me ATM.
Come check out RWX :). We have per second billing but I think it won't even matter for your use case because most of those checks are going to take 0s on RWX. And our UI is optimized for showing you what is wrong at a glance without having to look at logs at all.
Sorry not for me
* Your per min billing is double blacksmith's * RWX is a proprietary format? vs blacksmith's one line change. * No fallback option, blacksmith goes down and I can revert back to GitHub temporarily.
Re: Reverse engineering GitHub Actions cache to make it fast
#33I am struggling with justification for CI/CD pipelines that are so complex this kind of additional tooling becomes necessary. There are ways to refactor your technology so that you don't have to suffer so much at integration and deployment time. For example, the use of containers and hosted SQL where neither are required can instantly 10x+ the complexity of deploying your software. The last few B2B/SaaS projects I wo…
> The last few B2B/SaaS projects I worked on had CI/CD built into the actual product. Writing a simple console app that polls SCM for commits, runs dotnet build and then performs a filesystem operation is approximately all we've ever needed. The only additional enhancement was zipping the artifacts to an S3 bucket so that we could email the link out to the customer's IT team for install in their secure on-prem instances.
That sounds like the biggest yikes.
Re: Reverse engineering GitHub Actions cache to make it fast
#34Earlier quoted context omitted.
Here is my cynical take in ci. Firstly, testing is almost never valued by management which would rather close a deal on half finished promises than actually build a polished, reliable product (they can always scapegoat the eng team if things go wrong with the customer anyway). So, to begin with, testing is rarely prioritized. But most developer orgs eventually realize that centralized testing is necessary or else eve…
> they typically are going to sell you one of two things or both: support or cpu time At what point does the line between CPU time in GH Actions and CPU time in the actual production environment lose all meaning? Why even bother moving to production? You could just create a new GH action called "Production" that gets invoked at the end of the pipeline and runs perpetually. I think I may have identified a better canar…
> I think I may have identified a better canary here. If the CI/CD process takes so much CPU time that we are consciously aware of the resulting bill, there is definitely something going wrong.
CPU time is cheaper than an engineers time, you should be offloading formatting/linting/testing checks to CI on PRs. This will add up though when multiple by hundreds or thousands, it isn't a good canary.
Re: Reverse engineering GitHub Actions cache to make it fast
#35I'm currently using blacksmith for my arm64 Docker builds. Unfortunately my workflow currently requires invoking a custom bash script which executes the Docker commands. Does this mean, I can now utilize Docker image caching without needing to migrate to useblacksmith/build-push-action?
Yes! This is documented in our docs: https://docs.blacksmith.sh/blacksmith-caching/docker-builds#..., the TLDR is that you can use the `build-push-action` with `setup-only: true`.
Re: Reverse engineering GitHub Actions cache to make it fast
#36I'm currently migrating some stuff from azdo to GHA, and have been putting past lessons to serious use: * Perf: don't use "install X" (Node, .Net, Ruby, Python, etc.) tasks. Create a container image with all your deps and use that instead. * Perf: related to the last, keep multiple utility container images around of varying degrees of complexity. For example, in our case, I decided on PowerShell because we have some…
I've seen this pattern quite a bit with our users, and we've implemented "snapshots" for this.
It essentially makes a copy of your entire state of the runner, which you can then use in subsequent runs, with no concurrency limits.
This essentially automates out a lot of the work you're doing to make the jobs faster.
https://docs.warpbuild.com/snapshot-runners
We provide a product similar to Blacksmith.
Re: Reverse engineering GitHub Actions cache to make it fast
#37I'm currently using blacksmith for my arm64 Docker builds. Unfortunately my workflow currently requires invoking a custom bash script which executes the Docker commands. Does this mean, I can now utilize Docker image caching without needing to migrate to useblacksmith/build-push-action?
Founder of WarpBuild here.
Our arm64 runners are ~2x faster than Blacksmith. This is because we use a newer generation of more powerful instances.
If you're building multi arch images, here is a page that may be useful: https://docs.warpbuild.com/docker-builders#multi-platform-bu...
We natively support multi platform builds in our drop in replacement action.
Re: Reverse engineering GitHub Actions cache to make it fast
#38I am struggling with justification for CI/CD pipelines that are so complex this kind of additional tooling becomes necessary. There are ways to refactor your technology so that you don't have to suffer so much at integration and deployment time. For example, the use of containers and hosted SQL where neither are required can instantly 10x+ the complexity of deploying your software. The last few B2B/SaaS projects I wo…
I invested a few days at the start of building my B2B SaaS company wherein on every deploy we automate branching our actual staging/production databases with Neon, spinning up a complete preview environment with all the real dependencies, applying all migrations to the real data, and running end to end tests on that environment. Once it was set up I almost never need to touch it (and it's documented, but my engineers also don't need to touch it either).
In a few years of that setup being in place, while it's only partially attributable to this CI/CD process, we have never had a single incident—a stark contrast to the volatile on-call experiences I've had at past startups. For me, never being woken up in the night or needing to stress out while reverting deploys that broke our users needs was well worth it.
Re: Reverse engineering GitHub Actions cache to make it fast
#39I'm currently using blacksmith for my arm64 Docker builds. Unfortunately my workflow currently requires invoking a custom bash script which executes the Docker commands. Does this mean, I can now utilize Docker image caching without needing to migrate to useblacksmith/build-push-action?
Founder of WarpBuild here. Our arm64 runners are ~2x faster than Blacksmith. This is because we use a newer generation of more powerful instances. If you're building multi arch images, here is a page that may be useful: https://docs.warpbuild.com/docker-builders#multi-platform-bu... We natively support multi platform builds in our drop in replacement action.
It's poor form in my opinion to come and solicit business and bash your competitors here on HN. Just a tip.