For anyone looking for fast ci, I'd highly recommend webapp.io. it automatically caches layers if it detects it doesn't need to be run, and you can split a VM to shard test execution. Runs way faster than GitHub actions, you can ssh into any layer to debug, and it's just like a Linux machine/dockerfiles so no crazy syntax/mental model to learn. Support is great and we've been super happy with the experience. Best CI platform I've used by far.
We built the fastest CI and it failed
31–40 of 301 posts
Re: We built the fastest CI and it failed
#32At least for me, whenever I have to set up CI or touch CI, it's incredibly painful. I usually end up having to push a bunch of commits to debug stuff, and there's a lot of configuration hell. I don't know if Earthly solves this, perhaps it does. But if I were considering moving onto a new CI, I'd immediately think about having to redo this painful process. And that's at the team level. At the org level, switching CI…
CI systems are important to me, but my requirement is pretty much: Can it run a bash script. Every CI I've ever used has degraded into a bash script runner. It starts out well enough, you try to use all the build in features but at some point you fallback to running a shells script and then slowly more and more move to the script, because it's easier. Someone should just build a bash CI, something that can manage sec…
In a lot of ways that's pretty much what Earthly is, except those bash scripts run inside containers (that you can configure inside Earthly itself) and you pass files/artifacts between steps/containers.
Re: We built the fastest CI and it failed
#33What does fast CI even mean? CI is an overgrown shell script running your build and telling you when it fails. In general build tooling has gotten so slow that the cost of whatever CI runner relative to it should be nil. If you want fast CI you need fast tsc, clang, rustc, etc... not a faster program that calls exec on them. A bit more on topic, if you're selling CI and your business fails it's because people you're…
Perusing this article quickly, it means a CI that is automatically handling things like caching build artifacts so that you don't need to recompile your entire repository every single commit. It's not about a faster program to call exec; it's about a program that knows it need not even call exec.
Yes, exactly.
Earthly is explicit about inputs ( using COPY) and outputs of each build step. This let's it be 100% certain about whether it can cache parts of the build.
If you are familiar with docker layer-based caching and Bazel, you can imagine how Earthly works to eliminate rework in builds.
Here is a small monorepo example: https://github.com/earthly/earthly-solutions
Re: We built the fastest CI and it failed
#34Unless CI is _really_ painful, it's just not that painful. I spend a day cursing about my build when I spin up a new project, then I forget about it for months.
Specs run fast enough - especially locally where I only run failing or known-modified specs.
Re: We built the fastest CI and it failed
#35"People will buy a developer tool, but you can’t sell it."
For me the killer quote was “if the prospect requires a demo, then they’re not worth going after” — because as a prospect, if your product requires a demo, I’m already halfway to no. I am so tired of “enterprise” products that have to be purchased as a pig in a poke because they can’t stand on their own during a trial period.
Re: We built the fastest CI and it failed
#365 minutes reading the website and I still don't know how it actually works.
Marketing fluff and bits of Earthfile's ugly syntax.
Re: We built the fastest CI and it failed
#37It was honestly a bit hard for me to understand what happened from this write-up. I read it a few times, and the terms are still bit confusing. Seems like there are at least two separate issues - migration of the CI configuration -- from existing CI YAML (github/gitlab) to Earthly (Makefile/Dockerfile hybrid) - migration of the job runners -- from existing CI to a service hosted by Earthly I would have thought the fi…
It's not about migrating the syntax.
It's that people's CI over time become some kind of amalgamated model encapsulating how a firm makes every individual piece of software it makes and lands it places, think (ab)using a CI as Airflow (arbitrary automation job runner), and that the migration is first reverse engineering what people used to know, before even starting untangling all of that to express it some different way.
Nobody really wants to stop the world and sort that.
Re: We built the fastest CI and it failed
#38Earlier quoted context omitted.
> If you want fast CI you need fast tsc, clang, rustc, etc... not a faster program that calls exec on them Yes and no. Caching stuff and knowing when to run tsc/clang/rustc also improves performance.
Sure, but you have to be 100% perfect in guessing when the cache file must be rebuilt. If there is any situation at all where you use the cached file when you should have rebuilt your tool is not useful. Note that you can go too far. If you rebuild a file on linux even though the only changes were in a "#ifdef freebsd" (windows, mac...) section that is a waste of time, but not harmful. However we already have tools t…
This is a solved problem; modern build tools and workflow orchestrators face the same challenge. You declare your inputs for a given task and cache the output as long as the inputs are unchanged, as determined by their hash. So there is no wrong guessing, only an incomplete specification of inputs. It's an elegant solution, if I do say so, and has worked well at companies I've seen it used.
Re: We built the fastest CI and it failed
#39Is the concept that requiring a demo for developer tools is a negative signal that strange? CI is also notoriously expensive to migrate, which the author alluded to. The pumpkin spice latte of software engineering is disdain for things like Jira and Jenkins but what keeps them in their seat is all the custom integration work that has to be done.
The thing with Jenkins is you can also make it custom enough so you have different teams handling different parts (say the building and the deployment, or the releases). In most CIs this is a bit harder to do - I think the Azure Classic pipeline also had interesting ways to delegate responsibilities. In the end the good thing of the different CIs is they apply to different problems and one can use them all at different parts of a company - teams that work in more innovative developments vs teams that maintain products that have to be trusted and adhere to many regulations.
Re: We built the fastest CI and it failed
#40Anyone else find it strange how the author uses CI as a noun?
I've never heard anyone refer to a CI platform/system this way before and it sort of strikes me as a red flag.
Also the idea of running CI on your laptop seems questionable at best, downright irresponsible and insecure at worst.
Sure faster builds are nice, but by the time you're running your code through CI/CD its not usually after adding a few lines of for a feature, it's when the completed feature ready to merge or some sort of integration test is needed.