Live data from Hacker News

We built the fastest CI and it failed

earthly.dev

121–130 of 301 posts

Re: We built the fastest CI and it failed

#121

Earlier quoted context omitted.

Incremental builds are something even enterprise build systems fail at. I've had Visual Studio and MSBuild fail on me multiple times by deciding not to recompile a certain .cpp file, because they got the dependency graph wrong. This results in a running but inconsistent executable that's a bitch to debug. If you're selling me another tool that duplicates this work, but that has even less context on my project than th…

Bazel does this. I have to be honest - MS Build does not scale well. Companies use it because it’s the default.

Seems they came up with a way to scale up build toolchains with BuildXL[0]

Doesn't seem fully baked yet though

[0]: https://github.com/microsoft/BuildXL

Re: We built the fastest CI and it failed

#122

What does fast CI even mean? CI is an overgrown shell script running your build and telling you when it fails. In general build tooling has gotten so slow that the cost of whatever CI runner relative to it should be nil. If you want fast CI you need fast tsc, clang, rustc, etc... not a faster program that calls exec on them. A bit more on topic, if you're selling CI and your business fails it's because people you're…

The tricky part of CI is figuring out what work doesn't need to be done. That is how you save time.

Re: We built the fastest CI and it failed

#123
I think the biggest moral of the story - don't build a product without considering the switching/adoption cost. Higher the switching/adoption cost, lower the product/market fit.

I have heard this story again and again from so many startups who have built products which provide incremental benefits (1.3x cheaper or 1.5x faster) and not getting enough traction.

Re: We built the fastest CI and it failed

#124

Earlier quoted context omitted.

Is there an actual technical point beyond ”MS bad” for this sentiment? Honestly asking, as I’ve been mostly hearing positive things about AzDO, and from developers of all people.

ADO makes me yearn for Jira. The boards are terrible, the UI is hard, the permissions are incomprehensible, it doesn't even integrate that nicely with Azure. On paper it does everything, it practice it can be made to do everything, but it's like getting citrus drenched sandpaper papercuts every day.

Hah, thanks for that mental image.

As I said the positivive sentiments I heard came from developers, and TFA is about CI (rather than boards etc. that JIRA handles, so is there maybe anything remotely positive in how AzDO implements CI, git integration, testing, etc? Or is it just rotten to the core?

Hard to imagine someone would hate it more than Bitbucket Pipelines or, god forbid, Bamboo. Buuut I’m more than open-minded if need be.

Re: We built the fastest CI and it failed

#125
post #61
post #19

Earlier quoted context omitted.

except not really.. their tutorial [0] includes no-compile languages like python and JS, with only non-trivial compilation being and Java... and how do they do it? RUN gradle build At least in this case, Earthly has no insight into how repository is organized, and it _will_ recompile the entire repository with every single commit. So what's the real value of it? this basically seems like a better "docker build" alter…

That's step one, 'how to write a basic Earthfile that just does the thing', so they don't delve into the details of caching and how it works, but look at the whole block and not just that one line; it operates like a Dockerfile: build: COPY build.gradle ./ COPY src src RUN gradle build RUN gradle install SAVE ARTIFACT build/install/java-example/bin /bin SAVE ARTIFACT build/install/java-example/lib /lib The COPY phase…

A CI system like TeamCity will give you that sort of thing for free though, without needing an Earthfile. It just dedupes builds at the same commit, and keeps checkout directories between builds on the agents, so if the build tool has a cache then it's automatically reused. Works great.

Given that it's hard to understand what exactly this is for? Build systems normally try to avoid redundant work for local purposes anyway. Is it some hack for companies that insist on resetting build environments to zero on every single build, and then decide they want caching back? If so, why not just ... let the builds take place in directories containing previous builds, and fix the problems? That's what we do at my company and it works great (using gradle), there are only rarely problems with unclean builds and it's easy to rerun with a clean checkout if that's suspected to be an issue (the bugs are mostly in our own code which does a layer of caching above the build system anyway).

Re: We built the fastest CI and it failed

#126

Stopped at "Imagine you live in a world where no part of the build has to repeat unless the changes actually impacted it." because I'm already in that world by using Nx and Nx Cloud.

I must admit I'm a bigger fan of the wireit[0] approach, the only pause I have is its a Google project, my temptation is to fork it. The code isn't terribly complex

My biggest complaint with NX is: lack of a sane API for plugins, and it has more overhead than I'd care for. For the amount of complexity that NX has, I'd rather use Rush[1] which gives you everything NX does. My only complaint with Rush is that its development is really slow going, they really need to focus up on Rush plugins (they're good, but still experimental, and I'd love to see them clean up how `autoinstalls` work to be more intutive)

I'm on the fence about turbo from Vercel

[0]: https://github.com/google/wireit

[1]: https://rushjs.io/

Re: We built the fastest CI and it failed

#127
This article is confusing because the author is too close to the problem, and it needs to be explained from an outsider’s point of view before diving into the details. But it looks like the information is there.

It sounds to me like they ended up with an encapsulation layer between language-specific build systems and the continuous build systems that kick them off.

A comparison might be with something like Bazel, which does everything, but full adoption requires going all-in on using their build language to replace language-specific build systems, often including moving source files to make it work. This feels foreign compared to using a language’s own build system, but it might seem natural for C programmers, where the language doesn’t have its own build system. Similarly, Java went through several unfortunate build systems before unfortunately settling on Gradle.

Language-specific build systems end up not doing everything because each language has its own conventions and ecosystem. Modern ones don’t try, they know what they’re good at and stick to it.

So, the tool that actually knows about multiple languages and their artifacts is often a shell script, makefile, docker file, or the continuous build itself. Or even doing it by hand. That’s the layer they’re trying to improve.

I don’t have a clear idea what their fundamental insight is that makes their way of doing it better, though.

Re: We built the fastest CI and it failed

#128

Stopped at "Imagine you live in a world where no part of the build has to repeat unless the changes actually impacted it." because I'm already in that world by using Nx and Nx Cloud.

I must admit I'm a bigger fan of the wireit[0] approach, the only pause I have is its a Google project, my temptation is to fork it. The code isn't terribly complex My biggest complaint with NX is: lack of a sane API for plugins, and it has more overhead than I'd care for. For the amount of complexity that NX has, I'd rather use Rush[1] which gives you everything NX does. My only complaint with Rush is that its devel…

Curious about your thoughts on turbo?

Re: We built the fastest CI and it failed

#129

Not to rain on the parade here but this is literally a copy-pasta product. Jenkins, Google Borg, Cloud Foundry, Concourse Pipelines...and surprise surprise when you look at where he came from...Ex-Google, Ex-VMW, RabbitMQ... If OP wasn't in and around the source of all these tools above then they were at the very least first cousins to the story. The sales cycle is long, integrations require multiple dimensions of ex…

> Not to rain on the parade here but this is literally a copy-pasta product.

More to the point, the product offered no compelling reason to use it, let alone pay for it.

Being "fast" is not a selling point. Developers don't want slow pipelines, but that does not mean they want fast pipelines. The speed that the pipeline works is more depending on how the pipeline is setup than the overhead of the pipeline service, and other CICD services are already blazing fast. Take for instance CircleCI, which is already a tough sell over GitHub actions and GitLab CICD: how did the service stood out? Did it added tangible value over GitHub/Gitlab/CircleCI/etc? Shaving milliseconds off a minutes-long build ain't it.

Re: We built the fastest CI and it failed

#130

This is a good write up of why you shouldn’t give away the house when you open source things. The issue was really this: Earthly being open-source, Earthy Satellite users were already seeing the benefit from 95% of Earthly CI . I’m a huge fan of open source, however, if your business model includes an open source model - you need a differentiator. Beyond blazingly fast(tm). You need a reason for people to offer up th…

I'm not sure feature withholding has traditionally worked out well in the developer space. The churn rate for tooling is so high that if Earthly were to intentionally cripple their product I'm sure the majority of developers would just move on to other inferior, but free, options (eg. taskfile). Maybe those that remain could be converted successfully but it's quite a gamble. A great turn around story in recent times…

I think a nice compromise is the CircleCI/Travis model of providing limited build credits/hours. If you’re a small shop or single dev, you can use the core product for free, with the hope that you’d convert to a paying user if your business grows past the free constraint.
Post reply on HN