Live data from Hacker News

We built the fastest CI and it failed

earthly.dev

171–180 of 301 posts

Re: We built the fastest CI and it failed

#171

Earlier quoted context omitted.

AzDO is "decent", depending on your needs, but it's in an unfortunate zombie state where Microsoft is supporting it just enough to keep it stable and keep certain enterprises happy and consistently insisting that AzDO has a roadmap and is still beloved, but it is very clear that all of the actual resources are going to the GitHub side of the house in 2023. (One of the most recent signs of this crazy zombie state that…

This. AzDO will die. It will be replaced entirely by Github. It originally was only designed as a way to get people onto Azure using their already-existing MS partnership to give it away for free. The only purpose, drive adoption of Azure Cloud. In runners, resources, stickiness, and preventing abandoning ship.

Azure Devops existed well before GitHub was owned by MS under Visual Studio Online moniker.

It was intended to be a SAAS version of the on Prem offerings MS has provided since before GitHub even existed.

Re: We built the fastest CI and it failed

#172
post #147

This is a good write up of why you shouldn’t give away the house when you open source things. The issue was really this: Earthly being open-source, Earthy Satellite users were already seeing the benefit from 95% of Earthly CI . I’m a huge fan of open source, however, if your business model includes an open source model - you need a differentiator. Beyond blazingly fast(tm). You need a reason for people to offer up th…

From my experience hosting our own instance of Gitlab + few CI/CD workers was few weeks of initial work (integrating with company's LDAP etc.) then around one ops-week per year worth of maintenance over last few years (50+ developers, dozens and dozens of projects). Before that we had just "a git server + jenkins + some slaves" that took even less maintenance (althought I'd imagine more fumbling on dev side with job…

You just described a great starting point for a small shop, I would recommend this route 100%.

Initially your CI/CD needs to be close to the code. It needs to be close to developers. Not a DevOps department. It needs to be almost forgettable. Include a config in your repo, bam, welcome to the pipeline.

Where this breaks down is when you don’t have LDAP, you have Azure IDP + Amazon’s Cognito + Keycloak 3rd Parties + SAML (barf) + Form Login for contractors + JWT/OAuth for services.

This is the inflection point where you need DevSecOps + Architecture + Cloud Engineers to ensure everyone’s playing nicely. Enjoy your pods. Let us know when you need that VPN tunnel to your customers backend because they don’t trust SSL.

Re: We built the fastest CI and it failed

#173
post #156

Earlier quoted context omitted.

I'm not sure feature withholding has traditionally worked out well in the developer space. The churn rate for tooling is so high that if Earthly were to intentionally cripple their product I'm sure the majority of developers would just move on to other inferior, but free, options (eg. taskfile). Maybe those that remain could be converted successfully but it's quite a gamble. A great turn around story in recent times…

In this particular case, developers could also move on to a superior and completely open alternative (Nix), so making even a semi-closed alternative with much fewer people contributing etc. was already a hard sell.

Seriously, is there an email distro or something to notify Nix users when there's a new post to go troll? It's not suitable for 90%+ of users, which seems a far cry from superior.

Re: We built the fastest CI and it failed

#174

I've used Earthly for a year and absolutely love it. It allows you to have two dependencies on the host system for any project -- just Docker and Earthly. I've used it to build cross-compilers, system software, web apps, my resume, school projects. The speed up is real. You can definitely get caching and parallelization and containizeration with other build systems, but Earthly's value is that it comes for free with…

Ditto, it's been a huge success where I work. Keeping my fingers crossed that they find a way to stay in business :)

I feel like it's just a patience game of staying alive long enough that organic traction takes off. Earthly is bloody fantastic, the flexibility and expressiveness IMO is hard to beat.

Re: We built the fastest CI and it failed

#175
post #156

Earlier quoted context omitted.

In this particular case, developers could also move on to a superior and completely open alternative (Nix), so making even a semi-closed alternative with much fewer people contributing etc. was already a hard sell.

Seriously, is there an email distro or something to notify Nix users when there's a new post to go troll? It's not suitable for 90%+ of users, which seems a far cry from superior.

It's just very noticeable when half of what the industry seems to spend its time on is solved problems. You and all the others have to learn why touching the hot stove is a bad idea the same way most children do. It's frustrating to observe.

It's fine to criticise Nix for its various flaws, but any suggestion of an "alternative" that has learned nothing from it is just a collective waste of time.

Re: We built the fastest CI and it failed

#176

> Imagine you live in a world where no part of the build has to repeat unless the changes actually impacted it. A world in which all builds happened with automatic parallelism. A world in which you could reproduce very reliably any part of the build on your laptop. That sounds similar to https://concourse-ci.org/ I quite like it, but it never seemed to gain traction outside of Cloud Foundry.

We spent some time evaluating this on my team, and we're still experimenting with it.

I like it a lot, but the project appears to be mostly unmaintained since mid-2021, when the creator left it to work on a lispy CI/CD tool [0] that feels very complicated... not sure what's going on there.

[0]: https://github.com/vito/bass

Re: We built the fastest CI and it failed

#177
Now is a good time to read "Crossing the Chasm:" https://en.wikipedia.org/wiki/Crossing_the_Chasm

The book talks about the state of the business Earthly CI was in: They tried to go from enthusiasts to generalists... But with a completely new product!

I don't think it was wrong to build a new CI; instead, I think the thing they did "wrong" was build a super-complicated CI; or that when they tried to cross the chasm, they tried to sell a (cough) completely different product than what their early users were using.

Re: We built the fastest CI and it failed

#178
post #51

Earlier quoted context omitted.

It's not about migrating the syntax. It's that people's CI over time become some kind of amalgamated model encapsulating how a firm makes every individual piece of software it makes and lands it places, think (ab)using a CI as Airflow (arbitrary automation job runner), and that the migration is first reverse engineering what people used to know, before even starting untangling all of that to express it some different…

I started work in December at an awesome company as a build/release engineer. Our current release workflow for our open-source components: 1. Run the CI build/test pipeline 2. If it passes, run the 'Publish' promotion (manually, on any pipeline you want) 3. The pipeline runs a make target 4. The make target starts up a VM (from the VM it's running on) 5. The make target SSH'es to the VM to run a make target 4. The ma…

Taskfile looks worse than make to me in every possible dimension (except maybe windows compatibility, but I don't believe that it meaningfully supports that).

You should at least read and understand the paper "Recursive Make Considered Harmful" before attempting to replace make with something "better":

https://aegis.sourceforge.net/auug97.pdf

Most people use make incorrectly (and it sounds like the system you describe makes most of the classic mistakes). The paper I linked explains how to use Make's templating language to help it scale to complicated setups.

Here are a few critiques of taskfile (from skimming their documentation):

- The syntax is incredibly verbose and weird. For instance, you have to write a tree of yaml nodes to loop over a set of files. Make's syntax is weird, but at least it is not verbose.

- The documentation makes no mention that I could find of deriving a set of tasks from a list of files (Make's $(wildcard) macro, or stemming rules like "%.o : %.c"). So, I guess the taskfile will end up being huge (like the size of autogenerated makefiles from autoconf or cmake), or you'll end up using additional ad-hoc build systems that are invoked by your taskfile.

- From the documentation, they haven't thought through having more than one project per unix account "When running your global Taskfile with -g, tasks will run on $HOME by default, and not on your working directory!"

- They seem to have partially reimplemented the bad ideas from recursive make: https://taskfile.dev/usage/#including-other-taskfiles

- I suggest taking an idiomatic but non-trivial bash script (which is 99% of what make is invoking, and that taskfiles support), and then trying to port it to python directly. There's usually a 10-100x line of code blowup from doing that, and the python thing usually bitrots every year or so (vs. every few decades for shell).

Re: We built the fastest CI and it failed

#179

It failed because the marketing is outright bunk and overly dishonest. If I compile with Jenkins, Actions or Earthly, that compile time is going to be the same under each build system assuming the same build node. Claiming you're 20x faster when CI is firing within seconds is kind of meaningless. Caching and parallel execution are age old concepts when it comes to CI and every modern build system can do it. CI is all…

> Caching and parallel execution are age old concepts when it comes to CI and every modern build system can do it. It is much harder to set up caching/parallelization with GitHub Actions, Jenkins, Docker, and Make than it is with just Earthly

> It is much harder to set up caching/parallelization with GitHub Actions, Jenkins, Docker, and Make than it is with just Earthly

I never used Earthly but I have to call nonsense on this take.

Caching and parallelization is a solved problem in GitLab, GitHub, CircleCI, etc. It's so trivial that it's pipelines 101.

Caching and parallelization might be easier in Earthly but any improvement over "trivial" is something not worth wasting time over.

Re: We built the fastest CI and it failed

#180

This is a good write up of why you shouldn’t give away the house when you open source things. The issue was really this: Earthly being open-source, Earthy Satellite users were already seeing the benefit from 95% of Earthly CI . I’m a huge fan of open source, however, if your business model includes an open source model - you need a differentiator. Beyond blazingly fast(tm). You need a reason for people to offer up th…

I'm not sure feature withholding has traditionally worked out well in the developer space. The churn rate for tooling is so high that if Earthly were to intentionally cripple their product I'm sure the majority of developers would just move on to other inferior, but free, options (eg. taskfile). Maybe those that remain could be converted successfully but it's quite a gamble. A great turn around story in recent times…

> I'm not sure feature withholding has traditionally worked out well in the developer space.

It’s worked well for GitHub, Slack, and pretty much every service that only offer SSO behind enterprise subscriptions.

The trick is to offer enough to make people productive but not so much that there isn’t a worthwhile upgrade path.

Post reply on HN