Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built the fastest CI and it failed

earthly.dev

Post reply on HN
Print view View original

51–60 of 301 posts

Re: We built the fastest CI and it failed

#51

Post by danudey »

Terretta wrote: post #37
chubot wrote: post #20

It was honestly a bit hard for me to understand what happened from this write-up. I read it a few times, and the terms are still bit confusing. Seems like there are at least two separate issues - migration of the CI configuration -- from existing CI YAML (github/gitlab) to Earthly (Makefile/Dockerfile hybrid) - migration of the job runners -- from existing CI to a service hosted by Earthly I would have thought the fi…

It's not about migrating the syntax. It's that people's CI over time become some kind of amalgamated model encapsulating how a firm makes every individual piece of software it makes and lands it places, think (ab)using a CI as Airflow (arbitrary automation job runner), and that the migration is first reverse engineering what people used to know, before even starting untangling all of that to express it some different…

I started work in December at an awesome company as a build/release engineer. Our current release workflow for our open-source components:

1. Run the CI build/test pipeline 2. If it passes, run the 'Publish' promotion (manually, on any pipeline you want) 3. The pipeline runs a make target 4. The make target starts up a VM (from the VM it's running on) 5. The make target SSH'es to the VM to run a make target 4. The make target builds a binary 5. The binary runs more make targets 6. The make targets run various scripts in docker containers on the VM the other VM started 7. The scripts in the docker containers build the components 8. The binary from step five runs more make targets to publish the releases 9. We test everything with another promotion which runs make targets which build docker containers to run python scripts (pytest)

This is also built by a complicated web of wildcarded makefile targets, which need to be interoperable and support a few if/else cases for specific components.

My plan is to migrate all of this to something simpler and more straightforward, or at least more maintainable, which is honestly probably going to turn into taskfile[0] instead of makefiles, and then simple python scripts for the glue that ties everything together or does more complex logic.

My hope is that it can be more straightforward and easier to maintain, with more component-ized logic, but realistically every step in that labyrinthine build process (and that's just the open-source version!) came from a decision made by a very talented team of engineers who know far more about the process and the product than I do. At this point I'm wondering if it would make 'more sense' to replace it with a giant python script of some kind and get access to all the logic we need all at once (it would not).

[0] https://taskfile.dev/

Re: We built the fastest CI and it failed

#52

Post by duped »

esafak wrote: post #38
bluGill wrote: post #25

Earlier quoted context omitted.

Sure, but you have to be 100% perfect in guessing when the cache file must be rebuilt. If there is any situation at all where you use the cached file when you should have rebuilt your tool is not useful. Note that you can go too far. If you rebuild a file on linux even though the only changes were in a "#ifdef freebsd" (windows, mac...) section that is a waste of time, but not harmful. However we already have tools t…

This is a solved problem; modern build tools and workflow orchestrators face the same challenge. You declare your inputs for a given task and cache the output as long as the inputs are unchanged, as determined by their hash. So there is no wrong guessing, only an incomplete specification of inputs. It's an elegant solution, if I do say so, and has worked well at companies I've seen it used.

It's not a solved problem, very few builds in the wild are deterministic and reproducible. You have to build up the universe from scratch (like guix and nix) for the caching to be sound enough to be reliable.

Just because an algorithm exists that can help doesn't mean that the universe of software can fit cleanly into a model where that algorithm is perfect.

The other fundamental problem with this model is that it cannot model cyclical dependencies (a degenerate case: glibc needs python which needs glibc). Since you can't hash the output until you've hashed the input, and the input hash depends on the output hash you need a way to break the dependency cycle. Tools have varying ways of mitigating this but it's not exactly "solved" in an elegant sense.

And finally, a lot of build steps are faster than hashing their input and outputs. So if you want the utmost performance you need heuristics or a model that allows for some things to be hashed and others to always be considered fresh.

Re: We built the fastest CI and it failed

#53

Post by Guvante »

pshirshov wrote: post #16

Judging by the description, that project seems to be another DAG-planning meta-program with input-based memoization. This is the superior approach to fight complexity but, unfortunately, most engineers just don't understand the problem and the solution.

It can be superior and not valuable to customers.

People who don't have an existing build system don't have money.

People who have an existing build system would need to migrate which can be super painful.

And nothing in the article points to making it easier.

Re: We built the fastest CI and it failed

#54

Post by bastardoperator »

It failed because the marketing is outright bunk and overly dishonest.

If I compile with Jenkins, Actions or Earthly, that compile time is going to be the same under each build system assuming the same build node. Claiming you're 20x faster when CI is firing within seconds is kind of meaningless. Caching and parallel execution are age old concepts when it comes to CI and every modern build system can do it.

CI is all about feedback, I didn't see much here in terms of collaboration and bubbling data up, I didn't look very hard, but this should be frontline and center. Lastly, I'm not interested in adopting a DSL for my builds ever again, sorry.

Re: We built the fastest CI and it failed

#55

Post by wpietri »

aftbit wrote: post #41
Terretta wrote: post #37

Earlier quoted context omitted.

It's not about migrating the syntax. It's that people's CI over time become some kind of amalgamated model encapsulating how a firm makes every individual piece of software it makes and lands it places, think (ab)using a CI as Airflow (arbitrary automation job runner), and that the migration is first reverse engineering what people used to know, before even starting untangling all of that to express it some different…

I think the key is that nobody "wants" to spend any time at all on CI or their build system. It is one of those necessary evils that you have to do to have functioning software. Once you do it, even if it's a slow buggy pile of hacks, you will just ignore it until the pain gets very very bad. This was the same problem that caused our team to abandon CircleCI. They kept wanting us to rewrite our CI configs, touting va…

Some people do want to spend time on that! It depends whether you see fast feedback as a competitive advantage.

I once worked on a system where we had excellent test coverage and all our unit tests ran in under a minute. We worked hard to keep it that way, because it was a huge boon to productivity, especially when improving design issues. It also enabled frequent commits and merges, which made collaboration easier. It was a great experience.

But yes, from the business perspective, the leading edge is a pretty small audience. (Circa 15%, per Moore.) And the bottom half of the audience by competence, which Moore called the Late Majority and the Laggards, can be well more than half your revenue in a SaaS context because they will create problems for themselves (like slow/buggy builds) that mean higher per-user revenue. And they may also have special feature needs where they'll pay more rather than clean up their messes.

In that context, doing the right thing for your business can be very different than doing the right thing for users. Sort of like a physical therapist who makes most of their money from renting crutches and in-home meal delivery, and so doesn't push people to actually get better.

Re: We built the fastest CI and it failed

#56

Post by reactordev »

This is a good write up of why you shouldn’t give away the house when you open source things. The issue was really this: Earthly being open-source, Earthy Satellite users were already seeing the benefit from 95% of Earthly CI.

I’m a huge fan of open source, however, if your business model includes an open source model - you need a differentiator. Beyond blazingly fast(tm). You need a reason for people to offer up their credit cards, or better, their accounting PO process. You need a feature you withhold or a service you provide from the open source offering to make it appealing.

GitLab restricts CI/CD to paying customers. Travis/CircleCI limit on build time/credits unless also open source. Azure DevOps is the devil. ArgoCD is complicated. GitHub actions are nice if you have hardware to run the runners. Swarm’s of Jenkins is what most are used to in the enterprise.

As a former Director of DevOps, my immediate question will be what features make me want to buy vs host myself. I’m technically capable. Why should I buy with you when I can run my cloud and my DevOps pipelines for my workforce the way that makes the most sense for my business? Help me make the most sense.

*EDIT* I'm not suggesting withholding features that are critical for the software to operate, but withholding business features that provide support or enterprise-level integration. Even making it tiered so the power users pay less because they support it more themselves?

Re: We built the fastest CI and it failed

#58

Post by seanr88 »

I'm having a very similar issue getting https://brisktest.com/ out there. People complain about things like the speed of their build system, but convincing people that your CI system is faster just seems like an uphill battle. Maybe the era of launching new indie dev tools to developers is over and you just need a massive marketing budget to bombard devs with advertising and lots of sponsored content.

Re: We built the fastest CI and it failed

#59

Post by dgb23 »

jcranmer wrote: post #8
alexvitkov wrote: post #4

What does fast CI even mean? CI is an overgrown shell script running your build and telling you when it fails. In general build tooling has gotten so slow that the cost of whatever CI runner relative to it should be nil. If you want fast CI you need fast tsc, clang, rustc, etc... not a faster program that calls exec on them. A bit more on topic, if you're selling CI and your business fails it's because people you're…

Perusing this article quickly, it means a CI that is automatically handling things like caching build artifacts so that you don't need to recompile your entire repository every single commit. It's not about a faster program to call exec; it's about a program that knows it need not even call exec.

Isn't this a complex solution for a problem that shouldn't exist?

Re: We built the fastest CI and it failed

#60

Post by jatins »

> We ended up with the most ridiculous negative qualification criteria I have ever heard of: if the prospect requires a demo, then they’re not worth going after

Based on my experience at enterprise companies, this is _all_ enterprise sales. So sounds very counter intuitive to me.

Post reply on HN