Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built the fastest CI and it failed

earthly.dev

Post reply on HN
Print view View original

201–210 of 301 posts

Re: We built the fastest CI and it failed

#201

Post by code_biologist »

rewmie wrote: post #179
shepherdjerred wrote: post #84

Earlier quoted context omitted.

> Caching and parallel execution are age old concepts when it comes to CI and every modern build system can do it. It is much harder to set up caching/parallelization with GitHub Actions, Jenkins, Docker, and Make than it is with just Earthly

> It is much harder to set up caching/parallelization with GitHub Actions, Jenkins, Docker, and Make than it is with just Earthly I never used Earthly but I have to call nonsense on this take. Caching and parallelization is a solved problem in GitLab, GitHub, CircleCI, etc. It's so trivial that it's pipelines 101. Caching and parallelization might be easier in Earthly but any improvement over "trivial" is something n…

This unanswered Github Actions community request for parallel steps support drives me up the wall: https://github.com/orgs/community/discussions/14484?sort=top

It can be papered over, but intelligibility by more junior engineers is one of the reasons I'd opt for GHA over something like Jenkins.

Re: We built the fastest CI and it failed

#202

Post by justin_oaks »

BoorishBears wrote: post #160
justin_oaks wrote: post #145

Earlier quoted context omitted.

I know these are examples, so I don't want to pick on you too much, but I have one nitpick: Reserving SSO for enterprise customers is awful. Please don't do it. See https://sso.tax/ for details but I'll quote this from it "SSO is a core security requirement for any company with more than five employees"

As others are pointing out, once you've grown big enough that you can't trust everyone with the keys go the kingdom: enterprise pricing is for you. SSO Tax is a fun meme, but it misses the point: Whether it's 5 people or even 3, you've scaled your business enough to hire people who aren't in the inner circle. That's a fair place to consider you an enterprise.

Not sure what you're talking about. It's not like without SSO everyone in the company has to all use the same password. Services allow people to have separate users with separate privileges.

Re: We built the fastest CI and it failed

#203

Post by paxcoder »

reactordev wrote: post #56

This is a good write up of why you shouldn’t give away the house when you open source things. The issue was really this: Earthly being open-source, Earthy Satellite users were already seeing the benefit from 95% of Earthly CI . I’m a huge fan of open source, however, if your business model includes an open source model - you need a differentiator. Beyond blazingly fast(tm). You need a reason for people to offer up th…

That is not at all the conclusion the author draws from this experience of his. Here's more of the paragraph, you quoted:

>Earthly being open-source, you could already run your own Earthly remote runner (since we use Buildkit underneath, it was essentially a remote Buildkit), connect Earthly to it, and get similar benefits to Satellites even before this was a commercial offering. And so people did this on their own, hosting it in their own environment, without us managing it for them. Once we had this packaged up in a managed offering, people were flocking to it, mainly because they did not want to manage remote runners on their own.

The author blames the failure on not validating what the were building was what clients wanted.

Re: We built the fastest CI and it failed

#204

Post by justin_oaks »

debarshri wrote: post #162
justin_oaks wrote: post #145

Earlier quoted context omitted.

I know these are examples, so I don't want to pick on you too much, but I have one nitpick: Reserving SSO for enterprise customers is awful. Please don't do it. See https://sso.tax/ for details but I'll quote this from it "SSO is a core security requirement for any company with more than five employees"

Let me give a contrarian argument. I dont think it is awful to have sso reserved only for enterprise customers. I think enterprise customer who are going through compliance will definitely have to but into that tier therefore. If you think you need SSO and but you do not have the budget for upgrading to their enterprise tier, may be you are not the right customer that they are targeting to sell. It is not you, it is…

You're right. I was thinking from the position of the consumer for which SSO is both convenient and a security feature.

I wasn't thinking as the service provider that needs a way to force the folks with the big bucks to pay up.

It sucks for the little companies, but it may be necessary for the service to do so.

Re: We built the fastest CI and it failed

#205

Post by IshKebab »

duped wrote: post #52
esafak wrote: post #38

Earlier quoted context omitted.

This is a solved problem; modern build tools and workflow orchestrators face the same challenge. You declare your inputs for a given task and cache the output as long as the inputs are unchanged, as determined by their hash. So there is no wrong guessing, only an incomplete specification of inputs. It's an elegant solution, if I do say so, and has worked well at companies I've seen it used.

It's not a solved problem, very few builds in the wild are deterministic and reproducible. You have to build up the universe from scratch (like guix and nix) for the caching to be sound enough to be reliable. Just because an algorithm exists that can help doesn't mean that the universe of software can fit cleanly into a model where that algorithm is perfect. The other fundamental problem with this model is that it ca…

It is a solved problem - you need to use a build system with sandboxing like Bazel, and also vendor all your tools.

Almost nobody does that because very few people realise it is the solution and even fewer realise they're going to need that before they set up their build system.

Re: We built the fastest CI and it failed

#206

Post by smashed »

reactordev wrote: post #136
prometheon1 wrote: post #131

Earlier quoted context omitted.

I thought using your own runners for free in Gitlab was already an option before Github Actions even existed

When it was an open source project, yes. You are correct. Ever since GitLab went commercial, it's been behind a paywall. Apparently you can run runners on self-managed CE so I'm going to look into that.

On Gitlab.com you need to input a valid credit card once in order to validate your account, but you can still get 400 minutes on shared runners for free.

This is used to combat abuse and crypto miners, they don't actually charge the card.

You can also connect your own private runner to either the saas or self-hosted gitlab-ce for free, you just have to provide your own runner.

Re: We built the fastest CI and it failed

#207

Post by Klugistiono »

I'm quite passioned about CI/CD and working in this field for the last 7 years.

I'm keeping a close eye on jenkins, tekton, circleci, travis, github actions, gitlab etc. and never ever heard about earthly.

Btw. one thing the person doesn't get: Everyone wants to have fast and reliable builds but no one cares.

No one.

CI/CD Systems in a developers eye are blackboxes. They don't care.

And its already a solved problem somehow. I was waiting for Jenkins X to happen (something like jenkins working nicely in a cloud native way because Jenkins Plugins!) but that didn't happen.

What happend instead is Github Actions. Holy shit are Github Actions good. Srsly.

GitLab stuff was slow and effort. GitHub Actions just do what they should do 99% of the time.

If you do not have a huge build, you will not care about your CI/CD System and everyone who has real build issues, has to solve it by another strategy than just switching the build system.

Re: We built the fastest CI and it failed

#208

Post by ljm »

dpeckett wrote: post #86
reactordev wrote: post #56

This is a good write up of why you shouldn’t give away the house when you open source things. The issue was really this: Earthly being open-source, Earthy Satellite users were already seeing the benefit from 95% of Earthly CI . I’m a huge fan of open source, however, if your business model includes an open source model - you need a differentiator. Beyond blazingly fast(tm). You need a reason for people to offer up th…

I'm not sure feature withholding has traditionally worked out well in the developer space. The churn rate for tooling is so high that if Earthly were to intentionally cripple their product I'm sure the majority of developers would just move on to other inferior, but free, options (eg. taskfile). Maybe those that remain could be converted successfully but it's quite a gamble. A great turn around story in recent times…

The open source version has to be good and valuable enough for you to bring it in as a dependency in a basic setup, and then the paid versions provide support for larger scale setups where the requirements are a bit different.

There are many ways to do that, depending on what you're offering. Limits and quotas are easy enough.

Re: We built the fastest CI and it failed

#209

Post by sanderjd »

dpeckett wrote: post #189
sanderjd wrote: post #170

Earlier quoted context omitted.

I know this is controversial, but I wish commercial "source available" were the more mainstream / not-shat-upon approach to services, rather than "open core". I find it very frustrating to not be able to read the code or contribute bug fixes or self-host anything outside the "open core", and I have no need for any of the things that distinguish "open source" licenses from "source available" ones.

Stallman was right. I think it's inevitable we'll see the next generation of developer focused tooling go in the direction of AGPL or source available (eg. BSL). Ironically enough Earthly started off as BSL.

What did Stallman say about this?

Re: We built the fastest CI and it failed

#210

Post by howinteresting »

shepherdjerred wrote: post #200
rewmie wrote: post #179

Earlier quoted context omitted.

> It is much harder to set up caching/parallelization with GitHub Actions, Jenkins, Docker, and Make than it is with just Earthly I never used Earthly but I have to call nonsense on this take. Caching and parallelization is a solved problem in GitLab, GitHub, CircleCI, etc. It's so trivial that it's pipelines 101. Caching and parallelization might be easier in Earthly but any improvement over "trivial" is something n…

How do you usually do caching for complex, multi-language projects in this case? How do you guarantee _only_ changed targets are rebuilt when dependencies, like a language version or external library, change? With Earthly you just... write your Earthfile. You get everything for free. Caching and parallelization _also_ works locally, so I see that speedup in development.

I think there is certainly some value to that (the other posts don't appear to be fully grasping the complexity of the problem), but Earthly feels closer to an open source consultancy project than venture capital funded startup.
Post reply on HN