Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built the fastest CI and it failed

earthly.dev

Post reply on HN
Print view View original

161–170 of 301 posts

Re: We built the fastest CI and it failed

#161

Post by verdverm »

There are a bunch of these next-gen CI / container building systems. They all pretty much build on BuildKit like Docker, but unlock way more of the potential, namely DAG layers (think multi-stage docker on steroids), and better / more granular caching. These next-gen CI tools are fighting to be the UX on top of BuildKit.

Dagger is where it is at, and it's built by the creators of Docker, based on their lessons learned. If you're looking to try something out, try Dagger. You write in the same languages you do every day, no need for some weird syntax like Earthly or the flatness of a Dockerfile. Their CLI/TUI is pretty sweet too, but you don't even need it.

Re: We built the fastest CI and it failed

#162

Post by debarshri »

justin_oaks wrote: post #145
gabereiser wrote: post #114

Earlier quoted context omitted.

plugins, hardware, hosting, support, there's a number of ways to offer a service for an open source project. Open Core models come to mind. Another is not having enterprise SSO as part of the core. Or not having clustering. Something that would make a VP think twice about adopting your open source solution vs using you as a vendor.

I know these are examples, so I don't want to pick on you too much, but I have one nitpick: Reserving SSO for enterprise customers is awful. Please don't do it. See https://sso.tax/ for details but I'll quote this from it "SSO is a core security requirement for any company with more than five employees"

Let me give a contrarian argument. I dont think it is awful to have sso reserved only for enterprise customers. I think enterprise customer who are going through compliance will definitely have to but into that tier therefore.

If you think you need SSO and but you do not have the budget for upgrading to their enterprise tier, may be you are not the right customer that they are targeting to sell. It is not you, it is them.

Re: We built the fastest CI and it failed

#163

Post by verdverm »

shepherdjerred wrote: post #84
bastardoperator wrote: post #54

It failed because the marketing is outright bunk and overly dishonest. If I compile with Jenkins, Actions or Earthly, that compile time is going to be the same under each build system assuming the same build node. Claiming you're 20x faster when CI is firing within seconds is kind of meaningless. Caching and parallel execution are age old concepts when it comes to CI and every modern build system can do it. CI is all…

> Caching and parallel execution are age old concepts when it comes to CI and every modern build system can do it. It is much harder to set up caching/parallelization with GitHub Actions, Jenkins, Docker, and Make than it is with just Earthly

That caching is not earthly, it's buildkit

Earthly is just a skin and a hosted service as I understand it

Re: We built the fastest CI and it failed

#164

Post by verdverm »

jordemort wrote: post #46

I'm glad that they're just shutting down the service and not closing up shop entirely. I really like the tool (to the point that I regularly stop by their careers page); the syntax for an Earthfile is a very sensible and incremental evolution of Dockerfile syntax, and it makes a whole bunch of things easy that were either previously impossible or very awkward with a Dockerfile alone. I remember when Docker introduced…

Try Dagger, it's better than Earthly

I'm a happy user, not paying them yet

Re: We built the fastest CI and it failed

#165

Post by shepherdjerred »

verdverm wrote: post #163
shepherdjerred wrote: post #84

Earlier quoted context omitted.

> Caching and parallel execution are age old concepts when it comes to CI and every modern build system can do it. It is much harder to set up caching/parallelization with GitHub Actions, Jenkins, Docker, and Make than it is with just Earthly

That caching is not earthly, it's buildkit Earthly is just a skin and a hosted service as I understand it

Yes, you're right.

Re: We built the fastest CI and it failed

#166

Post by bastardoperator »

marcinzm wrote: post #140
MatthiasPortzel wrote: post #98

Earlier quoted context omitted.

From the post: > Earthly gives you write once, run anywhere build consistency, making it super easy to reproduce CI failures on your local computer.

That doesn't help with non-deterministic failures and I've also yet to see a true "write once, run anywhere" system ever. It may be 99% "write once, run anywhere" but there's always that 1% edge case.

Exactly. This just reminds me of "works on my computer". Building locally is more advantageous for developing CI workflow, but I want to be as close to prod as possible and doing that on snowflake developer workstations is an exercise in futility.

Re: We built the fastest CI and it failed

#167

Post by unbalancedevh »

fydorm wrote: post #155
unbalancedevh wrote: post #134

It really bugs me when authors use acronyms or abbreviations without expanding what they mean. It tells the reader, "I didn't write this article for you, so don't bother trying to understand it."

Which terms are you referring to? If you aren't familiar with "CI/CD", then this article really isn't for you, as you haven't run into the problems this startup is trying to solve.

Yes, CI/CD. It would be nice to know in the first paragraph that the article isn't for me, instead of having to read 1/3 of the article and then infer it, or even keep guessing what it's about. Sometimes that's fine, but I seem to be seeing it more often, and it discourages people new to the field or who are still exploring.

Re: We built the fastest CI and it failed

#168

Post by Phelinofist »

hedora wrote: post #93
bastardoperator wrote: post #81

Earlier quoted context omitted.

It's one thing to know a build failed, it's another thing to know why and how a build failed, where it failed, and what the cause was. If I have to swim through or context shift to a separate build system's output to get that, what is this actually doing for me?

Also, which of my 1,000 test suites are flaky, and when did they start being flaky? Flamebox almost got this right, but groups by wall-clock instead of git sha: https://flamebox.sourceforge.net The screenshot is most of what there is to see. Clicking a test run opens the test log. Does anyone know of a currently-maintained alternative with a less-bad dashboard?

That looks really cool, thanks for the link. Something like this well integrated into Jenkins would be super useful.

Our DevOps team build something custom: they check all test logs of all builds of all branches and aggregate all that into a Grafana dashboard. We use it to monitor tests that are failing, to get a better grip on flakiness. Works okay but could be better.

Re: We built the fastest CI and it failed

#169

Post by verdverm »

pxc wrote: post #154

I looked briefly at Earthly a while ago for work because I have to write CI integrations for a whole bunch of CI platforms (GitLab, Azure DevOps, Jenkins, and maybe GitHub Actions). I ended up going with a very similar product (i.e., another nice BuildKit frontend with integrations into a bunch of CI systems), Dagger, because the DSL it used for defining pipelines at the time seemed better to me. I ended up deeply re…

CI/CD & provisioning are inherently order driven tasks, I actually prefer the SDKs over the CUE implementation. The main issue with the old CUE implementation is that they had to try and align BuildKits DAG solver with CUE's DAG solver, and they worked in opposite ways. I tried to help them through this about 3 years ago (being a CUE expert). I definitely think the SDKs is a much better solution for them. Whether we like it or not, a lot of the industry is moving this way. Pulumi is another example of this, though I'm sold on imperative cloud infra, builds do seem to have alignment here.

Side note, I will be exploring a new CUE + Dagger setup, though it will work differently than the old Dagger engine

Re: We built the fastest CI and it failed

#170

Post by sanderjd »

reactordev wrote: post #56

This is a good write up of why you shouldn’t give away the house when you open source things. The issue was really this: Earthly being open-source, Earthy Satellite users were already seeing the benefit from 95% of Earthly CI . I’m a huge fan of open source, however, if your business model includes an open source model - you need a differentiator. Beyond blazingly fast(tm). You need a reason for people to offer up th…

I know this is controversial, but I wish commercial "source available" were the more mainstream / not-shat-upon approach to services, rather than "open core".

I find it very frustrating to not be able to read the code or contribute bug fixes or self-host anything outside the "open core", and I have no need for any of the things that distinguish "open source" licenses from "source available" ones.

Post reply on HN