Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built the fastest CI and it failed

earthly.dev

Post reply on HN
Print view View original

291–300 of 301 posts

Re: We built the fastest CI and it failed

#291

Post by MaxBarraclough »

sanderjd wrote: post #170
reactordev wrote: post #56

This is a good write up of why you shouldn’t give away the house when you open source things. The issue was really this: Earthly being open-source, Earthy Satellite users were already seeing the benefit from 95% of Earthly CI . I’m a huge fan of open source, however, if your business model includes an open source model - you need a differentiator. Beyond blazingly fast(tm). You need a reason for people to offer up th…

I know this is controversial, but I wish commercial "source available" were the more mainstream / not-shat-upon approach to services, rather than "open core". I find it very frustrating to not be able to read the code or contribute bug fixes or self-host anything outside the "open core", and I have no need for any of the things that distinguish "open source" licenses from "source available" ones.

> I have no need for any of the things that distinguish "open source" licenses from "source available" ones

This strikes me as short-term thinking.

If you've ever used OpenSSH or MariaDB, say, then you've benefited from the right to fork, granted by Free and Open Source licences. [0][1] There are of course thousands of other examples.

This is what the FSF calls Freedom 3, [2] and what the OSI's Open Source Definition calls Criterion 3: Derived Works. [3]

Needless to say this kind of thing typically isn't permitted by source-available licences.

[0] https://en.wikipedia.org/wiki/OpenSSH#History

[1] https://en.wikipedia.org/wiki/MariaDB

[2] https://www.gnu.org/philosophy/free-sw.en.html

[3] https://opensource.org/osd/

Re: We built the fastest CI and it failed

#292

Post by XCSme »

reactordev wrote: post #56

This is a good write up of why you shouldn’t give away the house when you open source things. The issue was really this: Earthly being open-source, Earthy Satellite users were already seeing the benefit from 95% of Earthly CI . I’m a huge fan of open source, however, if your business model includes an open source model - you need a differentiator. Beyond blazingly fast(tm). You need a reason for people to offer up th…

This is why I don't believe in businesses that offer both open-source and a paid cloud version. Their business goal is literarily: make sure the open-source version is not so good and easy to use/install/maintain, so that people do need to use our cloud offering.

Re: We built the fastest CI and it failed

#293

Post by sanderjd »

MaxBarraclough wrote: post #291
sanderjd wrote: post #170

Earlier quoted context omitted.

I know this is controversial, but I wish commercial "source available" were the more mainstream / not-shat-upon approach to services, rather than "open core". I find it very frustrating to not be able to read the code or contribute bug fixes or self-host anything outside the "open core", and I have no need for any of the things that distinguish "open source" licenses from "source available" ones.

> I have no need for any of the things that distinguish "open source" licenses from "source available" ones This strikes me as short-term thinking. If you've ever used OpenSSH or MariaDB, say, then you've benefited from the right to fork, granted by Free and Open Source licences. [0][1] There are of course thousands of other examples. This is what the FSF calls Freedom 3 , [2] and what the OSI's Open Source Definitio…

I mean, I'm very glad that open source (and also free) software exists. I don't think there should only be software with source available licenses, and I certainly admire and appreciate the giant amount of foundational work that has been done under those licenses.

But I think it comes with its own trade-offs. For instance, I might say that preferring open source is short-term thinking, because projects can become both critical infrastructure and essentially un-funded, and use one of the same examples you did, of openssh.

And I lament that people seem to be less critical of entirely opaque proprietary software than of source available software. I think if the only viable choices are nonprofit+open and commercial+closed, then we are going to end up with more closed software that is of higher quality than its open alternatives.

In general my interest in this topic flows from dissatisfaction with funding models for open source software. I really hate that so much important software relies on under- or even entirely un-paid labor to create and maintain it. I'm not under illusions that alternative models are a panacea, but I'm also not satisfied with the status quo. I get paid well for my work, and I hate that so many people whose labor I rely on aren't getting paid well for theirs!

Re: We built the fastest CI and it failed

#294

Post by MaxBarraclough »

sanderjd wrote: post #293
MaxBarraclough wrote: post #291

Earlier quoted context omitted.

> I have no need for any of the things that distinguish "open source" licenses from "source available" ones This strikes me as short-term thinking. If you've ever used OpenSSH or MariaDB, say, then you've benefited from the right to fork, granted by Free and Open Source licences. [0][1] There are of course thousands of other examples. This is what the FSF calls Freedom 3 , [2] and what the OSI's Open Source Definitio…

I mean, I'm very glad that open source (and also free) software exists. I don't think there should only be software with source available licenses, and I certainly admire and appreciate the giant amount of foundational work that has been done under those licenses. But I think it comes with its own trade-offs. For instance, I might say that preferring open source is short-term thinking, because projects can become bot…

> I might say that preferring open source is short-term thinking, because projects can become both critical infrastructure and essentially un-funded, and use one of the same examples you did, of openssh.

I don't follow. All told, OpenSSH is still going strong, and it owes its existence to the right to fork.

The FOSS model has a good history of software project longevity, especially compared to, say, proprietary software developed by start-ups.

More generally, the rights granted by FOSS licences are a good (albeit imperfect) protection against software turning user-hostile. To that end, a FOSS licence might not be sufficient, but it seems to be necessary.

> people seem to be less critical of entirely opaque proprietary software than of source available software

The Free Software Foundation used to consider semifree software to be its own subcategory of non-Free software, but they no longer make the distinction. [0] This makes sense given the organisation's goals - they don't want to be seen to be giving partial credit to licences that, while not as bad as some others, still deliberately deny the user the '4 freedoms'.

I agree that non-Free-but-source-available is better than non-Free-and-source-unavailable, but there are good reasons the Free Software and Open Source software movements have the principles they have.

> my interest in this topic flows from dissatisfaction with funding models for open source software.

Agreed. It's frustrating that so few people or organisations think to contribute to FOSS projects, even ones they deeply rely on.

There are ways to monetise aside from patronage/donation, such as paid support and paid managed services, but unfortunately it seems these are rarely successful in generating appreciable income.

[0] https://www.gnu.org/philosophy/categories.en.html#Proprietar...

Re: We built the fastest CI and it failed

#295

Post by smaudet »

lelanthran wrote: post #271
BoorishBears wrote: post #270

Earlier quoted context omitted.

Their reply is not SSO, it's some toy alternative they're proposing that none of your customers would accept (like saying "Dropbox is just rsync") SSO is hairy enough that you can't write it from scratch in any reasonable amount of time for what a typical SaaS needs. There's OSS SSO you can host yourself that supports enterprise : https://www.keycloak.org/ If you're B2C Firebase Auth is cheap, and doesn't actually re…

> SSO is hairy enough that you can't write it from scratch in any reasonable amount of time for what a typical SaaS needs. I expected so, looking at how much it costs to provide just SSO as a service. Thanks for keycloak, didn't know about that.

The comment about being a 'toy service' is disingenuous...

The comment about customers is possibly valid, the difference entirely your use case. If you consume services as a corp, and your goal is to be able to decouple any employee quickly, then you do not need SSO, assuming each service you use allows you the option to delete by username (with proper auth, ofc).

SSO is only needed when you want to give a customer federated access (ability to assign roles without activation on their part). For an internal corp it is largely unnecessary and a convenience thing not a security one (it is much more secure to require different passwords for every service versus a global SSO credential that gets you everywhere).

If your "customers" are your own employees, then this is your decision not a use case or a business requirement.

Re: We built the fastest CI and it failed

#296

Post by sanderjd »

MaxBarraclough wrote: post #294
sanderjd wrote: post #293

Earlier quoted context omitted.

I mean, I'm very glad that open source (and also free) software exists. I don't think there should only be software with source available licenses, and I certainly admire and appreciate the giant amount of foundational work that has been done under those licenses. But I think it comes with its own trade-offs. For instance, I might say that preferring open source is short-term thinking, because projects can become bot…

> I might say that preferring open source is short-term thinking, because projects can become both critical infrastructure and essentially un-funded, and use one of the same examples you did, of openssh. I don't follow. All told, OpenSSH is still going strong, and it owes its existence to the right to fork. The FOSS model has a good history of software project longevity, especially compared to, say, proprietary softw…

> I don't follow. All told, OpenSSH is still going strong, and it owes its existence to the right to fork.

Sorry, this was too implicit. What I was alluding to was the period of time when openssh had some showstopper security issues, and everyone realized it was being maintained by essentially one person with essentially no funding.

I don't want to downplay that the copyright owner of source-available software taking it in a bad direction or stopping maintenance or going out of business altogether is a big risk of that model (and of fully proprietary software as well). It totally is, and it is definitely something that the ability to fork solves.

But my point is that it's a trade-off. Both models have failure modes. And they both have (different) imperfect solutions for them.

> Agreed. It's frustrating that so few people or organisations think to contribute to FOSS projects, even ones they deeply rely on.

This is probably where our fundamental difference in philosophy comes from. I don't think it is at all the case that people don't "think to contribute to FOSS projects", it's that the incentives don't work out in favor of it, for most organizations. And I don't think the answer to that is to wish for altruism from organizations and free labor from individuals.

> There are ways to monetise aside from patronage/donation, such as paid support and paid managed services, but unfortunately it seems these are rarely successful in generating appreciable income.

Yeah, as I said, I find all of these solutions deeply unsatisfying. Patronage/donation has the "relying on altruism" issue. Paid support misaligns incentives, because if you manage to make perfect software, you've also managed to put yourself out of business, whereas if your software is difficult to use, that makes it lucrative. Paid managed services would be great, if Amazon couldn't just slap your code behind an official AWS service that 99% of people are going to use instead of your also-ran; but keeping them from doing that requires the restraint on commercial re-distribution that everyone seems to hate.

Re: We built the fastest CI and it failed

#297

Post by jcrawfordor »

gabereiser wrote: post #243
jcrawfordor wrote: post #226

Earlier quoted context omitted.

(GitLab employee but familiar with this because I use it personally as well) You can always connect your own runners to GitLab, whether SaaS or self-hosted, and whether free/CE or premium/ultimate/EE. GitLab SaaS also comes with free use of the central runners up to a quota. I'm not sure about the history here but as far as I know this has always been the case.

Circa 2013? I remember vividly implementing gitlab at a certain cable company I worked at because they were too cheap for GitHub. I remember sometime in 2012 some merging of worlds of GitLab CI and GitLab.com, it was around that timeframe until 2015 or so when the unofficial runner because official and CI and Repos were merged? You work there, ask the oldest grey beard if this was the case. I could very well be mista…

you might be right, especially in that time period new features were often EE only initially and then merged into CE. When CICD was first created it was its own software project basically, but it may have required APIs that weren't merged into CE yet. To be totally honest I forgot that CICD was around that early on.

Re: We built the fastest CI and it failed

#298

Post by aftbit »

aeyes wrote: post #76
aftbit wrote: post #41

Earlier quoted context omitted.

I think the key is that nobody "wants" to spend any time at all on CI or their build system. It is one of those necessary evils that you have to do to have functioning software. Once you do it, even if it's a slow buggy pile of hacks, you will just ignore it until the pain gets very very bad. This was the same problem that caused our team to abandon CircleCI. They kept wanting us to rewrite our CI configs, touting va…

What did they make you change? I have been using CircleCI since the beginning and the only breaking change I saw was when they introduced workflows. This didn't need any changes to the difficult parts of my own code and was mostly copy-paste. I believe that my configs are fairly complex using almost all existing features across many different projects and build targets.

We started using CircleCI in early 2016, I believe. There have been two major migration points. The first time was when they went from circle.yml to .circleci/config.yml, around 2018, which might have corresponded with Workflows. The second time was when they got rid of their old converged docker containers and moved to custom container runtimes (and a new set of images, none of which did _quite_ the same thing as their old images).

Neither of these took tons of effort to resolve, but both broke our builds and made us spend time thinking about CI when we would rather have been iterating on our products. When their later migration happened, Github Actions was just coming around, so we figured that if we had to rebuild and re-test all of our CI, we might as well move it to a more reliable and cheaper runner. Now we only use CircleCI on a handful of legacy repos.

Re: We built the fastest CI and it failed

#299

Post by wesleyyue »

shepherdjerred wrote: post #75
wesleyyue wrote: post #31

For anyone looking for fast ci, I'd highly recommend webapp.io. it automatically caches layers if it detects it doesn't need to be run, and you can split a VM to shard test execution. Runs way faster than GitHub actions, you can ssh into any layer to debug, and it's just like a Linux machine/dockerfiles so no crazy syntax/mental model to learn. Support is great and we've been super happy with the experience. Best CI…

That's literally what Earthly does

It's not. Have you tried both? I tried Earthly first and switched to webapp.io. It's a way better product.

Re: We built the fastest CI and it failed

#300

Post by shuntress »

pshirshov wrote: post #17
shuntress wrote: post #5

> Why pay both the CI vendor and us Exactly. Complexity , not speed, is the problem with builds. It makes perfect sense that "another layer of complexity that wraps up all the other layers" was tough to sell.

In this particular domain, performance problems are tightly coupled with complexity.

This is true to a certain extent. If a build lacks the automation required to run without significant human oversight then it will be slow because it is complex.

But a normal build should be completely automatic (or at least close to it) and at that point whether it takes 30 seconds or 3 minutes doesn't really matter.

Post reply on HN