Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Year old startup overloaded GitHub – Incident report

lovable.dev

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Year old startup overloaded GitHub – Incident report

#31

Post by qeternity »

remram wrote: post #11
qeternity wrote: post #6

I don't really understand how a well funded startup like this, with something that is relatively trivial, yet critical to their product, decided to just shove it into GitHub.

Their product is the creation of Git repos. Putting it on the platform their customers want to use makes a lot of sense. They probably should have had a backup location from day 2 though, I agree. If nothing else, in case of a GitHub outage.

If you read the post-mortem, you will see that this has nothing to do with the platform their customers want to use, and everything to do with creating repos internally for their own organization.

Their product has change tracking which is clearly powered by git repos on GitHub.

So again, I ask, for something that is a SPOF, why rely on a third party?

Re: Year old startup overloaded GitHub – Incident report

#32

Post by chrisgd »

gabeio wrote: post #25
chrisgd wrote: post #16

Earlier quoted context omitted.

If I reach out and they can’t direct me to the right solution, that doesn’t seem like something I need to continue to solve for them. Seems too onerous.

If it is the critical path for your project or worse for your job, seems like that’s just due diligence. > Seems to onerous. I would only say that about a non-critical personal project.

Fair points

Re: Year old startup overloaded GitHub – Incident report

#33

Post by kgeist »

temp8388344 wrote: post #20

The same user who posted this (Henrik501) also posted a comment two days ago (their only HN comment so far) praising the Lovable team for their incident response: https://news.ycombinator.com/item?id=42646297 And now this post with an exaggerated title. Seems like they're shilling and trying to make Lovable sound like a product with such huge traction that it "even brought Github down". They keep making outlandish cl…

>praising the Lovable team for their incident response

For 8 hours, no one is aware the service is down. It then takes ~3 more days to fix it. One of their first decisions is to 'make as much noise as possible on social media' (?), and every step seems to create additional problems (corrupt repos etc.) Nothing appears to be well thought out, the blog post reads like they weren't ready for this at all, panicked and chaotic decisions without understanding the tech stack on a deeper level (race conditions, rate limits etc.) Not a lot of confidence in the team behind a project that looks like nothing more than a glue between an LLM and a storage backend.

Re: Year old startup overloaded GitHub – Incident report

#34

Post by cudgy »

mschuster91 wrote: post #5
diggan wrote: post #3

Why is the title of the post-mortem "GitHub Outage"? It makes it sound like Lovable somehow brought down GitHub, when in reality it seems like they were rate-limited by GitHub for creating lots of repositories, then got their GitHub App completely blocked for breaching the Terms of Service. > Incident report for the GitHub outage on January 2-3, 2025 Writing it like that looks like you're pushing the blame on your do…

> Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application be up, instead of taking full responsibility for it. Well, they did what they were supposed to - they explicitly asked Github what they were up to, Github gave an explicit "we're ok with this, go ahead", and once Github sees that, whoops, it's causing errors they don't even bothe…

If I read it correctly, it was a support person that provided them with assurance. Not an executive or vice president or manager or vp of sales. GitHub did not give them permission nor their approval; it was a single person in support department.

Who relies on support people to determine the basis of their business when it’s obvious that they were concerned with the high usage rate and that it might cause problems for their customers?

Re: Year old startup overloaded GitHub – Incident report

#36

Post by cudgy »

diggan wrote: post #12
qeternity wrote: post #6

I don't really understand how a well funded startup like this, with something that is relatively trivial, yet critical to their product, decided to just shove it into GitHub.

Honestly, doesn't surprise me much. Homebrew, the most popular package manager/repository for macOS, basically lives on GitHub and bases everything on top of it. Over the years, I think there been times when they've actually brought down GitHub (or close to at least). Most folks seem fine with it, at least it still lives on like normal as far as I know. I think engineering principles flew out the window a long time a…

Your observation about shipping fast to build an MVP house of cards is true. However, Homebrew is a free package manager, that likely relies on GitHub due to cheap or free storage for a free tool.

My assumption is that lovable is not free, so using a cheap or free service while it is taking money from their customers fits squarely into the categorization of poor engineering and possibly incompetence.

Re: Year old startup overloaded GitHub – Incident report

#37

Post by Mathnerd314 »

diggan wrote: post #22
Mathnerd314 wrote: post #10

Earlier quoted context omitted.

Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users." And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fau…

> Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users." No, GitHub won't ever take responsibility for what you have between you and your projects/apps/companies users. Nor did they do so in this case. If I use service X for doing Y, and I write an email asking if it's OK that I uplo…

Well, the app store, for example. Sure, it is of course a good idea to comply with the app store policies to the extent possible, but ultimately there not much you can do to prevent Google or Apple saying "we don't like this app" and pulling it. For example with the UTM emulator. So how then can Google or Apple making such a decision be "your responsibility"?

As another example, let's say you build a house in a hurricane-prone area. It's your responsibility to ensure the owner buys hurricane insurance, as mandated by law. It's not your responsibility to build a nuclear-bunker-grade house that is impervious to hurricanes. It is easy to point the finger and say "you should have thought of that", but in practice it is easier to deal with such catastrophes as they happen.

Re: Year old startup overloaded GitHub – Incident report

#38

Post by remram »

qeternity wrote: post #31
remram wrote: post #11

Earlier quoted context omitted.

Their product is the creation of Git repos. Putting it on the platform their customers want to use makes a lot of sense. They probably should have had a backup location from day 2 though, I agree. If nothing else, in case of a GitHub outage.

If you read the post-mortem, you will see that this has nothing to do with the platform their customers want to use, and everything to do with creating repos internally for their own organization. Their product has change tracking which is clearly powered by git repos on GitHub. So again, I ask, for something that is a SPOF, why rely on a third party?

What? Well that's crazy.

Re: Year old startup overloaded GitHub – Incident report

#39

Post by elicksaur »

temp8388344 wrote: post #20

The same user who posted this (Henrik501) also posted a comment two days ago (their only HN comment so far) praising the Lovable team for their incident response: https://news.ycombinator.com/item?id=42646297 And now this post with an exaggerated title. Seems like they're shilling and trying to make Lovable sound like a product with such huge traction that it "even brought Github down". They keep making outlandish cl…

I like that HN is so minimal, but obvious stuff like this makes me want to write a browser extension that lets me custom tag accounts for my own notes.

Re: Year old startup overloaded GitHub – Incident report

#40

Post by AstralStorm »

diggan wrote: post #22
Mathnerd314 wrote: post #10

Earlier quoted context omitted.

Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users." And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fau…

> Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users." No, GitHub won't ever take responsibility for what you have between you and your projects/apps/companies users. Nor did they do so in this case. If I use service X for doing Y, and I write an email asking if it's OK that I uplo…

Besides the fact that you also need packet traversal via a multitude of layer 3 protocols a lot of which would pass through some corp cloud. And the net neutrality is dead in the USA.

As for the repositories, there are no good SLA terms for such storage. Nobody offers this service because it's expensive to offer. Not on this scale. So if you need it, you have to invest a lot in hardware or colocation or specialized clouds. Plus admin. A whole datacenter, and suddenly you're an ant vs three goliaths.

Post reply on HN