Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Year old startup overloaded GitHub – Incident report

lovable.dev

Post reply on HN
Print view View original

21–30 of 43 posts

Re: Year old startup overloaded GitHub – Incident report

#21

Post by happytoexplain »

qeternity wrote: post #6

I don't really understand how a well funded startup like this, with something that is relatively trivial, yet critical to their product, decided to just shove it into GitHub.

I see this sentence a lot:

"How can somebody who does X just do Y?", implying that Y is somehow bad or wrong in the context of X, or contrasts with the experience level implied by X.

Maybe it's true of the subject. Maybe they should have known better. But I think it's beneficial to take a step back and view it from the perspective of readers: Many people - even people familiar with X and Y - may have no idea why Y is bad in the given context, or, more relevantly, why this should be obvious, as you imply by using the word "just". I think there's a lot of benefit to be gained by trying to observe yourself, and notice when you are writing in this style, and add more context.

Re: Year old startup overloaded GitHub – Incident report

#22

Post by diggan »

Mathnerd314 wrote: post #10
diggan wrote: post #7

Earlier quoted context omitted.

But that's true for any 3rd party you'd depend on. Everything will work until it doesn't. Doesn't mean you're less responsible for your project being down. A title like "GitHub caused our outage" would still make it clear the downtime wasn't the direct action of anyone on the team, yet still take responsibility over that it happened. Instead, labeling it "Incident report for the GitHub outage" just seems like straigh…

Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users." And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fau…

> Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users."

No, GitHub won't ever take responsibility for what you have between you and your projects/apps/companies users. Nor did they do so in this case.

If I use service X for doing Y, and I write an email asking if it's OK that I upload 1000s of files every day, even if they say OK today, but then next week turn around and say "Nah", I'm still responsible for my users, who trust me and my team. Service X has no responsibility towards those users at all. It sucks though, I agree with that.

> And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fault.

Besides DNS and BGP I suppose, what services are "unavoidable" exactly? Git hosting isn't some arcane distributed network technology needing years of reading/experience to understand, the CLI ships with a web ui you can basically copy-paste to have your own Git hosting.

I'd say you are responsible for everything that you use and depend on. And if you think "Ah, I'll just chunk 10K repositories at GitHub a day, they say it's fine" and then don't have any plan in case GitHub suddenly says it isn't OK, you are responsible for the falloff if shit hits the fan.

Re: Year old startup overloaded GitHub – Incident report

#23

Post by hiatus »

Mathnerd314 wrote: post #10
diggan wrote: post #7

Earlier quoted context omitted.

But that's true for any 3rd party you'd depend on. Everything will work until it doesn't. Doesn't mean you're less responsible for your project being down. A title like "GitHub caused our outage" would still make it clear the downtime wasn't the direct action of anyone on the team, yet still take responsibility over that it happened. Instead, labeling it "Incident report for the GitHub outage" just seems like straigh…

Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users." And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fau…

> but if you can't trust a service to perform its advertised function, it is the service's fault.

Your customers don't care if it's the service's fault and will blame you.

Re: Year old startup overloaded GitHub – Incident report

#24

Post by hiatus »

temp8388344 wrote: post #20

The same user who posted this (Henrik501) also posted a comment two days ago (their only HN comment so far) praising the Lovable team for their incident response: https://news.ycombinator.com/item?id=42646297 And now this post with an exaggerated title. Seems like they're shilling and trying to make Lovable sound like a product with such huge traction that it "even brought Github down". They keep making outlandish cl…

The same username on Github follows only one person, Niklas Vatn, who appears to be a lovable employee. https://github.com/Henrik501?tab=following

Looking more into it, Henrik Westerlund works on "growth" at lovable (via LinkedIn) and posted Lovable to Product Hunt https://www.producthunt.com/@henrik_westerlund. So it appears they are shilling.

Re: Year old startup overloaded GitHub – Incident report

#25

Post by gabeio »

chrisgd wrote: post #16
aimazon wrote: post #8

Many mistakes were made by Lovable that they could be berated for but on a more positive note, there is a lesson for us all: if you're doing something that you're worried about being problematic (e.g: creating a large volume of GitHub repositories) reaching out is a good thing but it is important to understand who you are reaching out to. GitHub is a huge organization, front-line support is not likely to have intimat…

If I reach out and they can’t direct me to the right solution, that doesn’t seem like something I need to continue to solve for them. Seems too onerous.

If it is the critical path for your project or worse for your job, seems like that’s just due diligence.

> Seems to onerous.

I would only say that about a non-critical personal project.

Re: Year old startup overloaded GitHub – Incident report

#26

Post by temp8388344 »

hiatus wrote: post #24
temp8388344 wrote: post #20

The same user who posted this (Henrik501) also posted a comment two days ago (their only HN comment so far) praising the Lovable team for their incident response: https://news.ycombinator.com/item?id=42646297 And now this post with an exaggerated title. Seems like they're shilling and trying to make Lovable sound like a product with such huge traction that it "even brought Github down". They keep making outlandish cl…

The same username on Github follows only one person, Niklas Vatn, who appears to be a lovable employee. https://github.com/Henrik501?tab=following Looking more into it, Henrik Westerlund works on "growth" at lovable (via LinkedIn) and posted Lovable to Product Hunt https://www.producthunt.com/@henrik_westerlund . So it appears they are shilling.

Pathetic company built on lies and shady "growth hacks"

Re: Year old startup overloaded GitHub – Incident report

#27

Post by arccy »

mschuster91 wrote: post #5
diggan wrote: post #3

Why is the title of the post-mortem "GitHub Outage"? It makes it sound like Lovable somehow brought down GitHub, when in reality it seems like they were rate-limited by GitHub for creating lots of repositories, then got their GitHub App completely blocked for breaching the Terms of Service. > Incident report for the GitHub outage on January 2-3, 2025 Writing it like that looks like you're pushing the blame on your do…

> Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application be up, instead of taking full responsibility for it. Well, they did what they were supposed to - they explicitly asked Github what they were up to, Github gave an explicit "we're ok with this, go ahead", and once Github sees that, whoops, it's causing errors they don't even bothe…

sounds like a pretty sane thing to do: github protects the majority of their customers from instability caused by a few.

unless you're a super important partner, the people on call might never have heard of your little app and just decide that it's the only safe thing to do to protect the reliability of the system.

Re: Year old startup overloaded GitHub – Incident report

#28

Post by CamouflagedKiwi »

315,000 repositories + 10,000 per day? They were obviously correctly concerned this wouldn't be able to go on forever, hence the pre-emptive email, and of course they got the response saying it was okay, but I really feel like this is the kind of thing that's too dangerous to leave your company sitting on because sooner or later they were going to be told "no". It feels too much like it's found a point of arbitrage in Github's ToS, and indeed ended up causing problems.

I suppose they did respond pretty fast, but if I were them I'd have liked to have had the S3 option in my back pocket earlier. Maybe I'm just being too risk-averse here...

Re: Year old startup overloaded GitHub – Incident report

#30

Post by Kwpolska »

cratermoon wrote: post #15
remram wrote: post #11

Earlier quoted context omitted.

Their product is the creation of Git repos. Putting it on the platform their customers want to use makes a lot of sense. They probably should have had a backup location from day 2 though, I agree. If nothing else, in case of a GitHub outage.

What value does Lovable's product add, then?

It doesn’t add value. It fills them with LLM-generated code.
Post reply on HN