Year old startup overloaded GitHub – Incident report
1–10 of 43 posts
Re: Year old startup overloaded GitHub – Incident report
#2Discussion yesterday (14 points, 8 comments) https://news.ycombinator.com/item?id=42645887
Re: Year old startup overloaded GitHub – Incident report
#3Why is the title of the post-mortem "GitHub Outage"? It makes it sound like Lovable somehow brought down GitHub, when in reality it seems like they were rate-limited by GitHub for creating lots of repositories, then got their GitHub App completely blocked for breaching the Terms of Service.
> Incident report for the GitHub outage on January 2-3, 2025
Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application to be up, instead of taking full responsibility for it.
Re: Year old startup overloaded GitHub – Incident report
#4[dead]
Re: Year old startup overloaded GitHub – Incident report
#5Why is the title of the post-mortem "GitHub Outage"? It makes it sound like Lovable somehow brought down GitHub, when in reality it seems like they were rate-limited by GitHub for creating lots of repositories, then got their GitHub App completely blocked for breaching the Terms of Service. > Incident report for the GitHub outage on January 2-3, 2025 Writing it like that looks like you're pushing the blame on your do…
> Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application be up, instead of taking full responsibility for it.
Well, they did what they were supposed to - they explicitly asked Github what they were up to, Github gave an explicit "we're ok with this, go ahead", and once Github sees that, whoops, it's causing errors they don't even bother to check if there are support tickets open with the customer, they just go and disable their access.
Re: Year old startup overloaded GitHub – Incident report
#6I don't really understand how a well funded startup like this, with something that is relatively trivial, yet critical to their product, decided to just shove it into GitHub.
Re: Year old startup overloaded GitHub – Incident report
#7Why is the title of the post-mortem "GitHub Outage"? It makes it sound like Lovable somehow brought down GitHub, when in reality it seems like they were rate-limited by GitHub for creating lots of repositories, then got their GitHub App completely blocked for breaching the Terms of Service. > Incident report for the GitHub outage on January 2-3, 2025 Writing it like that looks like you're pushing the blame on your do…
> Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application be up, instead of taking full responsibility for it. Well, they did what they were supposed to - they explicitly asked Github what they were up to, Github gave an explicit "we're ok with this, go ahead", and once Github sees that, whoops, it's causing errors they don't even bothe…
But that's true for any 3rd party you'd depend on. Everything will work until it doesn't. Doesn't mean you're less responsible for your project being down.
A title like "GitHub caused our outage" would still make it clear the downtime wasn't the direct action of anyone on the team, yet still take responsibility over that it happened. Instead, labeling it "Incident report for the GitHub outage" just seems like straight up blaming someone else.
Re: Year old startup overloaded GitHub – Incident report
#8Many mistakes were made by Lovable that they could be berated for but on a more positive note, there is a lesson for us all: if you're doing something that you're worried about being problematic (e.g: creating a large volume of GitHub repositories) reaching out is a good thing but it is important to understand who you are reaching out to. GitHub is a huge organization, front-line support is not likely to have intimate knowledge of how exactly the acceptable usage policy is enforced nor the permission to make agreements. The key when reaching out is to find someone who has authority on the subject. Ideally, GitHub's front-line support would have escalated to the appropriate person/team but that isn't always possible (maybe they don't know who, maybe they're having a bad day and forgot). If the answer you get seems too convenient, it is probably not correct.
Re: Year old startup overloaded GitHub – Incident report
#9Besides the outage, isn't it kind of an insane setup to use GitHub to store every clients' project, especially with 10k new repos every day?
Re: Year old startup overloaded GitHub – Incident report
#10Earlier quoted context omitted.
> Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application be up, instead of taking full responsibility for it. Well, they did what they were supposed to - they explicitly asked Github what they were up to, Github gave an explicit "we're ok with this, go ahead", and once Github sees that, whoops, it's causing errors they don't even bothe…
But that's true for any 3rd party you'd depend on. Everything will work until it doesn't. Doesn't mean you're less responsible for your project being down. A title like "GitHub caused our outage" would still make it clear the downtime wasn't the direct action of anyone on the team, yet still take responsibility over that it happened. Instead, labeling it "Incident report for the GitHub outage" just seems like straigh…
Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users."
And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fault.