Live data from Hacker News

Year old startup overloaded GitHub – Incident report

lovable.dev

41–43 of 43 posts

Re: Year old startup overloaded GitHub – Incident report

#41
post #34

Earlier quoted context omitted.

> Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application be up, instead of taking full responsibility for it. Well, they did what they were supposed to - they explicitly asked Github what they were up to, Github gave an explicit "we're ok with this, go ahead", and once Github sees that, whoops, it's causing errors they don't even bothe…

If I read it correctly, it was a support person that provided them with assurance. Not an executive or vice president or manager or vp of sales. GitHub did not give them permission nor their approval; it was a single person in support department. Who relies on support people to determine the basis of their business when it’s obvious that they were concerned with the high usage rate and that it might cause problems fo…

Oh yeah, support staff aren't always aware of everything. For example, with one of our latest features, we didn’t have time to add a UI option to disable the feature. The expectation was that support staff could disable it via their special admin panel upon user request. However, I accidentally discovered that when users asked to hide the feature, tech support told them it wasn’t possible! It turned out the tech support lead forgot to share that information with the team.

As for the OP, they should have conducted load testing and implemented rate limits on their end, rather than blindly relying on someone’s word that GitHub was ready to handle all their product's load for free.

Re: Year old startup overloaded GitHub – Incident report

#42
Interestingly, currently the site has this:

>We're currently investigating issues. Please stay tuned until this error banner has been updated.

When I try to create a new project, it says "An unknown error occurred with the code sandbox"

Something about S3-backed repos didn't work out?

UPD. "Under maintenance: Lovable is currently not able to reach the cloud provider for our previews, fly.io" Now it's fly.io's problem, not GitHub's

Re: Year old startup overloaded GitHub – Incident report

#43

315,000 repositories + 10,000 per day? They were obviously correctly concerned this wouldn't be able to go on forever, hence the pre-emptive email, and of course they got the response saying it was okay, but I really feel like this is the kind of thing that's too dangerous to leave your company sitting on because sooner or later they were going to be told "no". It feels too much like it's found a point of arbitrage i…

If nothing else, I was a little surprised they didn't (or at least didn't mention) having a fail over plan in place already. Seems like "Prepare for the worst, hope for the best" would have been the logical game plan.
Post reply on HN