I had a lot of goodwill for GitHub but I think today is the tipping point. Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again. The hope is dead.
That unicorn was already served in the RubyOnRails days!
Incident with Github.com [resolved]
681–690 of 1001 posts
Re: Incident with Github.com [resolved]
#682The exponential growth they are having right now is absolutely insane. I would caution against assuming its an easy problem, or that anyone else has demonstrated an ability to handle this level of scale/growth any better. In May they had to solve for 30x growth overnight, and I would guess its closer to 100x by now. The reality is a centralized solution like this may no longer be a viable option with the scale of cod…
Github definitely didn't 100x its paying customers, and Github Copilot doesn't seem to have any serious users. Should we really be calling it growth when revenue is flat but server load is exploding due to everyone and their cat vibecoding dozens of throwaway weekend projects?
If Github didn't have insane margins before, they are definitely hemorrhaging money now. If they want to survive they have to introduce strict rate limits and 10x "Vibecoder" pricing tier sooner rather than later - but I bet Microsoft won't let them because it would hurt their already-bad AI revenue...
Re: Incident with Github.com [resolved]
#683I had a lot of goodwill for GitHub but I think today is the tipping point. Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again. The hope is dead.
Re: Incident with Github.com [resolved]
#684I had a lot of goodwill for GitHub but I think today is the tipping point. Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again. The hope is dead.
Re: Incident with Github.com [resolved]
#685Almost 3 hours in and still "We are still working to identify the root cause" Can't even view diffs in the web interface! I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issue…
Update 16:36 UTC - We identified the problematic component and have taken corrective actions. There are strong signs of recovery but we are still working to completely restore service, with error rates still remaining slightly elevated. We will post further updates as recovery continues. just shy of 3 hours to find the issue
Re: Incident with Github.com [resolved]
#686I don't understand why Github hasn't solved this problem with pricing updates. My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude. So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over? This seems like a basic economics problem.
I have wondered the same thing. Even a small fee like $50 a year would get rid of a lot of AI slop that justs eats up resources. I would gladly pay that fee if it means Github is reliable for paying customers.
Re: Incident with Github.com [resolved]
#687Re: Incident with Github.com [resolved]
#688Re: Incident with Github.com [resolved]
#689Earlier quoted context omitted.
People overestimate how much they care about stuff. Moving off GitHub would be more costly for us than having 5% downtime. Obviously there’s a tipping point, but it shows people are tolerant given the price tags.
For a reasonably common class of deployment and organization (E.g. an online service where the cost of a short outage is significant) it becomes a pain point. For instance if there's a sudden security flap and you need to re-spin your system and redeploy, but that process is gated on GitHub working, now you're screwed if that security flap happens when GitHub is in its 5% down time. Basically you've coupled your upti…
Re: Incident with Github.com [resolved]
#690I don't understand why Github hasn't solved this problem with pricing updates. My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude. So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over? This seems like a basic economics problem.
Because any price at all will immediately cause users to shift to another platform, and GitHub's value is that it is _the_ place to put your code on the internet.