Good that the Billing functionality is still at 3 nines at least.
Incident with Pull Requests, Issues, Git Operations and API Requests
131–140 of 221 posts
Re: Incident with Pull Requests, Issues, Git Operations and API Requests
#132is it me or ever since AI coding became the norm, there have been way more outages with otherwise reliable services? I get downtime on Supabase every few weeks. Even Cloudflare. And now Github
Re: Incident with Pull Requests, Issues, Git Operations and API Requests
#133Earlier quoted context omitted.
Has nothing to do with Microsoft acquisition... AI usage has increased demand and load. More PRs, more Action runners, more of everything firing. GitHub just wasn't ready for the scale and are now having issues catching up with it as it continues to increase exponentially.
Yeah, that and Microsoft has been slow to move the infrastructure to something that scales better to handle that load. The more surpassing part is that Microsoft hasn't figured out a way to manage/contain the AI-sourced traffic better so it doesn't create all this noisy neighbor problems for non-AI usage/users.
Re: Incident with Pull Requests, Issues, Git Operations and API Requests
#134Earlier quoted context omitted.
Commits are up 14x year-over-year https://x.com/kdaigle/status/2040164759836778878
Yea but thats not really an excuse, is it? They offer a service, (some) people pay for that service and should therefore expect it to work. If GitHub cannot keep up with the growth then they could disable new account registrations or start reducing free tiers so people either use the free tier more mindfully or need to pay for usage-base products like Actions which would GitHub allow to scale.
I don't envy their position of having to scale that fast on something that has to be instant and real-time. As far as I know, you can't do CDN/edge caching shenanigans with a remote git repository like Google can with a YouTube video. It's gotta always be reading/writing to the latest, single source of truth.
Re: Incident with Pull Requests, Issues, Git Operations and API Requests
#135https://isgithubcooked.com Normally I defend GH in the comments of these incidents but it’s been an impressively bad month by their standards, even when you filter for critical components filter out sev-2’s and 3’s.
They should install OpenClaw for that as well.
Re: Incident with Pull Requests, Issues, Git Operations and API Requests
#136Earlier quoted context omitted.
Yea but thats not really an excuse, is it? They offer a service, (some) people pay for that service and should therefore expect it to work. If GitHub cannot keep up with the growth then they could disable new account registrations or start reducing free tiers so people either use the free tier more mindfully or need to pay for usage-base products like Actions which would GitHub allow to scale.
I mean it's an easy problem to solve when it's just speculating solutions. But there's a very possible reality where in 5 years guys are making YouTube video essays about the fall of Github caused by their "obviously stupid decision" to throttle access to people who were trying to use their service in record numbers, leaving opportunity for someone else to come in and take their lunch. I don't envy their position of…
Re: Incident with Pull Requests, Issues, Git Operations and API Requests
#137Again? It was just yesterday [0] that GA was down and another incident today? I am convinced that Copilot and Tay.ai are destroying GitHub and there is no CEO of GitHub to contact. Now will you please self-host as I said 6 years ago? [1] [0] https://news.ycombinator.com/item?id=48278635 [1] https://news.ycombinator.com/item?id=22867803
You think a Microsoft chatbot from 2016 is destroying GitHub?
Re: Incident with Pull Requests, Issues, Git Operations and API Requests
#138Re: Incident with Pull Requests, Issues, Git Operations and API Requests
#139Earlier quoted context omitted.
Yea but thats not really an excuse, is it? They offer a service, (some) people pay for that service and should therefore expect it to work. If GitHub cannot keep up with the growth then they could disable new account registrations or start reducing free tiers so people either use the free tier more mindfully or need to pay for usage-base products like Actions which would GitHub allow to scale.
I mean it's an easy problem to solve when it's just speculating solutions. But there's a very possible reality where in 5 years guys are making YouTube video essays about the fall of Github caused by their "obviously stupid decision" to throttle access to people who were trying to use their service in record numbers, leaving opportunity for someone else to come in and take their lunch. I don't envy their position of…
The easy solutions like caching and read replicas don't work and you're forced to go the route of sharding or similar techniques that have much more painful tradeoffs.
I'm not sure if that's why everything keeps breaking but at that scale write-heavy workloads are never going to be easy
Re: Incident with Pull Requests, Issues, Git Operations and API Requests
#140Earlier quoted context omitted.
This is a convenient lie that GH likes to tell. Growth is nothing like exponential, its at most 300% over several years according to their own public numbers (presented misleadingly on graphs) But a couple of years ago they were crowing about how much work they were doing to prepare for “a billion developers”. If they had actually done that then the actual load from agents should have been no problem.
Is this growth in resource usage or growth in revenue? Because those numbers aren't necessarily coupled. I.e most action runners are free