Live data from Hacker News

Incident with Pull Requests, Issues, Git Operations and API Requests

githubstatus.com

131–140 of 221 posts

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#131

Good that the Billing functionality is still at 3 nines at least.

Well, the significant growth comes from freemium usage. A whole lot of vibe slop triggering actions, with no supporting business. So revenue has not tracked all other growth and the billing system isn’t stressed.

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#132

is it me or ever since AI coding became the norm, there have been way more outages with otherwise reliable services? I get downtime on Supabase every few weeks. Even Cloudflare. And now Github

They are definitely more outages but the question is if these outages are due to the providers using LLMs to build there products and are therefore not delivering the quality they did before or have LLMs enabled a completely new user base to create projects which they deploy in the free tiers of named providers and they simply cannot keep up with the growth and the new influx of free users is skewing their mixed calculations (free vs paid) so heavily that they cannot scale without losing money. I'd probably say it's a mix of both.

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#133

Earlier quoted context omitted.

Has nothing to do with Microsoft acquisition... AI usage has increased demand and load. More PRs, more Action runners, more of everything firing. GitHub just wasn't ready for the scale and are now having issues catching up with it as it continues to increase exponentially.

Yeah, that and Microsoft has been slow to move the infrastructure to something that scales better to handle that load. The more surpassing part is that Microsoft hasn't figured out a way to manage/contain the AI-sourced traffic better so it doesn't create all this noisy neighbor problems for non-AI usage/users.

Github's core platform doesn't really make that separation, anything a human can leverage on github an AI agent can as well, just faster and with heavier usage. End of day agents and humans are using the same services.

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#134
post #129

Earlier quoted context omitted.

Commits are up 14x year-over-year https://x.com/kdaigle/status/2040164759836778878

Yea but thats not really an excuse, is it? They offer a service, (some) people pay for that service and should therefore expect it to work. If GitHub cannot keep up with the growth then they could disable new account registrations or start reducing free tiers so people either use the free tier more mindfully or need to pay for usage-base products like Actions which would GitHub allow to scale.

I mean it's an easy problem to solve when it's just speculating solutions. But there's a very possible reality where in 5 years guys are making YouTube video essays about the fall of Github caused by their "obviously stupid decision" to throttle access to people who were trying to use their service in record numbers, leaving opportunity for someone else to come in and take their lunch.

I don't envy their position of having to scale that fast on something that has to be instant and real-time. As far as I know, you can't do CDN/edge caching shenanigans with a remote git repository like Google can with a YouTube video. It's gotta always be reading/writing to the latest, single source of truth.

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#135
post #6

https://isgithubcooked.com Normally I defend GH in the comments of these incidents but it’s been an impressively bad month by their standards, even when you filter for critical components filter out sev-2’s and 3’s.

It's not physically possible to run post-mortems for issues at those rates.

They should install OpenClaw for that as well.

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#136
post #134

Earlier quoted context omitted.

Yea but thats not really an excuse, is it? They offer a service, (some) people pay for that service and should therefore expect it to work. If GitHub cannot keep up with the growth then they could disable new account registrations or start reducing free tiers so people either use the free tier more mindfully or need to pay for usage-base products like Actions which would GitHub allow to scale.

I mean it's an easy problem to solve when it's just speculating solutions. But there's a very possible reality where in 5 years guys are making YouTube video essays about the fall of Github caused by their "obviously stupid decision" to throttle access to people who were trying to use their service in record numbers, leaving opportunity for someone else to come in and take their lunch. I don't envy their position of…

Sure, backseat commenting is easier and I wouldn't wanna be in charge at github right now, but on the other side there also a reality where we'd see video essays about githubs downfall because their reliability crashed so hard that businesses could not trust them and moved to competitors / self hosted instances which then meant less paid users to subsidize the ever growing demand of the free users.

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#137
post #23
post #10

Again? It was just yesterday [0] that GA was down and another incident today? I am convinced that Copilot and Tay.ai are destroying GitHub and there is no CEO of GitHub to contact. Now will you please self-host as I said 6 years ago? [1] [0] https://news.ycombinator.com/item?id=48278635 [1] https://news.ycombinator.com/item?id=22867803

You think a Microsoft chatbot from 2016 is destroying GitHub?

Microsoft corporate culture destroys Github reputation and tech.

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#139
post #134

Earlier quoted context omitted.

Yea but thats not really an excuse, is it? They offer a service, (some) people pay for that service and should therefore expect it to work. If GitHub cannot keep up with the growth then they could disable new account registrations or start reducing free tiers so people either use the free tier more mindfully or need to pay for usage-base products like Actions which would GitHub allow to scale.

I mean it's an easy problem to solve when it's just speculating solutions. But there's a very possible reality where in 5 years guys are making YouTube video essays about the fall of Github caused by their "obviously stupid decision" to throttle access to people who were trying to use their service in record numbers, leaving opportunity for someone else to come in and take their lunch. I don't envy their position of…

Yes it's potentially a write-heavy workload which also needs to be consistent aka the worst case scenario.

The easy solutions like caching and read replicas don't work and you're forced to go the route of sharding or similar techniques that have much more painful tradeoffs.

I'm not sure if that's why everything keeps breaking but at that scale write-heavy workloads are never going to be easy

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#140

Earlier quoted context omitted.

This is a convenient lie that GH likes to tell. Growth is nothing like exponential, its at most 300% over several years according to their own public numbers (presented misleadingly on graphs) But a couple of years ago they were crowing about how much work they were doing to prepare for “a billion developers”. If they had actually done that then the actual load from agents should have been no problem.

Is this growth in resource usage or growth in revenue? Because those numbers aren't necessarily coupled. I.e most action runners are free

usage
Post reply on HN