Live data from Hacker News

Incident with Github.com [resolved]

githubstatus.com

671–680 of 1001 posts

Re: Incident with Github.com [resolved]

#671
post #228

Earlier quoted context omitted.

I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding. But maybe I’m wrong?

Services like Facebook have orders of magnitude more activity than GitHub and manage much better uptime.

Services like Facebook aggressively block automated activity, including AI agents.

GitHub sells this as a service.

Re: Incident with Github.com [resolved]

#672
post #145

I recall reading years ago that cloud services were expected to run with a reliability of 3 or 4 '9's and that if they didn't competing services would quickly overtake them in adoption. The industry was supposed to be that cut throat. Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn'…

People overestimate how much they care about stuff. Moving off GitHub would be more costly for us than having 5% downtime. Obviously there’s a tipping point, but it shows people are tolerant given the price tags.

For a reasonably common class of deployment and organization (E.g. an online service where the cost of a short outage is significant) it becomes a pain point. For instance if there's a sudden security flap and you need to re-spin your system and redeploy, but that process is gated on GitHub working, now you're screwed if that security flap happens when GitHub is in its 5% down time. Basically you've coupled your uptime to GitHub's uptime in some measure.

Re: Incident with Github.com [resolved]

#673

Almost 3 hours in and still "We are still working to identify the root cause" Can't even view diffs in the web interface! I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issue…

No idea about some of that stuff, as I have more basic needs; nearlyfreespeech.net is cheap and stays out of my way.

Re: Incident with Github.com [resolved]

#674
post #311

Earlier quoted context omitted.

I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding. But maybe I’m wrong?

Even Claude could have caught and addressed most of these scalability issues long ago. This can’t be due to load or capacity. This is something else.

I’d love to see the org chart sizes

How many hires went into infra and platform? How many went to Gen AI org units?

From what I see, and that is a constant wave of very meh features, it seems like the latter.

Re: Incident with Github.com [resolved]

#675
post #145

I recall reading years ago that cloud services were expected to run with a reliability of 3 or 4 '9's and that if they didn't competing services would quickly overtake them in adoption. The industry was supposed to be that cut throat. Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn'…

If I'm running, say, an e-commerce business, an hour of downtime on my serving-path might cost me $100k of revenue. An hour of downtime for my development stack might cost me 30 person-hours = a few grand. As a result, I'm much more willing to have unreliable github than unreliable CDN/Compute/etc.

Re: Incident with Github.com [resolved]

#676

Earlier quoted context omitted.

I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding. But maybe I’m wrong?

Found this reddit thread from a month ago that talks about the scale: Commit volume went from 1 billion total in all of 2025 to 275 million per week by early 2026. https://www.reddit.com/r/softwaredevelopment/comments/1uv5s7...

Do you know how many thousands/millions of dollars that would cost to serve!?

Microsoft has to spend more money on acquiring games studios for billions of dollars then shut them down.

Re: Incident with Github.com [resolved]

#677

Almost 3 hours in and still "We are still working to identify the root cause" Can't even view diffs in the web interface! I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issue…

Update 16:36 UTC - We identified the problematic component and have taken corrective actions. There are strong signs of recovery but we are still working to completely restore service, with error rates still remaining slightly elevated. We will post further updates as recovery continues.

just shy of 3 hours to find the issue

Re: Incident with Github.com [resolved]

#678

Earlier quoted context omitted.

I understand the need to level-head the discussion and calming everyone. However, your "(you don't)" comment is not going to calm down all the people, who, you know, DO have a hotfix to push now, and DO have an angry customer that could not care less for which part of our infrastructure is breaking _their_ workflow. The only things would calm everyone down is guarantee that Microsoft would be paying for _our_ SLA bre…

It's past time to move your critical infrastructure off Github then, no?

It's not exactly done in a pinch, but yeah, I guess someone must be working on disrupting CIs by the audacious value proposition of "we'll work when you need to".

Re: Incident with Github.com [resolved]

#679
post #211

I don't understand why Github hasn't solved this problem with pricing updates. My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude. So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over? This seems like a basic economics problem.

Because any price at all will immediately cause users to shift to another platform, and GitHub's value is that it is _the_ place to put your code on the internet.

The network effects isn’t that much, and like what we see with Steam, you want to be where the audience is.

More realistically, MAU or something is a metric/OKR. It’s easy to imagine why a business wouldn’t want to cap that.

Re: Incident with Github.com [resolved]

#680

Earlier quoted context omitted.

[flagged]

I have zero investments in agentic coding; in fact, my list of investments is pretty public, so I don't really know what point you're trying to make. But creating a new account to try to defend yourself against a threat you claim to be non-existent doesn't scream "I feel secure" to me.

[flagged]
Post reply on HN