Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Github.com [resolved]

githubstatus.com

Post reply on HN
Print view View original

841–850 of 1001 posts

Re: Incident with Github.com [resolved]

#841

Post by frollogaston »

pera wrote: post #573
stingraycharles wrote: post #52

Earlier quoted context omitted.

I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding. But maybe I’m wrong?

Save that for your local mom-and-pop store: Microsoft is a multi-billion dollar corporation with enough resources to, at the very least, provide a reliable service for enterprise clients. The "AI is using a lot of resources" excuse was maybe acceptable last year but not in Q3 2026.

Ok but if the same problem will happen with the mom-and-pop store, switching there doesn't help. This isn't a moral dilemma, people just want their stuff to be up.

Re: Incident with Github.com [resolved]

#842

Post by [deleted] »

s_dev wrote: post #145

I recall reading years ago that cloud services were expected to run with a reliability of 3 or 4 '9's and that if they didn't competing services would quickly overtake them in adoption. The industry was supposed to be that cut throat. Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn'…

[deleted]

Re: Incident with Github.com [resolved]

#843

Post by frollogaston »

s_dev wrote: post #145

I recall reading years ago that cloud services were expected to run with a reliability of 3 or 4 '9's and that if they didn't competing services would quickly overtake them in adoption. The industry was supposed to be that cut throat. Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn'…

That number of 9s might be a thing for the lower layer cloud services like on AWS or Azure or GCP. Even then, that us-east-1 outage was an accidental power move. It established that many are best off putting all their stuff on us-east-1 cause then their outages only happen at the best possible times, whereas multi-region complexity might cause its own kind of outage when nobody else is down.

Re: Incident with Github.com [resolved]

#844

Post by skydhash »

cortesoft wrote: post #811
adamddev1 wrote: post #697

Earlier quoted context omitted.

I was in a very similar situation and this is what I did: - put Forgejo on a little $5 Vultr VPS (I share the VPS with other things, Forgejo doesn't need much resources. I also added some block storage to host my Docker containers in the Forgejo Docker registry) - got a 8GB ram cheaper Hetzner VPS (you have to wait for their availability) for around 8 EUR and used that to install the CI runner - Use Netlify for hosti…

This sounds great for personal projects, but what about open source work? Or if you want to collaborate with more than a few people?

Unless you want those people to have merging rights, collaboration can be easily done with exporting a read-only version of the git repo via http (even a browsable one using cgit). Then anyone can work on their copy, generate patches (git format-patch) and send the files via any means of communication (email, slack,...).

Re: Incident with Github.com [resolved]

#845

Post by niffydroid »

If the LLM usage is the cause then they need to start placing limits or add costs, otherwise normal fee paying customers will move. I work in a financial institution, while moving away from GitHub is hard our parent group uses gitlab, so really it's just our resourcing to do that is the issue, but if we asked group for help it quickly becomes possible. I feel another 2 of these instances we'll make the decision

Re: Incident with Github.com [resolved]

#846

Post by ofjcihen »

19skitsch wrote: post #819
tom1337 wrote: post #677

Earlier quoted context omitted.

Update 16:36 UTC - We identified the problematic component and have taken corrective actions. There are strong signs of recovery but we are still working to completely restore service, with error rates still remaining slightly elevated. We will post further updates as recovery continues. just shy of 3 hours to find the issue

I mean to be fair it’s a huge production system, there are many moving parts. Having worked on outages before in complex systems it is very difficult to identify root causes under pressure

Especially when you’re worn down because it happens weekly right?

Re: Incident with Github.com [resolved]

#847

Post by sqren »

frollogaston wrote: post #824
leishman wrote: post #171

I don't understand why Github hasn't solved this problem with pricing updates. My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude. So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over? This seems like a basic economics problem.

It's easy from an armchair to say things like they should just get more capacity, or they should segregate free repos onto other servers. But maybe they are doing what you're thinking, and with that churn they get mistakes that cause outages, esp dealing with legacy systems. I've been there (not at GitHub or MSFT).

They do not need to make any technical changes. They could make the change overnight: $1 per repo per month (or whatever). This would stop the aggressive growth that they clearly cannot handle anyway.

The question is: are they willing to compromise growth for QoS. So far the answer is a resounding: "no". They are deliberately sacrificing the experience of existing users to gain new ones.

Re: Incident with Github.com [resolved]

#848

Post by mnewme »

jimbob45 wrote: post #721
stingraycharles wrote: post #52

Earlier quoted context omitted.

I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding. But maybe I’m wrong?

[flagged]

The vast majority of HN users are from the US.

Re: Incident with Github.com [resolved]

#849

Post by frollogaston »

sqren wrote: post #847
frollogaston wrote: post #824

Earlier quoted context omitted.

It's easy from an armchair to say things like they should just get more capacity, or they should segregate free repos onto other servers. But maybe they are doing what you're thinking, and with that churn they get mistakes that cause outages, esp dealing with legacy systems. I've been there (not at GitHub or MSFT).

They do not need to make any technical changes. They could make the change overnight: $1 per repo per month (or whatever). This would stop the aggressive growth that they clearly cannot handle anyway. The question is: are they willing to compromise growth for QoS. So far the answer is a resounding: "no". They are deliberately sacrificing the experience of existing users to gain new ones.

There are tons of existing free users, they'd be sacrificing those too. And yeah they don't want to sacrifice growth either, so the remaining is an engineering question.

Re: Incident with Github.com [resolved]

#850

Post by unknownfuture »

hombre_fatal wrote: post #714
unknownfuture wrote: post #685

Earlier quoted context omitted.

Weird, I've been told you can just throw AI at all your logs and metrics and it can magically solve all the things...

Well, it would be a mistake on your part to assume that everything in an organization, especially a large one like github, is a reflection of the capabilities of the state of the art in anything (AI, architecture, programming language, organizational management, operations, etc.). It's clear that updating the status page as quickly as possible isn't even something github cares about.

It would be a mistake to judge AI based on the performance of one of its strongest pushers and promoters? Interesting take.
Post reply on HN