Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Github.com [resolved]

githubstatus.com

Post reply on HN
Print view View original

961–970 of 1001 posts

Re: Incident with Github.com [resolved]

#961

Post by philippta »

tarxvf wrote: post #906
philippta wrote: post #711

The most shocking thing to me about the entire GitHub fiasco is: No matter how bad the uptime is, no matter how bad the UI is. We will keep using the software / service we depend on. Why do we as software engineers have set fault tolerance and high availability as the primary goal of any software / infrastructure architecture? This shows it's completely irrelevant.

What? No. The people that stay are the ones that don't care, that doesn't mean no one cares. That's a selection effect. The people that care left months or years ago. I'm one of them. It was really easy, Github doesn't offer anything special. Try and run a Pagerduty with the same level of attention to detail as Github. Far fewer people use Pagerduty casually. That Github has casual users and lazy users does not mean…

That might apply to personal users. But to businesses the switching cost is just too high. We had a discussion about it on our team. It was decided to keep using GitHub unfortunately.

Re: Incident with Github.com [resolved]

#962

Post by account42 »

Waterluvian wrote: post #257
leishman wrote: post #171

I don't understand why Github hasn't solved this problem with pricing updates. My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude. So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over? This seems like a basic economics problem.

The people with the spreadsheets have determined the value of this loss leader to be too enormous to shut it off.

The loss leader for their code hosting service or the loss leader for their AI subscriptions?

Re: Incident with Github.com [resolved]

#963

Post by account42 »

internetter wrote: post #779
BryanBeshore wrote: post #242

Earlier quoted context omitted.

They would benefit from a HackerNews-esque karma system.

That is essentially your number of followers. Pretty hard to have a system that cannot be gamed. I think followers are probably loosely correlated with tech ecosystem impact in some capacity, but the second there is an incentive to accumulate them, well...

I can't wait for the "please like and subscribe" popups on all software I use.

Re: Incident with Github.com [resolved]

#964

Post by account42 »

pera wrote: post #573
stingraycharles wrote: post #52

Earlier quoted context omitted.

I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding. But maybe I’m wrong?

Save that for your local mom-and-pop store: Microsoft is a multi-billion dollar corporation with enough resources to, at the very least, provide a reliable service for enterprise clients. The "AI is using a lot of resources" excuse was maybe acceptable last year but not in Q3 2026.

They are also at least partially responsible for and profit from the LLM spam.

Re: Incident with Github.com [resolved]

#965

Post by account42 »

pdimitar wrote: post #708
mhitza wrote: post #692

Earlier quoted context omitted.

They had large increase in volume, and trying to move to Azure at the same time. I don't envy them for either work they need to do. But also don't feel pithy because it's Microslop, at the end of the day.

All being said, I have most sympathy towards engineers of all people. They don't get to call the shots, they do as they are told. 99% of the time by an extremely out-of-touch managers. Managers and executives though? Those I do blame. Surely at this point it should be blindingly obvious their current strategy is not working.

Those engineers also chose to work in US big tech for the ridiculously high salaries that come with it. Having to deal with the problems resulting from that scale is just part of the deal.

Re: Incident with Github.com [resolved]

#966

Post by account42 »

kokanee wrote: post #713
worble wrote: post #129

Earlier quoted context omitted.

I have little sympathy that they're buckling over a problem they helped create.

Yeah let's not forget that in those brief few days when Sam Altman was fired, Satya Nadella swooped in and hired him. Despite the failure of that move, Microsoft has invested $13 billion directly into OpenAI, and has spent $100 billion on the partnership. No sympathy from me watching their software crumble as a result.

They have also been pushing people very hard to use the slop generators, including through GitHub.

Re: Incident with Github.com [resolved]

#968

Post by Aeolun »

__float wrote: post #773
Aeolun wrote: post #727

Earlier quoted context omitted.

I’m working on it. Unfortunately ‘high reliability’ also means ‘tons of testing’, and ‘finding all the performance holes before your customers’. It’s kind of a shame, because Github keeps giving me fantastic ways to toot my own horn, but I can hardly do that if I can’t improve on their numbers.

Architecturally, how are you storing Git data? If it isn't fundamentally different from bare-repo-on-disk with libgit2 or wrapping the git binary, you're not going to scale any better.

Blobs are unique. Whether you have 700 forks or not, data doesn’t increase. I’m unpacking the git stuff, keeping all the blobs on disk as alternates. Then syncing to object storage. This proved really slow so the blobs are still repacked to sync to object storage. It was fun finding out that some of the repos I test with have 500k blobs, I didn’t know either, but half a million individual put calls made it excessively obvious that wasn’t going to work.

Between the git hosts things move around based on available space, IO pressure and whether it’s been recently accessed or not (spinning disks are far cheaper these days). Theoretically there will be a point where the routing becomes a problem, but probably not until there’s several hundred git hosts (I’ll probably have bandwidth problems before that happens).

I’m not sure what the largest git repo is that github has, but I can theoretically keep repos up to 1TB around (plus a sheer unlimited number of forks for the same). I imagine the API’s get problematic before git does at that point (e.g. shuffling data around is much harder than storing it).

Re: Incident with Github.com [resolved]

#969

Post by [deleted] »

unknownfuture wrote: post #850
hombre_fatal wrote: post #714

Earlier quoted context omitted.

Well, it would be a mistake on your part to assume that everything in an organization, especially a large one like github, is a reflection of the capabilities of the state of the art in anything (AI, architecture, programming language, organizational management, operations, etc.). It's clear that updating the status page as quickly as possible isn't even something github cares about.

It would be a mistake to judge AI based on the performance of one of its strongest pushers and promoters? Interesting take.

[deleted]

Re: Incident with Github.com [resolved]

#970

Post by asdc18 »

scrollaway wrote: post #529
cbmalll wrote: post #455

June 17, 2023: Dohmke says Copilot will write 80% of code: https://www.freethink.com/robots-ai/github-copilot Slop code is written and pushed ... Aug 11, 2025: Dohmke leaves the sinking ship: https://www.entrepreneur.com/business-news/github-ceo-thomas... This is a true AI victory.

Do y'all have any evidence that github's recent outages are caused by slopcode rather than the significant increase in load, which is exactly what Github & MS have been claiming as the root cause? I'm tired of these BS stories being pushed by green accounts on HN like there's some damn agenda by someone who thinks their job will be saved "if only the CEOs realized that Slopcode Bad".

Go to the Ukraine front, chickenhawk.
Post reply on HN