Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Actions and Pages are experiencing degraded availability

githubstatus.com

Post reply on HN
Print view View original

421–430 of 446 posts

Re: GitHub Actions and Pages are experiencing degraded availability

#421

Post by mirsadm »

Syntaf wrote: post #204
cortesoft wrote: post #168

Earlier quoted context omitted.

Are these outages caused by introduced bugs, though, or by load issues? As someone who has spent many years working in high load environments, this is not an uncommon pattern. You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out. And then load keeps increasing and you suddenly hit a tipping point where everything keeps…

There was a great article awhile back that shed some light on just how dysfunctional azure is as a platform: https://news.ycombinator.com/item?id=47616242 If I had to guess it's because Github is sitting on top on infrastructure held up by toothpicks and duct tape

Dunno that's my experience with most businesses. I doubt AWS is that much better under the hood.

Re: GitHub Actions and Pages are experiencing degraded availability

#422

Post by Izkata »

axod wrote: post #171
IshKebab wrote: post #151

Earlier quoted context omitted.

I don't think so. GitHub was bought by Microsoft 8 years ago and people have only started complaining about its uptime in the last year or so - exactly correlating with the surge in LLM use.

https://damrnelson.github.io/github-historical-uptime/ Seems pretty conclusive. Very similar story when they bought skype.

Heatmap of the past year: https://isgithubcooked.com/

It got worse around April/May this year, while that graph stops at February.

Re: GitHub Actions and Pages are experiencing degraded availability

#423

Post by ModernMech »

Grombobulous wrote: post #399
ModernMech wrote: post #379

Earlier quoted context omitted.

GitHub had 30 million users in 2018 when it was acquired by Microsoft. This year alone they added 30 million users for total of 180 million. So pre-Microsoft GitHub faced very different problems compared to post-Microsoft GitHub not even considering poor management or AI. They're by far the largest platform of its kind, so I'm sure the problems are uniquely difficult to solve.

I have trouble believing it’s a uniquely difficult situation when no other tech company of the same or larger scale has those same problems. Is there a single Google service that has ever had reliability this bad? Facebook? Instagram? TikTok? Apple? Those companies all run hugely scaled write-heavy platforms. Why is GitHub so uniquely unreliable? Me and 1 billion of my best friends can upload 5GB 4K videos to iCloud…

I won't pretend to be an expert in the various systems of running these services, but to me it seems that GitHub's system is especially vulnerable to what AI is doing.

I'll give you an example: I pointed Codex at one of my repositories asking it to implement some features. Very quickly, it ballooned a 3 minute CI workflow to 30 minutes per commit, and the number of commits it started making increased 10 fold. So we're looking at a 100x increase in usage just from myself.

You factor that into the fact that all public repo get access to this same compute, basically unlimited minutes, up to 20 parallel tasks and 6 hours per request. Now because of AI, every small little weekend throw away side project is running a full dev ops shop. And the pressure never goes down because the AI just start a cron job to run these systems every single day, whether anyone is even looking at these repos including the authors, who may themselves just be bots.

And the systems are sophisticated: parallel builds across windows, mac, multiple linux distributions, multiple architectures, embedded targets, web targets, full feature matrix, the works. The default MO of these agents is to do as much testing as possible without any regard for resource constraints, and GitHub offers them unlimited resources, so it's like an addict meeting a dealer.

So in a sense, maybe being attached to Microsoft is the problem. Or at least their unlimited wallet is -- Microsoft is the enabler in all of this.

Re: GitHub Actions and Pages are experiencing degraded availability

#424

Post by talonx »

cortesoft wrote: post #168
zehaeva wrote: post #141

I don't think this portents anything great for software in general. We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year. I'm sure there's other things happening in the…

Are these outages caused by introduced bugs, though, or by load issues? As someone who has spent many years working in high load environments, this is not an uncommon pattern. You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out. And then load keeps increasing and you suddenly hit a tipping point where everything keeps…

AI slop-code has exacerbated the problem and GitHub is not able to keep up. If you look at the last 12 month's outage reports from GitHub's own status page, many of them mention capacity issues as an underlying cause.

Re: GitHub Actions and Pages are experiencing degraded availability

#425

Post by stephenway »

muralimadhu wrote: post #217
muralimadhu wrote: post #216

Earlier quoted context omitted.

There's stuff like permissions, integration with PRs etc

I dont mind going through the pain if there's a great alternative.

what would make an alternative great enough to justify the move?

Re: GitHub Actions and Pages are experiencing degraded availability

#426

Post by lenerdenator »

zamalek wrote: post #410
b112 wrote: post #376

Earlier quoted context omitted.

Except, lots of devs use github, and see what using Azure means. Management is seeing that downtime too. So what insane person will ever touch Azure willingly? Hotmail used to run on bsd, and they tried to move it to WinNT, back in the day. Same thing. Disaster. And it was held up as a reason to never use winnt for serious work, for years. Microsoft just always shoots themselves in the foot.

I think my age group (~40) is the tail end of widespread dogmatic trust in Microsoft. Once we retire I think they are going to be in serious trouble.

They don't need dogmatic trust, just oligopolies and monopolies.

Re: GitHub Actions and Pages are experiencing degraded availability

#427

Post by mhitza »

zehaeva wrote: post #141

I don't think this portents anything great for software in general. We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year. I'm sure there's other things happening in the…

General outage was uncommon, but unicorns and octocat popping up for particular pages/repos was common enough for me in the past.

Re: GitHub Actions and Pages are experiencing degraded availability

#428

Post by bloppe »

inigyou wrote: post #350
habinero wrote: post #304

Earlier quoted context omitted.

Someone else quoted a 14x increase in load due to AI hammering github. I wouldn't be at all shocked if that was true, considering how much AI tools are DDOSing the entire internet. That plus migrating clouds is insanely difficult to manage. They're almost certainly drowning in traffic and trying to keep up.

There's no evidence the internet DDOS is AI companies

In my own CLAUDE.md files, I tell it to liberally use the `gh` CLI to search for issues and error messages across the entire GitHub code index. It's insanely effective, but it feels like I'm part of the problem, and I'm sure there are a lot of other people doing the same.
Post reply on HN