Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Actions and Pages are experiencing degraded availability

githubstatus.com

Post reply on HN
Print view View original

431–440 of 448 posts

Re: GitHub Actions and Pages are experiencing degraded availability

#431

Post by Grombobulous »

ModernMech wrote: post #423
Grombobulous wrote: post #399

Earlier quoted context omitted.

I have trouble believing it’s a uniquely difficult situation when no other tech company of the same or larger scale has those same problems. Is there a single Google service that has ever had reliability this bad? Facebook? Instagram? TikTok? Apple? Those companies all run hugely scaled write-heavy platforms. Why is GitHub so uniquely unreliable? Me and 1 billion of my best friends can upload 5GB 4K videos to iCloud…

I won't pretend to be an expert in the various systems of running these services, but to me it seems that GitHub's system is especially vulnerable to what AI is doing. I'll give you an example: I pointed Codex at one of my repositories asking it to implement some features. Very quickly, it ballooned a 3 minute CI workflow to 30 minutes per commit, and the number of commits it started making increased 10 fold. So we'r…

I also can't pretend to be an expert on this. Conceptually, everything you're saying makes sense.

It still just sounds like a really vanilla "horizontally scaling VMs" problem, especially for yesterday's incident that was focused on pipelines.

If this is a "GitHub is giving away more capacity than it has" problem, that's easily solved by rate limiting and queuing.

This is GitHub's explanation:

> During the incident, some Actions Runner Controller (ARC) runner pods became stuck in an idle state. Affected users can delete those pods using kubectl or redeploy their Actions Runner Controller application. ARC will automatically create replacement runners.

> The next releases of Actions Runner and Actions Runner Controller will include an automatic recovery mechanism, preventing the need for these manual steps in the future.

That sounds a lot more like a major architectural flaw and a really obvious oversight.

Re: GitHub Actions and Pages are experiencing degraded availability

#432

Post by crabbone »

zehaeva wrote: post #141

I don't think this portents anything great for software in general. We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year. I'm sure there's other things happening in the…

I saw a graph of Github outages somewhere. The drop in uptime strongly correlates with the introduction of Github Actions. To be fair, it's a huge chunk of functionality, also, hard to make reliable. LLMs might have an effect, but the issues definitely started before LLMs were used for code generation, so, as much as I don't like the AI-generated code, I'd have to admit that it's probably not the cause here.

Re: GitHub Actions and Pages are experiencing degraded availability

#433

Post by crabbone »

acuteaura wrote: post #362
askonomm wrote: post #146

Earlier quoted context omitted.

To me this correlates more to them being bought by Microsoft, a company known for being seemingly incapable of creating quality software to the point that it's not even funny anymore, and also known for sloppifying all the products they touch.

GitHub Actions was always a mess. Because it is a Microsoft product. If you want to have a horrible time, try reading some of the runner code. It's early 2010s-style Windows-First, MSFT C# crud that has trouble not racing several threads to inconclusive status codes.

Yes. That was my impression too. Github Actions aren't a good product in many ways. It's also very complex and requires quite a bit more infrastructure than the rest of Github. Unfortunately, I don't think that the commercial side of the product would allow it to improve in the direction of better quality / uptime. It's cursed to be forever like MS Outlook: whenever it changes it's for the worse, even though it was never good.

Re: GitHub Actions and Pages are experiencing degraded availability

#434

Post by MarkMarine »

mirsadm wrote: post #421
Syntaf wrote: post #204

Earlier quoted context omitted.

There was a great article awhile back that shed some light on just how dysfunctional azure is as a platform: https://news.ycombinator.com/item?id=47616242 If I had to guess it's because Github is sitting on top on infrastructure held up by toothpicks and duct tape

Dunno that's my experience with most businesses. I doubt AWS is that much better under the hood.

Haven't worked in AWS but as a customer of AWS, GCP, and Azure there is a quite a difference between running workloads that are doing long-running computations on AWS (on demand is so rock solid that loosing a box is a rarity, I run on spot and still complete almost all of my runs) and running on Azure's "on-demand" style offering feels like spot (or worse) on AWS.

Re: GitHub Actions and Pages are experiencing degraded availability

#435

Post by crabbone »

onraglanroad wrote: post #163
IshKebab wrote: post #151

Earlier quoted context omitted.

I don't think so. GitHub was bought by Microsoft 8 years ago and people have only started complaining about its uptime in the last year or so - exactly correlating with the surge in LLM use.

I don't think so. I've seen those complaints for more than a year. I have an Ops background and I strongly suspect they were given a stupid timeline for the Azure migration. I've got to believe Microsoft have decent Ops people but the management wanted to move faster than was reasonable and screwed it up. Move one thing at a time and double check it all works and you can do a migration like this.

I've lived through two acquisitions by the world-largest companies (and few more smaller ones). Here's my impression of what often happens in situations like this:

* There's a huge disconnect between the actual product and what was sold to the buyer. It could be that the product was a borderline fraud, or it could be that the product was actually much better quality than the expectation on the buyer's side, but the buyer isn't interested in most of the product.

* Fear spreads in the acquired company that their product will be discontinued or reshaped into something else.

* The pay is good, probably much better than before the company was bought.

* Many internal teams end up lacking real purpose and try to insert themselves into every new internal project only to obfuscate their irrelevance.

This leads to some pathological developments, where the teams previously working on acquired products start doing a lot of useless, for-show work. Internal initiatives sprout like mushrooms after a summer rain, but they are all plagued by very broad (and mostly irrelevant) team involvement, duplication of existing products / services, and fear of being discovered. And while there are plenty of such initiatives, their role is to be a superficial distraction. In reality, everyone is afraid to touch the old code or do any sensible integration because it could lead to blanket firing of a lot of people. The middle-management behavior becomes a sort of exchange of favors, where everyone is afraid that the other can blackmail them into losing their job, and so everyone is trying to be extra nice by offering a slice of a pie to another manager.

What this leads to, in reality, is insane inertia (often despite highly shortened software release cycles), astronomic amounts of unmitigated tech. debt, opacity in communication with management, persecution of those who genuinely want to improve the system.

Based on this, my prediction is that Github will not survive. Just like Skype didn't. Somehow or other, Microsoft will find a way to replace the product with... MS Outlook with a new skin.

Re: GitHub Actions and Pages are experiencing degraded availability

#436

Post by vel0city »

MarkMarine wrote: post #434
mirsadm wrote: post #421

Earlier quoted context omitted.

Dunno that's my experience with most businesses. I doubt AWS is that much better under the hood.

Haven't worked in AWS but as a customer of AWS, GCP, and Azure there is a quite a difference between running workloads that are doing long-running computations on AWS (on demand is so rock solid that loosing a box is a rarity, I run on spot and still complete almost all of my runs) and running on Azure's "on-demand" style offering feels like spot (or worse) on AWS.

> on demand is so rock solid that loosing a box is a rarit

Maybe this really depends on region, but I can't say this is true. I've experienced tons of hardware failures that have caused instances to throw weird errors, instances to randomly stop, instances to randomly disappear (along with their corresponding EBS volume).

Standard EBS volumes are only 99.8% durable. If you run a lot of instances on EBS, some of them will disappear.

GCP meanwhile has balanced zonal PDs (not even regional, zonal) at >99.999% durability.

I've never lost a disk on GCP, I've never even had an instance stop once without me telling it to stop.

Azure is absolutely a hot mess though. Had a VM a client was paying for backups on. Couldn't restore the backup because they changed generations of VM platforms too many times, literally no way to restore it. Had to get their support to eventually give me a .vhd that magically appeared in a OneDrive share a few days after asking about it.

Re: GitHub Actions and Pages are experiencing degraded availability

#437

Post by crabbone »

UncleOxidant wrote: post #225
zehaeva wrote: post #141

I don't think this portents anything great for software in general. We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year. I'm sure there's other things happening in the…

I wonder if GitHub actions was a bad idea? Like maybe it's being abused for other kinds of compute besides just builds? And even builds themselves can require a lot of compute. I've only recently had a repo there where I wanted to do builds to make a release (both linux binaries and WASM) and whenever I do that tag and wait a few minutes for those builds to finish I think about all the other projects/repos out there…

They let you do it because GitLab was first to let you do that. MS had to match the offering of their most noticeable competitor. They also felt like thay had to one-up them... well, to win the competition.

So, from the sales point of view, Github Actions was, at the minimum, an alright idea. Not brilliant, but quite obvious and expected. From the engineering standpoint, however, this is a disaster on many levels. But, that never stopped Microsoft before. They don't try to win the market by making an objectively better product, their tactics are and always have been to make a product that can claim (with an asterisk) to be able to do a lot of things the customer wanted only to discover afterwards that those promises were phony.

Re: GitHub Actions and Pages are experiencing degraded availability

#438

Post by jeltz »

skinfaxi wrote: post #400
zehaeva wrote: post #141

I don't think this portents anything great for software in general. We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year. I'm sure there's other things happening in the…

> I don't think this portents anything great for software in general. I think it is a natural progression in technology. Failure rates were high for initial aircraft designs and safety improved over time, for instance.

But in this case it is the other way round, failure rate is going up.

Re: GitHub Actions and Pages are experiencing degraded availability

#440

Post by pythonaut_16 »

SoftTalker wrote: post #418
cortesoft wrote: post #168

Earlier quoted context omitted.

Are these outages caused by introduced bugs, though, or by load issues? As someone who has spent many years working in high load environments, this is not an uncommon pattern. You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out. And then load keeps increasing and you suddenly hit a tipping point where everything keeps…

One of the big motivating factors in the development of git was the desire for decentralized version control. With GitHub, we threw that away. It's centralized on steroids. Now we're all depending on one platform that has to be massively scaled to deal with the massive load of serving almost every major software project on the planet. And when it is down, we all notice.

> With Github, we threw that away.

I see this repeated a lot and IMO it's simply not true.

The main decentralized advantage of Git is that you can continue to do VCS operations without network access or access to the remote host.

Most of what Github does is managing collaboration. The only thing we've "thrown away" by using Github is the email based workflows or directly pushing git branches to various hosts. But unless you're gonna have team members SSH into each other's machines you'd still need a central repo somewhere.

Post reply on HN