Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Actions and Pages are experiencing degraded availability

githubstatus.com

Post reply on HN
Print view View original

241–250 of 446 posts

Re: GitHub Actions and Pages are experiencing degraded availability

#241

Post by woodruffw »

inigyou wrote: post #37
arandomhuman wrote: post #21

I love how even self hosted workers don't work during these outages - running jobs on their infrastructure being flakey is marginally acceptable, but the API to simply schedule workflows having this availability is mind boggling. Github just doesn't seem like a serious company anymore.

GitHub implements self hosted runners by running a normal runner that passes the environment to your runner and then polls it. That's why they cost as much as the smallest GitHub-hosted runner. This is no surprise given standard Microsoft operating procedure - https://news.ycombinator.com/item?id=47616242

I think they kind of have to operate that way if you want a control plane/access to GitHub’s own layered services like caching and artifact storage. There are plenty of things to blame GitHub for, but this one doesn’t seem sinister at all.

Edit: to be clear, I mean the part where they maintain state on their end. I have no idea if they do that with another runner instance, that seems unlikely.

Re: GitHub Actions and Pages are experiencing degraded availability

#243

Post by tom1337 »

timwis wrote: post #234
abidlabs wrote: post #224

It's not too hard to switch over from GitHub actions to other runners. For example, I wrote up the steps needed in order to use Hugging Face Jobs instead, which also enables GPU runners and other flavors. I use this for several of the repos I manage (such as Trackio): https://huggingface.co/blog/github-ci-hf-jobs

Is that not still down though because it still relies on GH actions to orchestrate?

It most likely is. We are using Ubicloud (which seems to work the same way hf jobs does) and it's also down. GitHub is not sending out Webhooks and is also not able to queue external jobs. If we had to emergency deploy anything now I'd have no idea how to really do that...

Re: GitHub Actions and Pages are experiencing degraded availability

#244

Post by yashap »

abidlabs wrote: post #224

It's not too hard to switch over from GitHub actions to other runners. For example, I wrote up the steps needed in order to use Hugging Face Jobs instead, which also enables GPU runners and other flavors. I use this for several of the repos I manage (such as Trackio): https://huggingface.co/blog/github-ci-hf-jobs

We use other runners (Blacksmith), and it hasn't helped, still hard down all day long. Brutal outage.

As a side note, I started getting these symptoms (actions staying queued forever or not running at all) at 5pm PDT yesterday, intermittently. And definitely full outage by 8:30pm PDT. So for sure full outage for 7 hours and counting, even if you use other runners, and I strongly believe partial outage for ~15.5+ hrs before that, even if it hasn't been acknowledged by GitHub yet.

Re: GitHub Actions and Pages are experiencing degraded availability

#245

Post by jitbit »

gershy wrote: post #157
zehaeva wrote: post #141

I don't think this portents anything great for software in general. We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year. I'm sure there's other things happening in the…

I guess github is kind of a shared garden. Interesting that, like in game theory, if everyone is using it too much, no one gets to use it.

tragedy of the commons, exactly

Re: GitHub Actions and Pages are experiencing degraded availability

#247

Post by woodruffw »

mgh95 wrote: post #240
woodruffw wrote: post #236

Earlier quoted context omitted.

Why not both? Higher base load combined with insufficient internal controls for ratelimiting/load-shedding (as in, they don’t know who to shed) would be explanatory.

If they can't implement something as simple as "decode upstream headers and determine if 429/503" I don't know what to say. Since this has knocked out all customers it indicates they likely don't have anything of this form implemented.

I meant shedding of legitimate base load, not retries. I think we can safely assume they do the latter.

Re: GitHub Actions and Pages are experiencing degraded availability

#248

Post by gtowey »

axod wrote: post #171
IshKebab wrote: post #151

Earlier quoted context omitted.

I don't think so. GitHub was bought by Microsoft 8 years ago and people have only started complaining about its uptime in the last year or so - exactly correlating with the surge in LLM use.

https://damrnelson.github.io/github-historical-uptime/ Seems pretty conclusive. Very similar story when they bought skype.

It's not the whole story. The biggest change is actually the internal rules for how downtime was reported, it wasn't actually such a large change in the actual reliability then.

Re: GitHub Actions and Pages are experiencing degraded availability

#249

Post by JohnTHaller »

zehaeva wrote: post #141

I don't think this portents anything great for software in general. We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year. I'm sure there's other things happening in the…

The last month GitHub hit four 9s of uptime was November 2024

Re: GitHub Actions and Pages are experiencing degraded availability

#250

Post by chiply »

carlosneves wrote: post #226
chiply wrote: post #185

Time to rant... This is absolutely unreal. Even self-hosted runners are impacted.... How can that be? The cost of this globally has got to be in the hundreds of millions to companies that use CI/CD through GitHub Actions. What if prod is broken and GitHub actions is stalling the deployment of your hotfix? What if this makes your organization miss and SLA and diminish user trust? What if this makes you miss a release…

That's exactly the situation I'm in... :crying-laughing: The fix is merged, but won't deploy... it's been hours Thankfully it's a batch job, and isn't interrupting production ATM

I feel for you.... This is not a position you should be put in.

There's always the escape hatch of running you GHA workflows locally, but unfortunately, despite the existence of packages like `act`, there is no way to fully recreate the GHA runtime locally. Tons of the special YAML syntax just can't (more accurately, "just doesn't") get interpreted by those local actions runners.

We never went this route, but at my old org, I always advocated for considering GHA to be wrapper around a single bash script (or whatever script you want to run), as a means of completely breaking out of the GHA hellscape that is programming in YAML, who's turing-completeness is pretty dubious.

Unless you have things set up this way, you (the client of GitHub) would have to completely redesign your CI on the fly, run it locally, and then figure out how to get the D compliment of the I to work in a way that is auditable. Fat chance for most teams I bet.

Thank god you're dealing with a batch scenario. Silver lining for sure. Still, embrace the anger.

What makes my blood boil is that there's millions of DEVs literally crying at the moment worrying about how GitHub's failure to be responsible will put their jobs in jeopardy.

And fingers crossed for you my friend. We're at 5+ hours at the time of this writing.... You're batch job may still have a chance!!!

Post reply on HN