Live data from Hacker News

GitHub Actions and Pages are experiencing degraded availability

githubstatus.com

61–70 of 446 posts

Re: GitHub Actions and Pages are experiencing degraded availability

#61
post #32

Earlier quoted context omitted.

I'm not sure that it's that mind boggling. The entire complexity and value of GHA is not in the runners but the scheduler that the GH frontend hooks into and runners subscribe to jobs from. It's the most likely thing to fall over. ' If the scheduling was self hosted it would be inexcusable but you can always just connect whatever you want to webhooks.

That "scheduling" is just a git hook and a message queue, maybe with some database updates in between with very clear boundaries that make sharding easy to reason about, assuming they have a sane architecture (they evidently don't)

The one thing that falls over is that the webhooks are actually self modifying code since you have an 'on commit' webhook that bootstraps the actions logic from the workflow.yaml file(s) (which are not really config files, they have logic that needs to be evaluated).

I don't disagree that it's obvious they've got problems but I'm just saying it's obvious to me the part that falls over (the scheduling of jobs) and why that would impact self hosted runners, which do no scheduling but depend on it to function.

As for 'just a message queue with some database updates and sharding that's easy to reason about'... Here's a job scheduling problem as an example: imagine you schedule a job, and there's no runner available. How do you disambiguate between no runners available because you've reached capacity, runners not being available because they're on a real network with faulty connections, and runners not being available because of a faulty rollout of internal updates?

A simple message queue for job scheduling is fine if you own everything and can deal with the operational overhead of identifying those cases by hand, but Github can't do that.

Re: GitHub Actions and Pages are experiencing degraded availability

#63

It's been hours :( I have sympathy for the on-call team trying to resolve it, most of us have been there done that. But seems something is systematically going wrong at GH

It's honestly insane how terrible their reliability is. Over the past 4 weeks, 8 days with GitHub Actions outages, many of them multi-hour outages: >3 hours on each of July 9th, July 20th and today, and 1.5 hrs on July 23rd.

Outages happen, but this many outages so close together, and so many of them so major/long lasting, something is systematically wrong for sure. It's been seriously hamstringing our ability to ship code at my company.

Re: GitHub Actions and Pages are experiencing degraded availability

#64
post #63

It's been hours :( I have sympathy for the on-call team trying to resolve it, most of us have been there done that. But seems something is systematically going wrong at GH

It's honestly insane how terrible their reliability is. Over the past 4 weeks, 8 days with GitHub Actions outages, many of them multi-hour outages: >3 hours on each of July 9th, July 20th and today, and 1.5 hrs on July 23rd. Outages happen, but this many outages so close together, and so many of them so major/long lasting, something is systematically wrong for sure. It's been seriously hamstringing our ability to shi…

Mon and Dad (MS) are (generally) pretty solid with uptime.

What is happening at GH?

Rate of change trying to keep up with new challengers? Over-reliance on AI? Engineers trying to debug slop?

Re: GitHub Actions and Pages are experiencing degraded availability

#65
On other occasions, I'd take this as a time to have a walk because I'm blocked. Unfortunately, I need to get some stuff out for a customer quickly. Can't do anything about that though. GitHub is the only product in it's size class that I use that has this kind of incredibly poor uptime.

I'd love to know what the most common root causes for these outages are.

Re: GitHub Actions and Pages are experiencing degraded availability

#66
post #63

Earlier quoted context omitted.

It's honestly insane how terrible their reliability is. Over the past 4 weeks, 8 days with GitHub Actions outages, many of them multi-hour outages: >3 hours on each of July 9th, July 20th and today, and 1.5 hrs on July 23rd. Outages happen, but this many outages so close together, and so many of them so major/long lasting, something is systematically wrong for sure. It's been seriously hamstringing our ability to shi…

Mon and Dad (MS) are (generally) pretty solid with uptime. What is happening at GH? Rate of change trying to keep up with new challengers? Over-reliance on AI? Engineers trying to debug slop?

Yeah who knows, would be interesting to hear an inside take if any readers here are also GH devs!

They're at 93.91% uptime over the past 90 days, according to https://mrshu.github.io/github-statuses/ , and that doesn't even include today's outage yet.

A glorious one nine of reliability.

Re: GitHub Actions and Pages are experiencing degraded availability

#67
post #65

On other occasions, I'd take this as a time to have a walk because I'm blocked. Unfortunately, I need to get some stuff out for a customer quickly. Can't do anything about that though. GitHub is the only product in it's size class that I use that has this kind of incredibly poor uptime. I'd love to know what the most common root causes for these outages are.

[dead]

Re: GitHub Actions and Pages are experiencing degraded availability

#68

I love how even self hosted workers don't work during these outages - running jobs on their infrastructure being flakey is marginally acceptable, but the API to simply schedule workflows having this availability is mind boggling. Github just doesn't seem like a serious company anymore.

Microsoft is not a serious company anymore.

One might question your perspective if you ever thought they were serious.

Re: GitHub Actions and Pages are experiencing degraded availability

#70
post #40
post #29

Are there any other CI providers that offer free cloud-hosted Windows runners? Or has Microsoft made sure (through Windows licensing terms and pricing) that it's not possible to compete with their own CI offering? Edit: CircleCI seems to offer 750 minutes/month (whereas GitHub offers 1000 minutes/month).

You can always pay for it, use your own machine, use a VM on your own machine, or if you are developing non-slop FOSS, ask Codeberg.

Does Codeberg support Windows runners at all? Forgejo only seems to have unofficial third-party builds of the Forgejo Actions runner for Windows.
Post reply on HN