Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Actions and Pages are experiencing degraded availability

githubstatus.com

Post reply on HN
Print view View original

251–260 of 449 posts

Re: GitHub Actions and Pages are experiencing degraded availability

#251

Post by crmsystems »

onraglanroad wrote: post #140
thecatapps wrote: post #117

Earlier quoted context omitted.

I've never understood this argument. Even if self-hosted things were offline more often (which I've never found to be the case, I've had Forgejo + runners running for a year now with no downtime), the real benefit is that you yourself can work to bring it back online when it does, rather than waiting on a large, slow-moving organization to figure out what slopped PR caused their global service serving ungodly amounts…

I wrote my reply before yours appeared but it's basically agreeing. A simple Forgejo hasn't given me any grief. It really seems simpler. Maybe they're hosting in us-east-1 though :)

lol

Re: GitHub Actions and Pages are experiencing degraded availability

#254

Post by chiply »

nlightcho wrote: post #187
chiply wrote: post #185

Time to rant... This is absolutely unreal. Even self-hosted runners are impacted.... How can that be? The cost of this globally has got to be in the hundreds of millions to companies that use CI/CD through GitHub Actions. What if prod is broken and GitHub actions is stalling the deployment of your hotfix? What if this makes your organization miss and SLA and diminish user trust? What if this makes you miss a release…

Counterparty risk is still a thing. This is the cost of convenience, reminds me of the milkman joke in that South Park episode. Maybe a global SPOF owned by people who do not care is not the way to go.

Dude that milkman joke made me cackle out loud.

It's hard to draw a direct analogy there, but I feel like it echoes the same sentiment.

A soapbox I have is that GHA workflows are scripts that could run on your machine without any of the YAML stuff. Who gives a flying about the DAG or the logs? Which, by the way, if you're willing to walk to the milk store to buy your milk, could be recreated in a much more testable and maintainable way without any of the YAML bs that GHA prescribes....

But DAGs are pretty, and logstreams showing up in a browser application instill trust (for reasons that fly far above the head of yours truly). So people go for that. Pretty DAG, nice logstream; therefore, deliver my milk. All of a sudden.... The CI/CD platform is having its merry way with your SLAs, contract abidements, and hotfix deployments.

What a time to be alive.

Not to rail on the South Park thing, but the blast radius of this issue also reminds me of the episode where the internet dried up.

If this bs with GitHub continues, Parker/Stone will have to make a GitHub episode. How seen would we all feel if that happened?

Re: GitHub Actions and Pages are experiencing degraded availability

#255

Post by g42gregory »

I may be wrong, but I think this could be traced to the Microsoft's increasing reliance on the outsourced engineering workforce. Regardless of the country it has been outsourced to, the quality of service will suffer, apparently greatly as we now see.

It seems to manifest in many places: AI slop, Github outages, recently Miscoroft sent me an email demanding that I subscribe to 365, in order for MS Office (which I already paid for) to continue to work. I simply moved on to the other, free, provider.

Perhaps, it's time to evaluate the use of the Microsoft products?

Re: GitHub Actions and Pages are experiencing degraded availability

#256

Post by Nextgrid »

mschuetz wrote: post #96
rootnod3 wrote: post #94

Self-hosting is king.

Except anything I'd self host would be down much more often. I've hardly ever actually experienced down time issues with github when I needed it.

Hard disagree.

Self-hosting a service like GitHub that operates at GitHub scale is difficult.

Self-hosting a service like GitHub that operates at the typical small/medium company's scale is trivial.

A single machine (with separate runners for CI) will cover many companies' needs. It being a single machine eliminates a lot of the complexity and failure modes associated with a distributed system and makes backups/restores/maintenance easy.

Re: GitHub Actions and Pages are experiencing degraded availability

#257

Post by nodesocket »

abidlabs wrote: post #224

It's not too hard to switch over from GitHub actions to other runners. For example, I wrote up the steps needed in order to use Hugging Face Jobs instead, which also enables GPU runners and other flavors. I use this for several of the repos I manage (such as Trackio): https://huggingface.co/blog/github-ci-hf-jobs

Also use Blacksmith for arm64 runners but this outage apparently affected webhooks so their runners are not picking up jobs either.

Re: GitHub Actions and Pages are experiencing degraded availability

#258

Post by msyea »

I think this is because all frontier AI providers integrate with GitHub as a default git forge through their GitHub apps platform and their free tier is too generous. They should either charge some of these huge partners access to GitHub apps or partition the free tier off and have a different SLA.

If the frontier AI used GitLab as a default I'm sure they'd be the ones suffering now.

Re: GitHub Actions and Pages are experiencing degraded availability

#259

Post by tofuahdude »

Melatonic wrote: post #206
zehaeva wrote: post #141

I don't think this portents anything great for software in general. We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year. I'm sure there's other things happening in the…

The funny thing is that if companies wanted they could probably use AI to instead increase uptime. Keep existing QA teams (instead of replacing them) and then use AI for better and more timely monitoring (and messaging even) and as an additional Always-Testing™ layer of QA

"Just use AI" is definitely not the solve for the problems Github faces.

Re: GitHub Actions and Pages are experiencing degraded availability

#260

Post by betaby »

markus_zhang wrote: post #160
hinkley wrote: post #38

Earlier quoted context omitted.

They never were. They were just really good at maintaining the bluster and backing it up with expensive lawyers. Now that they don't come in claws-first like they did when they were a NASDAQ rocket to the stars, it's easy to see. Their MO was to court an executive and sell second-rate tools to them before the people who had to use them had a chance to say anything. It doesn't matter how much evidence you can provide…

They definitely managed to create a lot of reliable products. Windows NT was one of them. Up to Windows XP the products were pretty solid and each had visible improvement against the previous one. Their language products were/are still solid IMO. Maybe Visual Studio is sluggish, but we can still use an older version if we want. Plus they put a lot of effort optimizing VSCode, too. Even back in the MS-DOS/16-bit Windo…

> They definitely managed to create a lot of reliable products. > Windows NT was one of them.

Disagree. I was admining NT4.0 boxes back then and it was

1. very slow

2. constantly leaking that required weekly scheduled reboots

3. security wise it was nightmare even by that times standard

Post reply on HN