Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub partial outage

githubstatus.com

Post reply on HN
Print view View original

61–70 of 88 posts

Re: GitHub partial outage

#61

Post by lysace »

mook wrote: post #60
lysace wrote: post #29

Earlier quoted context omitted.

Just your casual $3.8T company. There were so many severe Github Actions outages (10+ ?) in the past year. Cause: Migration to the disaster zone also known as Azure, I assume. Most of them happened during (morning) CET working hours, as to not inconvenience the americans and/or make headlines. Money doesn't buy competency. It's a long-term culture thing. You can never let go on maintaining competency in your organiza…

I thought GitHub Actions (in particular; not the rest of GitHub) was always Azure, because it was initially a fork of Azure Pipelines? GitHub as a whole, including the previously non-Azure bits, does seem flakier than a few years ago though, for sure.

You seem to be correct. Not that much visible from the outside, but yes it seems like they always ran on Azure, from the 2018 launch. (Apologies for the disinfo, although I qualified it with the "I assume".)

Re: GitHub partial outage

#62

Post by no_wizard »

isodev wrote: post #44
ecshafer wrote: post #23

Github is owned by Microsoft, so this is a pretty small time indie operation, you need to give them a break.

I bet Microsoft is sad not because people can’t push, but because the training data for Copilot has slowed down. PS: None of our 40+ engineers felt anything, our self hosted Forgejo is as snappy as ever.

until your hardware fails! Or your VPS provider goes down!

Or whatever else, software services going down is going to happen in some capacity, eventually. Real question is what is acceptable

Re: GitHub partial outage

#63

Post by cmckn »

gen220 wrote: post #43
kaishiro wrote: post #27

Earlier quoted context omitted.

We use GitLab on the daily. Roughly 200 repos pushing to ~20 on any given day. There have been a few small, unpublished outages that we determined were server side since we have a geo-distributed team, but as a platform seems far more stable than 5-6 years ago. My only real current complaint is that the webhooks that are supposed to fire in repo activity have been a little flaky for us over the past 6-8 months. We ha…

FWIW, GitHub is also unreliable with webhooks. Many recent GH outages have affected webhooks. They are pretty good, in my experience, at *eventually* delivering all updates. The outages take the form of a "pause" in delivery, every so often... maybe once every 5 weeks? Usually the outages are pretty brief but sometimes it can be up to a few hours. Basically I'm unaware of any provider whose webhooks are as reliable a…

> you can't really get around maintaining some sort of fall-back poll.

This has been my experience with GitHub Actions as well, which I imagine rely on the same underlying event system as webhooks.

Every so often, an Action will not be triggered or otherwise go into the void. So for Actions that trigger on push, I usually just add a cron schedule to them as well.

Re: GitHub partial outage

#64

Post by Arch485 »

0x1ch wrote: post #55
grepfru_it wrote: post #42

Earlier quoted context omitted.

We used to obsessively care about 500s. Like I would make a change that caused a 0.1% spike in 500s and I would silently say I'm sorry to the folks who got the unicorn page. I'm not sure the new school cares nearly as much. But then again this is how companies change as they mature. I saw this with StubHub as well.. The people who care the most are the initial employees, employee #7291 usually dgaf

I fall into the new school gen z category, and I think you're right. We don't care. We don't care about the problems started before us, and we owe nothing to no one (but our employers, must increase value for shareholders of course). I simply want to survive. I'll kiss ass where I have to, but not to people I don't work on behalf of.

I'll throw in my $0.02, as a fellow zoomer. I care about the things that are mine (as in, my code, my decisions, etc. etc.). But if management fucks up and tells me to fix it, there is no amount of money that will make me care. Especially if I advised management _not_ to do that in the first place.

Re: GitHub partial outage

#65

Post by prymitive »

Speaking of GitHub issues if you go to Insight->Traffic in your repo you’ll most likely see this banner:

“ Referring sites and popular content are temporarily unavailable or may not display accurately. We're actively working to resolve the issue.”

It’s been like that for months now with no sign of anyone working on it. They just don’t care about user experience anymore.

https://github.com/orgs/community/discussions/173494

Re: GitHub partial outage

#67

Post by ctkhn »

numbsafari wrote: post #20

Anyone using GitLab have any insight on how well their operations are running these days? We originally left GitLab for GitHub after being bit by a major outage that resulted in data loss. Our code was saved, but we lost everything else. But that was almost 10 years ago at this point.

My org hosts it on prem, and while I don't like the way pages are organized for projects, I only really interact with the PR page and that is laid out well. Most of my interaction with git is happening from my terminal anyway so ¯\_ (ツ)_/¯

Re: GitHub partial outage

#68

Post by tux3 »

colesantiago wrote: post #21
numbsafari wrote: post #20

Anyone using GitLab have any insight on how well their operations are running these days? We originally left GitLab for GitHub after being bit by a major outage that resulted in data loss. Our code was saved, but we lost everything else. But that was almost 10 years ago at this point.

No issues on GitLab. Haven't seen any outage from GitLab in like, ever.

That has definitely not been my experience. I like Gitlab, but they've had regular incidents all along. If a git push failed I wouldn't question it, it's almost never my network. I'd just open Gitlab's Gitlab and find the current active issue.

To Gitlab's credit their observability seems to be good, and they do a good job communicating and resolving incidents quickly.

Some companies that shall not be named have status pages that always show green and might as well be a static picture. Some use words like "some customers may have experienced partial service degradation" to mean "complete downtime". Gitlab also has incidents, but they're a lot more trustworthy. You can just open the issue tracker and there's the full incident complete with diagnosis.

Re: GitHub partial outage

#69

Post by geoffbp »

numbsafari wrote: post #20

Anyone using GitLab have any insight on how well their operations are running these days? We originally left GitLab for GitHub after being bit by a major outage that resulted in data loss. Our code was saved, but we lost everything else. But that was almost 10 years ago at this point.

We’re using gitlab, loads of issues and outages, we want to go to github

Re: GitHub partial outage

#70

Post by whoknowsidont »

stackskipton wrote: post #32
danfritz wrote: post #2

Related to the recent announcement they are moving to Azure?

Doubt it. I'm Ops person on Azure, while they just had terrible outage recently, they tend to be as stable as any other cloud provider and I haven't had many issues with Azure itself compared to whatever slop the devs are chucking into production.

>they tend to be as stable as any other cloud provider

Absolutely not.

Post reply on HN