Live data from Hacker News

GitHub incident: 2022/03/24

githubstatus.com

41–50 of 103 posts

Re: GitHub incident: 2022/03/24

#42
post #12

Earlier quoted context omitted.

Isn't that basically what happened to Skype? Microsoft paid $8.5B for Skype when it was on top of the world, and now I don't know a single person who uses it.

When my organization uses a Microsoft video conferencing product, we use Teams, not "Skype for Business". I have no idea why, as I don't schedule the meetings, but I can say that I can't imagine something sucking worse than Teams.

Teams is built on top of Skype for Business, SfB is one of the main backends for them.

Re: GitHub incident: 2022/03/24

#43

After this latest incident I brought it up to my company that we should start considering alternatives if this situation doesn't improve. From an outsiders perspective, it looks like all the new features they recently introduced seem to be crippling their databases. Microsoft needs to get a grip on this situation as the parent company. Their golden goose acquisition is about to become persona non grata if these outag…

My company uses the same DB sharding tech. It took us about a year of daily / weekly outages until we finally were able to fix our performance issues. 256 database splits, lots of cross-shard queries removed, etc before we finally reached a happy-ish state for a year.

Now it’s scheduled to fall over in about 6 months and everyone is freaking out again. It’s not new features that are hurting us, it’s the existing core product line. All the new stuff is built on horizontally sharded DBs from the get-go.

Re: GitHub incident: 2022/03/24

#44

What we need is some kind of distributed version control system, then this would never happen. Ideally it'd be a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency, but I think that's just unrealistic.

My irony meter is currently down for maintenance, but that's precisely what git is, a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency :)

Re: GitHub incident: 2022/03/24

#45
post #12
post #10

https://www.githubstatus.com/ at the time of posting, webhooks and actions are still struggling :( there have now been 23 incident outages in 90 days. if anyone from Microsoft is reading this, you need to start considering this degradation as the companies highest priority. At this point, Gitea, Gitlab and others (self hosted or otherwise) are beginning to gain traction as a more reliable and performant alternative.…

Isn't that basically what happened to Skype? Microsoft paid $8.5B for Skype when it was on top of the world, and now I don't know a single person who uses it.

€5.4bn for Nokia and that didn't work out for them either.

Re: GitHub incident: 2022/03/24

#46
From their earlier posts it sounds like they're encountering some kind of MySQL performance issue, which in my (horrible) experience can be extremely difficult for your jack of all trades software engineer or SRE to troubleshoot.

I would hope a company Github's size would have MySQL expertise on staff, but if not I will say a prayer for the poor souls who are feverishly reading the Percona blog and trying to decide whether to tune the doublewrite buffer or redo log, or both, or neither.

Re: GitHub incident: 2022/03/24

#48
post #5

How does GitLab compare in terms of it's CI/CD reliability?

My company recently switched to github actions from Gitlab CI because gitlab had so much downtime.

Were you using self-hosted runners or the SaaS per-minute ones? My experience with Gitlab.com is probably irrelevant at this point, but years ago I didn't have any issues whatsoever with the combination of a Gitlab.com org with self-hosted runners... on AWS, I think?

Re: GitHub incident: 2022/03/24

#49

What we need is some kind of distributed version control system, then this would never happen. Ideally it'd be a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency, but I think that's just unrealistic.

My irony meter is currently down for maintenance, but that's precisely what git is, a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency :)

I think that was the point OP was making, through a layer of thick sarcasm.
Post reply on HN