Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub incident: 2022/03/24

githubstatus.com

Post reply on HN
Print view View original

91–100 of 103 posts

Re: GitHub incident: 2022/03/24

#91

Post by rezonant »

danogentili wrote: post #56
phphphphp wrote: post #15

Earlier quoted context omitted.

I’m a big fan of GitHub Actions so take this with that in mind. GitLab (self-hosted) is very reliable however GitLab CI/CD is very different in design to GitHub Actions: it’s much more a bunch of shell scripts stuck together with YAML. Personally, I’d never go back (unless they redesign it).

Personally, I hate Github Actions from the bottom of my heart. Ever since Travis CI sold its soul and I've been forced to use Github Actions, and I've come to hate every single moment I have to work with some broken or buggy action, with the inability to restart only failed jobs (they added this vital feature only a few weeks ago, 2+ years after launching actions), with slow job spin-up times, which are only slightly…

I've been very happy with CircleCI

Re: GitHub incident: 2022/03/24

#92

Post by rezonant »

radicality wrote: post #82
peeters wrote: post #62

Earlier quoted context omitted.

I hear this sort of remark a lot, but does anyone actually practice this? Like git is great in that there a myriad of options for syncing remotes, but none that I've ever seen come close to having a central clone that also acts as the highest authority. Having used git prior to Github et al, where I had remotes set up to literally each collaborator that they individually hosted, I would never want that user experienc…

I’m with you, I don’t even want the decentralised model, but it feels like with GitHub I get… the worst of both worlds? No decentralisation and crappy centralisation? For example, in a work environment, what’s even the point of ‘forking’ in GitHub. I don’t want my own version of any repo, I don’t want to be playing weird rebase/merge upstream/push origin nonsense. I don’t even want to be creating branches or naming b…

I agree though that its hard to find use cases for forks in a typical business- however Unreal Engine would be a great example of it where the private repository is shared by multiple development teams. If you have access to it (because you've agreed to the Source License), you can make a private fork and send in a PR to the main repository.

Epic probably won't merge it though unless you represent a big studio.

Re: GitHub incident: 2022/03/24

#93

Post by stnmtn »

rezonant wrote: post #91
danogentili wrote: post #56

Earlier quoted context omitted.

Personally, I hate Github Actions from the bottom of my heart. Ever since Travis CI sold its soul and I've been forced to use Github Actions, and I've come to hate every single moment I have to work with some broken or buggy action, with the inability to restart only failed jobs (they added this vital feature only a few weeks ago, 2+ years after launching actions), with slow job spin-up times, which are only slightly…

I've been very happy with CircleCI

CircleCI has had multiple outages for us this past week, causing a lot of frustration.

Re: GitHub incident: 2022/03/24

#94

Post by bberenberg »

nijave wrote: post #79
_fat_santa wrote: post #11

After this latest incident I brought it up to my company that we should start considering alternatives if this situation doesn't improve. From an outsiders perspective, it looks like all the new features they recently introduced seem to be crippling their databases. Microsoft needs to get a grip on this situation as the parent company. Their golden goose acquisition is about to become persona non grata if these outag…

Used to work at BigBank$ where they had an internal 8 node Bit bucket cluster. Down multiple times a week for 6+ months before they could finally get it under control (Jira + CI was apparently putting a lot of load on and apparently BitBucket doesn't scale past 8 nodes) Needless to say, self hosting can be more reliable but you'll probably end up dedicating a lot of resources to building that out and supporting it

Great talk on this topic from 2015 by the team from Netflix: https://www.youtube.com/watch?v=aWcKsX0_qAg

Re: GitHub incident: 2022/03/24

#95

Post by throwaway89645 »

pzduniak wrote: post #55
iratewizard wrote: post #19

Earlier quoted context omitted.

They got patent trolled and flinched. They removed the things that made Skype best in class and turned it into just another low quality corporate messenger.

Was it really due to that lawsuit? IIRC the main issue with P2P was that you could discover anyone's IP address by knowing their Skype username. I recall that people used this to DoS people in multiplayer video games - they looked up accounts with roughly the same name as the in-game one and proceeded to drop their connection. It was especially common in League of Legends.

Yep - It was bad enough that it was common practice for people to proxy their skypes through a third party.

(It was pretty nice that Skype let you do that really easily though)

Re: GitHub incident: 2022/03/24

#96

Post by akshayshah »

llamataboot wrote: post #64
yazboo wrote: post #46

From their earlier posts it sounds like they're encountering some kind of MySQL performance issue, which in my (horrible) experience can be extremely difficult for your jack of all trades software engineer or SRE to troubleshoot. I would hope a company Github's size would have MySQL expertise on staff, but if not I will say a prayer for the poor souls who are feverishly reading the Percona blog and trying to decide w…

I agree that getting deep into the weeds on some of that stuff can be taxing on a smaller development team with a few senior generalists (of which I tend to be one) but I'm quite sure that companies at github scale have deep levels of performance expertise - still not always easy of course, because lots of these types of things only come up at some certain scale

GitHub probably _had_ deep levels of performance expertise, but getting acquired by a megacorporation comes with a big shift in culture. I’d bet that many tenured GitHubbers left, and that there are relatively few people remaining who understand the core systems deeply.

Medium-term, the more closely aligned with the rest of Microsoft’s technology they can become, the better - not many MSFT folks understand the ins and outs of sharded Percona, but many of them do understand SQL Server and .NET.

Re: GitHub incident: 2022/03/24

#97

Post by jen20 »

akshayshah wrote: post #96
llamataboot wrote: post #64

Earlier quoted context omitted.

I agree that getting deep into the weeds on some of that stuff can be taxing on a smaller development team with a few senior generalists (of which I tend to be one) but I'm quite sure that companies at github scale have deep levels of performance expertise - still not always easy of course, because lots of these types of things only come up at some certain scale

GitHub probably _had_ deep levels of performance expertise, but getting acquired by a megacorporation comes with a big shift in culture. I’d bet that many tenured GitHubbers left, and that there are relatively few people remaining who understand the core systems deeply. Medium-term, the more closely aligned with the rest of Microsoft’s technology they can become, the better - not many MSFT folks understand the ins an…

I wouldn't assume SQL Server is the only database MS have core understanding of - they bought Citus Data not so long ago.

Re: GitHub incident: 2022/03/24

#98

Post by akshayshah »

jen20 wrote: post #97
akshayshah wrote: post #96

Earlier quoted context omitted.

GitHub probably _had_ deep levels of performance expertise, but getting acquired by a megacorporation comes with a big shift in culture. I’d bet that many tenured GitHubbers left, and that there are relatively few people remaining who understand the core systems deeply. Medium-term, the more closely aligned with the rest of Microsoft’s technology they can become, the better - not many MSFT folks understand the ins an…

I wouldn't assume SQL Server is the only database MS have core understanding of - they bought Citus Data not so long ago.

Sure, but the pool of Citus folks at Microsoft is relatively small. I’d assume that many Microsoft teams have worked with SQL Server for years, so it’s likely a widely-available skillset internally.

Re: GitHub incident: 2022/03/24

#99

Post by rezonant »

stnmtn wrote: post #93
rezonant wrote: post #91

Earlier quoted context omitted.

I've been very happy with CircleCI

CircleCI has had multiple outages for us this past week, causing a lot of frustration.

Interesting. None of our builds have failed due to CircleCI during that time period, or ever, that I can remember. We do maybe 5-10 builds a day during PST working hours. Not a huge volume but still would think it would have hit our radar. CircleCI's status page, which they are usually pretty good at updating, indicates two outages over the last two months, though one of them was a long one (9 hours?) but apparently only affected remote docker and machine runners. Neither of those were in the last week though.

Re: GitHub incident: 2022/03/24

#100

Post by reasonabl_human »

anaccountexists wrote: post #43
_fat_santa wrote: post #11

After this latest incident I brought it up to my company that we should start considering alternatives if this situation doesn't improve. From an outsiders perspective, it looks like all the new features they recently introduced seem to be crippling their databases. Microsoft needs to get a grip on this situation as the parent company. Their golden goose acquisition is about to become persona non grata if these outag…

My company uses the same DB sharding tech. It took us about a year of daily / weekly outages until we finally were able to fix our performance issues. 256 database splits, lots of cross-shard queries removed, etc before we finally reached a happy-ish state for a year. Now it’s scheduled to fall over in about 6 months and everyone is freaking out again. It’s not new features that are hurting us, it’s the existing core…

Wow. What kind of tech debt are you reckoning with / what better design decisions would you implement if redesigning today?
Post reply on HN