Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub incident: 2022/03/24

githubstatus.com

Post reply on HN
Print view View original

81–90 of 103 posts

Re: GitHub incident: 2022/03/24

#81

Post by infamouscow »

kragen wrote: post #74
yazboo wrote: post #46

From their earlier posts it sounds like they're encountering some kind of MySQL performance issue, which in my (horrible) experience can be extremely difficult for your jack of all trades software engineer or SRE to troubleshoot. I would hope a company Github's size would have MySQL expertise on staff, but if not I will say a prayer for the poor souls who are feverishly reading the Percona blog and trying to decide w…

This is what happens when your company (notoriously) moves away from having meritocracy as a core value: you put people in charge of things who don't have the expertise to run them very well. https://www.businessinsider.com/githubs-ceo-ditches-meritocr...

There are lots of exceptional individual contributors with no desire to ever go into management.

Re: GitHub incident: 2022/03/24

#82

Post by radicality »

peeters wrote: post #62
AndrewOMartin wrote: post #37

What we need is some kind of distributed version control system, then this would never happen. Ideally it'd be a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency, but I think that's just unrealistic.

I hear this sort of remark a lot, but does anyone actually practice this? Like git is great in that there a myriad of options for syncing remotes, but none that I've ever seen come close to having a central clone that also acts as the highest authority. Having used git prior to Github et al, where I had remotes set up to literally each collaborator that they individually hosted, I would never want that user experienc…

I’m with you, I don’t even want the decentralised model, but it feels like with GitHub I get… the worst of both worlds? No decentralisation and crappy centralisation?

For example, in a work environment, what’s even the point of ‘forking’ in GitHub. I don’t want my own version of any repo, I don’t want to be playing weird rebase/merge upstream/push origin nonsense. I don’t even want to be creating branches or naming branches. I just want a centralised repo: I make a commit and… that’s it, get a commit hash and be able to have that commit anywhere else I check out the repo, and open a PR/diff to merge into master.

Re: GitHub incident: 2022/03/24

#84

Post by [deleted] »

gtirloni wrote: post #75
bob1029 wrote: post #13

After the incident yesterday I sent an email to GH sales to talk about moving to on-prem enterprise so we don't have to go down with the rest of the boat. Still waiting on that callback/reply. Starting to wonder if Microsoft even wants our money anymore.

Not even a full day since your email and you're making wild assumptions?

[deleted]

Re: GitHub incident: 2022/03/24

#85

Post by bob1029 »

gtirloni wrote: post #75
bob1029 wrote: post #13

After the incident yesterday I sent an email to GH sales to talk about moving to on-prem enterprise so we don't have to go down with the rest of the boat. Still waiting on that callback/reply. Starting to wonder if Microsoft even wants our money anymore.

Not even a full day since your email and you're making wild assumptions?

How long should I expect to wait for someone else to take more of my money, or at least answer 3 simple pre-sales questions?

If I dont hear back from GitHub enterprise sales before the end of the week, I am going to take this as a strong hint that we are too small for microsoft to care about, suggesting that we would be safer & happier on a different vendor's product.

Re: GitHub incident: 2022/03/24

#86

Post by skwirl »

raybb wrote: post #5

How does GitLab compare in terms of it's CI/CD reliability?

GitLab has its share of downtime. I don't use GitHub regularly so I can't really compare the two, but you will occasionally have a few hours where GitLab is down. Maybe 2 or 3 times per year?

Re: GitHub incident: 2022/03/24

#87

Post by MrJohz »

nijave wrote: post #79
_fat_santa wrote: post #11

After this latest incident I brought it up to my company that we should start considering alternatives if this situation doesn't improve. From an outsiders perspective, it looks like all the new features they recently introduced seem to be crippling their databases. Microsoft needs to get a grip on this situation as the parent company. Their golden goose acquisition is about to become persona non grata if these outag…

Used to work at BigBank$ where they had an internal 8 node Bit bucket cluster. Down multiple times a week for 6+ months before they could finally get it under control (Jira + CI was apparently putting a lot of load on and apparently BitBucket doesn't scale past 8 nodes) Needless to say, self hosting can be more reliable but you'll probably end up dedicating a lot of resources to building that out and supporting it

I've never tried to self-host something like that, but on the other hand, I've never worked at a place where the internal GitLab wasn't a shaky mess, particularly when CI or Pages or certificates is involved.

I believe people when they tell me they've managed it without major problems, but clearly they're the 10x developers of the self-hosted world, because the people I work with seem to find it hard - and that is not a slight on them, because clearly even Github can't do so much better!

Re: GitHub incident: 2022/03/24

#88

Post by gtirloni »

bob1029 wrote: post #85
gtirloni wrote: post #75

Earlier quoted context omitted.

Not even a full day since your email and you're making wild assumptions?

How long should I expect to wait for someone else to take more of my money, or at least answer 3 simple pre-sales questions? If I dont hear back from GitHub enterprise sales before the end of the week, I am going to take this as a strong hint that we are too small for microsoft to care about, suggesting that we would be safer & happier on a different vendor's product.

A few days to a week is normal for enterprise sales.

Re: GitHub incident: 2022/03/24

#89

Post by [deleted] »

meibo wrote: post #58
floodle wrote: post #53

Earlier quoted context omitted.

GitHub before Microsoft?

GitHub before Microsoft notoriously shied away from innovating/iterating on their core business for years. We had half as many features before then. IMO feature wise, GitHub really has improved tenfold. I remember having to fight third party CIs and project trackers, now we're running them all on GitHub and it's a well-integrated pleasure. If it does work, which it usually does and hopefully goes back to doing. I als…

[deleted]

Re: GitHub incident: 2022/03/24

#90

Post by outworlder »

llamataboot wrote: post #64
yazboo wrote: post #46

From their earlier posts it sounds like they're encountering some kind of MySQL performance issue, which in my (horrible) experience can be extremely difficult for your jack of all trades software engineer or SRE to troubleshoot. I would hope a company Github's size would have MySQL expertise on staff, but if not I will say a prayer for the poor souls who are feverishly reading the Percona blog and trying to decide w…

I agree that getting deep into the weeds on some of that stuff can be taxing on a smaller development team with a few senior generalists (of which I tend to be one) but I'm quite sure that companies at github scale have deep levels of performance expertise - still not always easy of course, because lots of these types of things only come up at some certain scale

Do not immediately assume that they do. Only those who have management that recognizes the need to have those experts even have a shot at getting them.

If they do have the required staff it still might not be readily available due to org chart boundaries.

Unclear if GitHub has the staff or if they are able to draw from the larger Microsoft pool.

If they keep having issues I expect Microsoft to push them to move everything to MSSQL

Post reply on HN