Everyone do your part! I took the rest of the week off to give GitHub a chance to recover. Flatten the peaks, let GitHub migrate
GitHub incident: 2022/03/24
71–80 of 103 posts
Re: GitHub incident: 2022/03/24
#72Earlier quoted context omitted.
I’m a big fan of GitHub Actions so take this with that in mind. GitLab (self-hosted) is very reliable however GitLab CI/CD is very different in design to GitHub Actions: it’s much more a bunch of shell scripts stuck together with YAML. Personally, I’d never go back (unless they redesign it).
Personally, I hate Github Actions from the bottom of my heart. Ever since Travis CI sold its soul and I've been forced to use Github Actions, and I've come to hate every single moment I have to work with some broken or buggy action, with the inability to restart only failed jobs (they added this vital feature only a few weeks ago, 2+ years after launching actions), with slow job spin-up times, which are only slightly…
… yeah, we just moved some stuff to Actions, and it's been rather disappointing as onboarding goes. While the YAML represents jobs as a DAG, it seems like Actions can't actually evaluate them as such, and it inserts dependencies where none exist, i.e., it inserts stalls in the pipeline. Action builds aren't cached, so they're slow. The default token can't be extended to allow cross-repository access, forcing you to write your own app that can. The jobs, not the workflow, is the CI status/result, so you can't branch protect on a workflow. (You have to just have a dummy job … but you can't have an empty job, no, it has to run `true`…)
Re: GitHub incident: 2022/03/24
#73After this latest incident I brought it up to my company that we should start considering alternatives if this situation doesn't improve. From an outsiders perspective, it looks like all the new features they recently introduced seem to be crippling their databases. Microsoft needs to get a grip on this situation as the parent company. Their golden goose acquisition is about to become persona non grata if these outag…
How are you doing cost analysis between moving off of GitHub and working around the issues?
Re: GitHub incident: 2022/03/24
#74From their earlier posts it sounds like they're encountering some kind of MySQL performance issue, which in my (horrible) experience can be extremely difficult for your jack of all trades software engineer or SRE to troubleshoot. I would hope a company Github's size would have MySQL expertise on staff, but if not I will say a prayer for the poor souls who are feverishly reading the Percona blog and trying to decide w…
This is what happens when your company (notoriously) moves away from having meritocracy as a core value: you put people in charge of things who don't have the expertise to run them very well.
https://www.businessinsider.com/githubs-ceo-ditches-meritocr...
Re: GitHub incident: 2022/03/24
#75After the incident yesterday I sent an email to GH sales to talk about moving to on-prem enterprise so we don't have to go down with the rest of the boat. Still waiting on that callback/reply. Starting to wonder if Microsoft even wants our money anymore.
Not even a full day since your email and you're making wild assumptions?
Re: GitHub incident: 2022/03/24
#76From their earlier posts it sounds like they're encountering some kind of MySQL performance issue, which in my (horrible) experience can be extremely difficult for your jack of all trades software engineer or SRE to troubleshoot. I would hope a company Github's size would have MySQL expertise on staff, but if not I will say a prayer for the poor souls who are feverishly reading the Percona blog and trying to decide w…
This is what happens when your company (notoriously) moves away from having meritocracy as a core value: you put people in charge of things who don't have the expertise to run them very well. https://www.businessinsider.com/githubs-ceo-ditches-meritocr...
Pretty sure that rug wouldn't have prevented downtime.
Re: GitHub incident: 2022/03/24
#77From their earlier posts it sounds like they're encountering some kind of MySQL performance issue, which in my (horrible) experience can be extremely difficult for your jack of all trades software engineer or SRE to troubleshoot. I would hope a company Github's size would have MySQL expertise on staff, but if not I will say a prayer for the poor souls who are feverishly reading the Percona blog and trying to decide w…
This is what happens when your company (notoriously) moves away from having meritocracy as a core value: you put people in charge of things who don't have the expertise to run them very well. https://www.businessinsider.com/githubs-ceo-ditches-meritocr...
What did they move away to?
Re: GitHub incident: 2022/03/24
#78Everyone do your part! I took the rest of the week off to give GitHub a chance to recover. Flatten the peaks, let GitHub migrate
[deleted]
Re: GitHub incident: 2022/03/24
#79After this latest incident I brought it up to my company that we should start considering alternatives if this situation doesn't improve. From an outsiders perspective, it looks like all the new features they recently introduced seem to be crippling their databases. Microsoft needs to get a grip on this situation as the parent company. Their golden goose acquisition is about to become persona non grata if these outag…
Used to work at BigBank$ where they had an internal 8 node Bit bucket cluster. Down multiple times a week for 6+ months before they could finally get it under control (Jira + CI was apparently putting a lot of load on and apparently BitBucket doesn't scale past 8 nodes)
Needless to say, self hosting can be more reliable but you'll probably end up dedicating a lot of resources to building that out and supporting it
Re: GitHub incident: 2022/03/24
#80After this latest incident I brought it up to my company that we should start considering alternatives if this situation doesn't improve. From an outsiders perspective, it looks like all the new features they recently introduced seem to be crippling their databases. Microsoft needs to get a grip on this situation as the parent company. Their golden goose acquisition is about to become persona non grata if these outag…
Used to work at BigBank$ where they had an internal 8 node Bit bucket cluster. Down multiple times a week for 6+ months before they could finally get it under control (Jira + CI was apparently putting a lot of load on and apparently BitBucket doesn't scale past 8 nodes) Needless to say, self hosting can be more reliable but you'll probably end up dedicating a lot of resources to building that out and supporting it
Single VM (backed up) with gitlab, k8s for workers.
Not difficult, not hard to maintain. Worked fine for our ~500 projects (each with pipelines/ci|cd) and ~100 devs.