What's the best crowdsourced status monitor?
GitHub incident 2022-03-23
121–130 of 206 posts
Re: GitHub incident 2022-03-23
#122Looking at the "GitHub" prefix in the title, I was half-expecting this to point to a report explaining the outage a week ago... But rest assured, it is a new outage!
Oh I thought it was about the one from yesterday :)
Re: GitHub incident 2022-03-23
#123Earlier quoted context omitted.
IIS Server had/has a memory leak in worker threads that many years ago always forced us to restart the server every few days. Starting in 6.0, they added worker thread recycling and made it a mandatory to choose a time period for every thread to be recycled. Why fix the error when you can just restart the service?
I doubt they use IIS
MSer here, yes we do… for some things
Re: GitHub incident 2022-03-23
#124https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.
And whom pays for fixing it? Downtimes of self hosted systems using external software can be far longer. GitHub, unlike Amazon and friends, doesn't lie about their downtime. Every saas has hundreds of downtime instances across the board every month. Some are small enough you don't see them. Yet the services still work exceptionally well - and when they don't they get fixed in a quick manner. What takes them an hour w…
> GitHub, unlike Amazon and friends, doesn't lie about their downtime.
Are you kidding? The last 2 incidents were called "degraded performance". Where "degraded" meant I would get nothing but 500 errors accessing GitHub.com either via browser or git itself for the duration of the outage. How is this not lying?
Re: GitHub incident 2022-03-23
#125Earlier quoted context omitted.
Only if you believe those numbers mean anything. What are the errors for ? Github has been adding lots of features and subproducts over the years, becoming a bigger and bigger platform as a result. What you want is the error-per-component, which may very well have actually gone down, with error spikes coming from "when github adds a completely new feature and it goes through a slew of incidents in its first year". Th…
Every place I have ever worked reported incidents going down would be good, not up.
Every place I ever worked at understood that if you x3 the codebase/infra/interaction surface/etc, you can expect x3 errors. If the total number of errors don't go up as you grow you're doing amazing, and if they go down even though you're landing more and more code for more and more features and subproducts, you have a genuine miracle.
Re: GitHub incident 2022-03-23
#126Looking at the "GitHub" prefix in the title, I was half-expecting this to point to a report explaining the outage a week ago... But rest assured, it is a new outage!
[deleted]
Re: GitHub incident 2022-03-23
#127Looking at the "GitHub" prefix in the title, I was half-expecting this to point to a report explaining the outage a week ago... But rest assured, it is a new outage!
Oh I thought it was about the one from yesterday :)
Are their CI/CD toys that shiny that people still willingly choose them even with all the issues?
I find myself regularly asking this — about every major SaaS used for critical ops stuff like this.
Re: GitHub incident 2022-03-23
#128Earlier quoted context omitted.
I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold.…
Also, looking at this it seems like GitHub isn't doing the common SaaS thing of just lying on their status page. Many providers, both internal and external, would look a lot worse if they had honest status pages.
They do intentionally or not lie about this on their status page.
From December 25th to December 31st 2021, Github actions had network problems almost every single day for hours and the status page was green out through out that period.
Same thing also happened few months back.
It feels like they do this manually and it's only done when enough people are effected.
Re: GitHub incident 2022-03-23
#129https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.
We run Gitea at my company. In fact, we forked it. It could reeeaaaalllly use a rewrite. If anyone is even mildly ambitious about creating a new alternative to Github/Gitea, it's a great time to do that.
You might be interested in sourcehut: https://sr.ht
Re: GitHub incident 2022-03-23
#130Earlier quoted context omitted.
Well, I think I have said that since 2020 [0] and it is self-evident that you are better off self-hosting your own Git repo. If you can host a website you can do it. If GNOME, ReactOS, Wireguard, Linux Kernel Project, Mozilla, etc can do it, so can you. Or even use it as a backup / failsafe just in case. But going 'all in' on GitHub just doesn't make any sense anymore. [0] https://hn.algolia.com/?dateRange=all&page=1…
But who can host a website? I would be wary of hosting something that isn’t a 100% static site, out of fear of the amount of attention maintenance would take. Also, quite a few of the non-profits behind the projects you mentioned have multi-million dollar budgets that they can use to administer their git instance, if needed. I don’t think “if they can do it, you can” is a strong argument for those.
My last bill from Hetzner was ~35€. I host gitea, drone CI, hashicorp vault and my own docker registry/pypi repository. I can add as many users as I want, and I had exactly zero incidents in the past ~6 years since I set this up.
I don't even worry about a strong backup strategy (besides just making occasional snapshots of the data volumes) because this was all set up with IaC tools (Terraform, Ansible) and I have copies of all the code in local repositories.