GitHub incident 2022-03-23 • Hacker News Board

Re: GitHub incident 2022-03-23

#131

Post by ManWith2Plans » Wed, Mar 23, 2022, 5:24 PM UTC

edgyquant wrote: post #22

nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold.…

I will say that for us this is a huge deal. We're a devops services company, and our customers expect their deployment pipelines to work. This is becoming a huge pain-point for a few of our customers and we recommended Github Actions to them. A couple of our customers want us to move away from GitHub actions because of how disruptive outages have been.

Re: GitHub incident 2022-03-23

#132

Post by AlexandrB » Wed, Mar 23, 2022, 5:27 PM UTC

TheRealPomax wrote: post #125

ejb999 wrote: post #120

Earlier quoted context omitted.

Every place I have ever worked reported incidents going down would be good, not up.

Every place I ever worked at understood that if you x3 the codebase/infra/interaction surface/etc, you can expect x3 errors. If the total number of errors don't go up as you grow you're doing amazing, and if they go down even though you're landing more and more code for more and more features and subproducts, you have a genuine miracle.

These features can't be rolled out incrementally to users? In this day and age it seems weird for a web app to do a global go-live with something before testing it with a smaller group first.

Re: GitHub incident 2022-03-23

#133

Post by annexrichmond » Wed, Mar 23, 2022, 5:28 PM UTC

frjalex wrote: post #107

Looking at the "GitHub" prefix in the title, I was half-expecting this to point to a report explaining the outage a week ago... But rest assured, it is a new outage!

I thought it was going to be a Postmortem. I couldn't have been more wrong!

Re: GitHub incident 2022-03-23

#134

Post by eckza » Wed, Mar 23, 2022, 5:28 PM UTC

mirekrusin wrote: post #121

What's the best crowdsourced status monitor?

https://outage.bingo/

Re: GitHub incident 2022-03-23

#135

Post by shepardrtc » Wed, Mar 23, 2022, 5:30 PM UTC

djbusby wrote: post #93

shepardrtc wrote: post #79

Earlier quoted context omitted.

IIS Server had/has a memory leak in worker threads that many years ago always forced us to restart the server every few days. Starting in 6.0, they added worker thread recycling and made it a mandatory to choose a time period for every thread to be recycled. Why fix the error when you can just restart the service?

Apache prefork had that since forever. Seems just a garbage collect type pattern.

It's not a bug, it's a pattern.

Seriously though, IIS 5.0 had no worker recycling. There was no method to fix the issue. Threads would eat up GB's of memory until you killed them.

Re: GitHub incident 2022-03-23

#136

Post by mbesto » Wed, Mar 23, 2022, 5:30 PM UTC

belter wrote: post #39

nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

Excluding ones reported as [Errors], [Scheduled] or [Notifications] 2019 -> 39 Incidents 2020 -> 67 Incidents 2021 -> 86 Incidents 2022 -> 20 Incidents so far Edit: Using Linear Regression...Prediction for total end 2022: 111 Incidents.

The number of incidents isn't so much of a problem as the amount of downtime is. That would be more interesting to see.

Re: GitHub incident 2022-03-23

#137

Post by [deleted] » Wed, Mar 23, 2022, 5:32 PM UTC

xtracto wrote: post #81

etimberg wrote: post #4

The quality of GH seems to be slipping

Funny that it happened since they were acquired by Microsoft... reminds me of Hotmail, Skype, LinkedIn, Rare, among several others.

[deleted]

Re: GitHub incident 2022-03-23

#138

Post by zrail » Wed, Mar 23, 2022, 5:32 PM UTC

zeku wrote: post #118

kleebeesh wrote: post #88

Earlier quoted context omitted.

...What? $10 x 1000 = $10k / month. $10k x 12 = $120k. That is a new grad software engineer salary in any US city. You'd pay more than that for a single dev with the devops and security experience to keep GHE running and patched for 1000 devs.

Just a bone to pick... new grad engineers in my US city started around 60-70k in 2018 when my college cohort graduated. Southern US...

Things have changed considerably over the last four years.

Re: GitHub incident 2022-03-23

#139

Post by speedgoose » Wed, Mar 23, 2022, 5:33 PM UTC

mrkramer wrote: post #106

belter wrote: post #39

Earlier quoted context omitted.

Excluding ones reported as [Errors], [Scheduled] or [Notifications] 2019 -> 39 Incidents 2020 -> 67 Incidents 2021 -> 86 Incidents 2022 -> 20 Incidents so far Edit: Using Linear Regression...Prediction for total end 2022: 111 Incidents.

One would thought when they got acquired by Microsoft that the number of incidents would go down considering all resources Microsoft would provide but no.

GitHub has a lot more features now though. A few years ago you didn’t have GitHub actions or workspaces, mostly a DDoS from Asia once in a while.

Re: GitHub incident 2022-03-23

#140

Post by ransom1538 » Wed, Mar 23, 2022, 5:34 PM UTC

nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

"21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive."

Oh stop the drama. Fine. Setup your gitlab.