Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s Incident on 2-22-22

slack.engineering

Post reply on HN
Print view View original

131–140 of 183 posts

Re: Slack’s Incident on 2-22-22

#131

Post by _jal »

bognition wrote: post #88
dormando wrote: post #44

Now a more philosoraptor style comment: I see Mcrib is a service built to quickly detect and replace memcached's. I treat memcached in infrastructure as a very stable service. Meaning it is infrequently necessary to upgrade it, and it will generally not fail on its own. If it does it will be highly infrequent compared to services with higher churn or more complexity/dependencies. This means if they're failing often e…

It’s likely that the memcached install is so large that the underlying instances themselves are failing. When you have hundreds or thousands of instances, failures in the instances themselves become pretty regular.

I don't see this. I have thousands of long-lived instances - full VMs, not containers, running in our hardware.

If they start "going bad", something is wrong. That's a signal I wouldn't want to ignore.

It has happened - once an HBA in a storage node was causing occasional corruption, another time due to a communication failure people were building things with the wrong version of something which had a memory leak and would eventually summon the OOM killer. There have been other issues.

"Have you tried turning it off and back on again" is still a terrible system management strategy.

Re: Slack’s Incident on 2-22-22

#132

Post by tempest_ »

muglug wrote: post #128
ransom1538 wrote: post #124

Earlier quoted context omitted.

"I treat memcached in infrastructure as a very stable service." I run memcached at a large scale. You are totally right. Every other year we will find ONE bad memcached node down. We use nutcraker instead of mcrouter for consistent hashing to each memcache node. Once i read "We also run a control plane for the cache tier, called Mcrib. Mcrib’s role is to generate up-to-date Mcrouter configurations" -- I was like oooo…

> I run memcached at a large scale I don't believe you run it at the scale Slack does. The people at Slack who decided to use Mcrouter (and created Mcrib) have experience running Memcached, Mcrouter and Nutcracker in production at two of the biggest web properties in the world. Trust that they know whereof they speak.

You may not be wrong, in fact you are very likely right, but this is not an argument.

The larger an org gets the more likely it is to do weird things to mitigate organizational difficulties be them budget, human or otherwise.

Those types of things rarely show up in postmortems for obvious reasons.

Re: Slack’s Incident on 2-22-22

#133

Post by anonu »

pierrebai wrote: post #12

Sometimes new roll-out causes outage, sometimes, roll-out are delayed due to the overall system architecture. Reading the post-mortem, I could not help but be reminded of this issue as described here: https://www.youtube.com/watch?v=y8OnoxKotPQ

One of my favorite videos. And yes, slack is way more complex than I could ever imagine

Re: Slack’s Incident on 2-22-22

#134

Post by selcuka »

perydell wrote: post #129

This is very transparent and a good write-up. I wonder if someone at Slack could explain how they calculate their downtime on their status page. This outage was for 3 hours and 14 minutes but they claim 99.79% uptime for the month of February.

It is possible that approximately 50% of all users were affected, or the requests timed out 50% of the time.

Re: Slack’s Incident on 2-22-22

#136

Post by hw »

oxfordmale wrote: post #3

Which major outage ? According to the Slack uptime, there was barely 1.5 hour of outage :-) P.S. Yes I know the uptime is decide by committee, and doesn't reflect reality. I am just being cynical.

1.5 hours for a tool like Slack is major. Lots of productivity lost (or gained depending on how you view Slack) and thus $ impacts at companies that heavily rely on Slack for internal/team comms

Re: Slack’s Incident on 2-22-22

#137

Post by philderbeast »

dijonman2 wrote: post #76
vxNsr wrote: post #70

Earlier quoted context omitted.

The fact that the current top comment thread is quibbling about the date format in the title seems to agree with this assessment, if there was anything real to complain about that’s what we’d be seeing, instead we get bikeshedding on the date format in the title of a post.

Which is aligned with US date formatting.

which only makes it confusing for the rest of the world

Re: Slack’s Incident on 2-22-22

#138

Post by sandGorgon »

dormando wrote: post #44

Now a more philosoraptor style comment: I see Mcrib is a service built to quickly detect and replace memcached's. I treat memcached in infrastructure as a very stable service. Meaning it is infrequently necessary to upgrade it, and it will generally not fail on its own. If it does it will be highly infrequent compared to services with higher churn or more complexity/dependencies. This means if they're failing often e…

more likely - they are using "spot instances" for memcached, which will cause them to be evicted fairly frequently.

Re: Slack’s Incident on 2-22-22

#139

Post by JohnTHaller »

philderbeast wrote: post #137
dijonman2 wrote: post #76

Earlier quoted context omitted.

Which is aligned with US date formatting.

which only makes it confusing for the rest of the world

It's not that confusing if you know there are only 12 months. If it happened to occur on 2-4-22, then it would be pretty confusing.

Re: Slack’s Incident on 2-22-22

#140

Post by prescriptivist »

sandGorgon wrote: post #138
dormando wrote: post #44

Now a more philosoraptor style comment: I see Mcrib is a service built to quickly detect and replace memcached's. I treat memcached in infrastructure as a very stable service. Meaning it is infrequently necessary to upgrade it, and it will generally not fail on its own. If it does it will be highly infrequent compared to services with higher churn or more complexity/dependencies. This means if they're failing often e…

more likely - they are using "spot instances" for memcached, which will cause them to be evicted fairly frequently.

Or horizontal autoscaling based on demand.
Post reply on HN