Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s Incident on 2-22-22

slack.engineering

Post reply on HN
Print view View original

161–170 of 183 posts

Re: Slack’s Incident on 2-22-22

#161

Post by 4ggr0 »

Symbiote wrote: post #154
orf wrote: post #28

Earlier quoted context omitted.

The official EU rules say 22.02.2022, but nobody in Europe would have trouble parsing 22/2/22 or any variation thereof. And the / (or -) separator is indeed used in parts of the EU. It’s the ordering that’s significant, not the separator.

The separator is often a good clue. Dots and dashes strongly imply d-m-y, slashes imply an English date, which might be m/d/y if it is from North America. A mixture is even more likely to be d-m-y, today is 27/4-2022 in Danish handwriting.

> 27/4-2022 in Danish handwriting

I've never seen a date written like this, interesting.

Re: Slack’s Incident on 2-22-22

#162

Post by Robin_Message »

sophiebits wrote: post #149
Robin_Message wrote: post #148

I am confused by the query that had the problem. Specifically, I am confused by why the sharding is done by user id. Even the largest Slack instance probably has under 100,000 users and less than 1000 peak messages per second. That feels to me like it could be served by a single master DB. It feels like that would be a better way to shard. Major downside is the major difference in shard sizes so some management/migra…

Workspaces are not completely independent, such as for the Slack Connect feature. Some more details here: https://slack.engineering/scaling-datastores-at-slack-with-v...

Thanks! Indeed, the variability of size and usage of each instance was the big issue, and then doing features that crossed instances meant they'd be crossing shards whatever they did, so it made sense to fix the variability issue.

(I'm also surprised/reminded how fast Slack grew and how quickly it became effectively ubiquitous — I think every company I've contracted for in the last five years has used Slack).

Re: Slack’s Incident on 2-22-22

#164

Post by tuetuopay »

dormando wrote: post #44

Now a more philosoraptor style comment: I see Mcrib is a service built to quickly detect and replace memcached's. I treat memcached in infrastructure as a very stable service. Meaning it is infrequently necessary to upgrade it, and it will generally not fail on its own. If it does it will be highly infrequent compared to services with higher churn or more complexity/dependencies. This means if they're failing often e…

I think you nailed the real issue that caused the incident: saying "consul down == unhealthy memcached", then evicting the node. If Mcrib instead did some actual applicative healthchecks (e.g. memcached ping), which could be correlated with some system metrics (cpu, ram), it could avoid evicting those perfectly good nodes with a warm cache that just happen to have a restarting consul agent.

Granted, this is easy to say once the incident happened with an excellent postmortem, but this should be an industry-wide wakeup call: don't do this.

I have the same issue at work, where people treat a "prometheus node_exporter down" as a "the app on the machine is down". I've started to add the actual app name in our alerts, and now people don't freak out anymore when they see "down" alerts: oh node_exporter is down, but not the app? Don't panic and calmly check why.

Re: Slack’s Incident on 2-22-22

#165

Post by sofixa »

ethical wrote: post #159

Doesn't anyone learn anything. Youth is not an achievement. Experience is. Having to refer to a book, rather than good change management practices, highlights the madness of agile tosh, and ignorance in terms of capacity and performance management. It's also a data breach incident (denial of service, unavailability), I hope they reported this to the UK ICO, and each countries data protection regulator. Amateurish rub…

> Youth is not an achievement. Experience is.

Old man yells at cloud vibes here.

> It's also a data breach incident (denial of service, unavailability)

How is unavailability a data breach?!

Re: Slack’s Incident on 2-22-22

#166

Post by ethical »

sofixa wrote: post #165
ethical wrote: post #159

Doesn't anyone learn anything. Youth is not an achievement. Experience is. Having to refer to a book, rather than good change management practices, highlights the madness of agile tosh, and ignorance in terms of capacity and performance management. It's also a data breach incident (denial of service, unavailability), I hope they reported this to the UK ICO, and each countries data protection regulator. Amateurish rub…

> Youth is not an achievement. Experience is. Old man yells at cloud vibes here. > It's also a data breach incident (denial of service, unavailability) How is unavailability a data breach?!

1: Yup. 2: Its the legal definition under the GDPR ( a European law, that applies to the US, via Privacy Shield). If you can't get to your data - you get the idea?

TLDR;

Re: Slack’s Incident on 2-22-22

#167

Post by macintux »

ethical wrote: post #159

Doesn't anyone learn anything. Youth is not an achievement. Experience is. Having to refer to a book, rather than good change management practices, highlights the madness of agile tosh, and ignorance in terms of capacity and performance management. It's also a data breach incident (denial of service, unavailability), I hope they reported this to the UK ICO, and each countries data protection regulator. Amateurish rub…

> The notification obligations under the GDPR are only triggered when there is a breach of personal data which is likely to result in a risk to the rights and freedoms of individuals.

https://www.lexology.com/library/detail.aspx?g=03e8a988-7c9e...

Re: Slack’s Incident on 2-22-22

#168

Post by ransom1538 »

muglug wrote: post #128
ransom1538 wrote: post #124

Earlier quoted context omitted.

"I treat memcached in infrastructure as a very stable service." I run memcached at a large scale. You are totally right. Every other year we will find ONE bad memcached node down. We use nutcraker instead of mcrouter for consistent hashing to each memcache node. Once i read "We also run a control plane for the cache tier, called Mcrib. Mcrib’s role is to generate up-to-date Mcrouter configurations" -- I was like oooo…

> I run memcached at a large scale I don't believe you run it at the scale Slack does. The people at Slack who decided to use Mcrouter (and created Mcrib) have experience running Memcached, Mcrouter and Nutcracker in production at two of the biggest web properties in the world. Trust that they know whereof they speak.

"I don't believe you run it at the scale Slack does."

Definitely not. We host about %80 of elementary schools in the US. Not slack scale but definitely face many of the same issues :/

Re: Slack’s Incident on 2-22-22

#169

Post by bognition »

_jal wrote: post #131
bognition wrote: post #88

Earlier quoted context omitted.

It’s likely that the memcached install is so large that the underlying instances themselves are failing. When you have hundreds or thousands of instances, failures in the instances themselves become pretty regular.

I don't see this. I have thousands of long-lived instances - full VMs, not containers, running in our hardware. If they start "going bad", something is wrong. That's a signal I wouldn't want to ignore. It has happened - once an HBA in a storage node was causing occasional corruption, another time due to a communication failure people were building things with the wrong version of something which had a memory leak and…

Failure rates in AWS are probably higher than what you're seeing in your own hardware.

Re: Slack’s Incident on 2-22-22

#170

Post by iamcal »

Robin_Message wrote: post #148

I am confused by the query that had the problem. Specifically, I am confused by why the sharding is done by user id. Even the largest Slack instance probably has under 100,000 users and less than 1000 peak messages per second. That feels to me like it could be served by a single master DB. It feels like that would be a better way to shard. Major downside is the major difference in shard sizes so some management/migra…

> Even the largest Slack instance probably has under 100,000 users and less than 1000 peak messages per second.

This is not true, by an order of magnitude.

Post reply on HN