Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s Incident on 2-22-22

slack.engineering

Post reply on HN
Print view View original

171–180 of 183 posts

Re: Slack’s Incident on 2-22-22

#171

Post by iamcal »

ransom1538 wrote: post #124
dormando wrote: post #44

Now a more philosoraptor style comment: I see Mcrib is a service built to quickly detect and replace memcached's. I treat memcached in infrastructure as a very stable service. Meaning it is infrequently necessary to upgrade it, and it will generally not fail on its own. If it does it will be highly infrequent compared to services with higher churn or more complexity/dependencies. This means if they're failing often e…

"I treat memcached in infrastructure as a very stable service." I run memcached at a large scale. You are totally right. Every other year we will find ONE bad memcached node down. We use nutcraker instead of mcrouter for consistent hashing to each memcache node. Once i read "We also run a control plane for the cache tier, called Mcrib. Mcrib’s role is to generate up-to-date Mcrouter configurations" -- I was like oooo…

Our underlying hardware (AWS) is nothing like this reliable. We see regular (several times a year) failure of racks of machines or whole DCs.

Across the whole fleet (all services), we lose 1-10 servers per day as a baseline. Major events are then on top of that and can impact thousand of hosts at once.

Re: Slack’s Incident on 2-22-22

#172

Post by dllthomas »

true_religion wrote: post #121
dllthomas wrote: post #30

Earlier quoted context omitted.

The complaint isn't about the particular other order, but the fact that the order is ambiguous. In this case that doesn't matter, but often it does. Americans memorize inches and yards, and often also memorize centimeters and meters, and working with either is fine, but we're not so often faced with numbers where it might be inches or centimeters and we have to figure out which (and when we are, it's sometimes a pain…

I instinctively felt that the ambiguity ought to not truly exist, but I wanted to find some backing. According to this: https://en.wikipedia.org/wiki/Date_format_by_country … all the major predominantly English speaking counties will use mostly hyphens in the dd-mm-yyyy format. So although there is ambiguity, it’s easily resolved by picking that as the default mentally and only back tracking on failure. Now in the mo…

> … all the major predominantly English speaking counties will use mostly hyphens in the dd-mm-yyyy format.

Yes, which makes reasonable the complaint about mm-dd-yyy.

> So although there is ambiguity, it’s easily resolved by picking that as the default mentally and only back tracking on failure.

This is both more work and also error prone in the general case (although it works out fine in this case).

> Now in the more general case, this whole thing feels like a lieutenant/leftenant situation.

Not at all. Whether I read "lieutenant" or "leftenant", I know what you're talking about. If I read 2-10-23, I might miss your birthday party.

Re: Slack’s Incident on 2-22-22

#174

Post by bugeats »

diarrhea wrote: post #11

That date format is actually the worst I have ever encountered. m-d-y, with year in 2 digits, numbers not zero-padded, US "order" yet using dashes. It's like a moderator of /r/ISO8601 came up with the worst possible format on purpose . Am I missing something?

The incident was caused by a database query that was "2-22-22" when it was supposed to be "2-22-22".

Re: Slack’s Incident on 2-22-22

#175

Post by [deleted] »

bognition wrote: post #169
_jal wrote: post #131

Earlier quoted context omitted.

I don't see this. I have thousands of long-lived instances - full VMs, not containers, running in our hardware. If they start "going bad", something is wrong. That's a signal I wouldn't want to ignore. It has happened - once an HBA in a storage node was causing occasional corruption, another time due to a communication failure people were building things with the wrong version of something which had a memory leak and…

Failure rates in AWS are probably higher than what you're seeing in your own hardware.

[deleted]

Re: Slack’s Incident on 2-22-22

#176

Post by xyzzy_plugh »

dx034 wrote: post #156
xyzzy_plugh wrote: post #127

Earlier quoted context omitted.

> The failures should be relatively rare; when I say relatively I mean on the level of natural node failure. And exactly how rare do you believe this to be? In my experience, node failures at scale of hundreds to thousands of nodes are monthly to weekly, if not daily. Generally speaking, stability is a normal distribution. Young, new instances experience similar failure rates as old instances. If you have any sort of…

Which still means you could implement a hard limit of 1 fail per hour and only allow more replacements with manual intervention. With a thousand nodes, several or hundreds failing within a few hours is so unlikely that you're probably better off preventing automatic failover in these cases. But that generally mirrors my experience that automatic failover for stable software tends to cause more issues than it solves.…

I think you're looking at it the wrong way. A server is never just postgres or memcached, there's always other stuff running, and it's that other stuff that can cause problems. Like maybe you're patching the fleet and a node fails to come back up, or due to misconfiguration the disk gets full.

I'd argue that stable systems are actually worse for operational stability as you become complacent and comfortable and when shit hits the fan you are unprepared.

Re: Slack’s Incident on 2-22-22

#177

Post by dijonman2 »

8n4vidtmkvmk wrote: post #150
JohnTHaller wrote: post #139

Earlier quoted context omitted.

It's not that confusing if you know there are only 12 months. If it happened to occur on 2-4-22, then it would be pretty confusing.

its particularly bad because US dates are written with slashes. please leave dashes for ISO so we have some hope of understanding anything.

Just because you don’t seem to like it does not mean the US date format is invalid. It should be followed while in the US.

Re: Slack’s Incident on 2-22-22

#178

Post by _jal »

bognition wrote: post #169
_jal wrote: post #131

Earlier quoted context omitted.

I don't see this. I have thousands of long-lived instances - full VMs, not containers, running in our hardware. If they start "going bad", something is wrong. That's a signal I wouldn't want to ignore. It has happened - once an HBA in a storage node was causing occasional corruption, another time due to a communication failure people were building things with the wrong version of something which had a memory leak and…

Failure rates in AWS are probably higher than what you're seeing in your own hardware.

Maybe. If you don't look, you don't know.

But given the number of people I've heard using "we're on AWS, out of my control" as an excuse, this appears to be an unofficial service they offer.

Re: Slack’s Incident on 2-22-22

#179

Post by philderbeast »

dijonman2 wrote: post #177
8n4vidtmkvmk wrote: post #150

Earlier quoted context omitted.

its particularly bad because US dates are written with slashes. please leave dashes for ISO so we have some hope of understanding anything.

Just because you don’t seem to like it does not mean the US date format is invalid. It should be followed while in the US.

Or you could use a sane, unambiguous date format regardless of where you are in the world like 22 Feb 2022.

Re: Slack’s Incident on 2-22-22

#180

Post by dijonman2 »

philderbeast wrote: post #179
dijonman2 wrote: post #177

Earlier quoted context omitted.

Just because you don’t seem to like it does not mean the US date format is invalid. It should be followed while in the US.

Or you could use a sane, unambiguous date format regardless of where you are in the world like 22 Feb 2022.

This is devolving into tribalism.
Post reply on HN