Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s Incident on 2-22-22

slack.engineering

Post reply on HN
Print view View original

121–130 of 183 posts

Re: Slack’s Incident on 2-22-22

#121

Post by true_religion »

dllthomas wrote: post #30
true_religion wrote: post #26

Earlier quoted context omitted.

It’s not really that hard. Like the imperial system, Americans just memorize how it works as children and don’t think about it anymore. Think about it like speaking a different language, except with numbers and not words.

The complaint isn't about the particular other order, but the fact that the order is ambiguous. In this case that doesn't matter, but often it does. Americans memorize inches and yards, and often also memorize centimeters and meters, and working with either is fine, but we're not so often faced with numbers where it might be inches or centimeters and we have to figure out which (and when we are, it's sometimes a pain…

I instinctively felt that the ambiguity ought to not truly exist, but I wanted to find some backing. According to this: https://en.wikipedia.org/wiki/Date_format_by_country

… all the major predominantly English speaking counties will use mostly hyphens in the dd-mm-yyyy format. So although there is ambiguity, it’s easily resolved by picking that as the default mentally and only back tracking on failure.

Now in the more general case, this whole thing feels like a lieutenant/leftenant situation. We are annoyed simply because it’s not they way that we do things in a peculiar case, when otherwise the language is fully intelligible.

Re: Slack’s Incident on 2-22-22

#122

Post by jdlshore »

mulmen wrote: post #36
jdlshore wrote: post #10

Earlier quoted context omitted.

This sort of handwavy conspiracy thinking is distressingly common. What basis do you have for your suspicion? Is it just “big company bad”?

I choose to believe it is not common thinking but instead commonly verbalized among the minority with such thoughts.

You're probably right. I've noticed a trend of people fantasizing out loud in the past 3-5 years. It's nearly always cynical / conspiracy-theory fantasies, and they're stated as fact (or near-fact, as GP did), but without any backing information or logic... just pure fantasizing.

But, thinking back, it's not that common. Hopefully just a fad. People expressing their frustration at inequality and fear of corporate dystopia.

Re: Slack’s Incident on 2-22-22

#123

Post by allannienhuis »

sverhagen wrote: post #93
ascar wrote: post #57

Earlier quoted context omitted.

> Europeans only got it a little better. There is a reasonable argument for little endian dates (as in the least significant information is usually the most relevant as it changes most often), but apart from the "it has been like this forever" I don't see any reasonable argument for middle endian date formats. Then again, the US is notoriously resistant to the metric system too.

So, European in the US, here. I switch my dates stubbornly to DD-MM-YYYY, 'cause that's the only way. Of course I would. But then there's so many US applications that don't adhere to my settings and use MM-DD-YYYY. So then I am still deciphering 05-07-2020-kind of stuff. All. The. Freakin'. Time. :)

I sometimes format dates in documents or emails dd-MMM-yyyy when the audience is international. ie 2-Feb-2022 using the short month form disambiguates the fields and I think avoids mental gymnastics like 'what month is 09 again?' for the reader. (or in my case the finger-counting....)

Re: Slack’s Incident on 2-22-22

#124

Post by ransom1538 »

dormando wrote: post #44

Now a more philosoraptor style comment: I see Mcrib is a service built to quickly detect and replace memcached's. I treat memcached in infrastructure as a very stable service. Meaning it is infrequently necessary to upgrade it, and it will generally not fail on its own. If it does it will be highly infrequent compared to services with higher churn or more complexity/dependencies. This means if they're failing often e…

"I treat memcached in infrastructure as a very stable service."

I run memcached at a large scale. You are totally right. Every other year we will find ONE bad memcached node down. We use nutcraker instead of mcrouter for consistent hashing to each memcache node. Once i read "We also run a control plane for the cache tier, called Mcrib. Mcrib’s role is to generate up-to-date Mcrouter configurations" -- I was like oooooh boy, here we go....

Knowing memcache is a rock comes with experience though.

Re: Slack’s Incident on 2-22-22

#125

Post by brigandish »

missedthecue wrote: post #22
diarrhea wrote: post #11

That date format is actually the worst I have ever encountered. m-d-y, with year in 2 digits, numbers not zero-padded, US "order" yet using dashes. It's like a moderator of /r/ISO8601 came up with the worst possible format on purpose . Am I missing something?

I mean, there is literally no way to confuse it with another date, unless you go back 100 years, when Slack didn't exist. There is no 22nd month, so we know the 22s are the day and the year, leaving only the 2 to be the month. Is it really that difficult to parse?

It's a waste of effort and makes me wonder about the competence of the person who wrote it (when looking at a mangled date generally). Display dates are for humans so write the month name, then it doesn't matter what the order is.

22 Feb 22

Feb 22 22 (weird but still better)

22 22 Feb (very weird but still better)

This also goes to show that 2022 is a better choice. My own personal preference - the 22nd of February 2022.

Re: Slack’s Incident on 2-22-22

#126

Post by RandallBrown »

mulmen wrote: post #46
rjh29 wrote: post #42

Earlier quoted context omitted.

It can't be either, 22 is not a valid month. I agree with you though, the point of a date like yyyy-mm-dd is to avoid working out stuff like this. You don't pick a date format based on whether the current date is ambiguous or not.

Good catch. I updated my post. The question remains, how can this format be disambiguated? Agreed, this is why ISO8601 exists.

It's disambiguated by context, like most ambiguities in language. This is the title of a blog post after all.

Re: Slack’s Incident on 2-22-22

#127

Post by xyzzy_plugh »

dormando wrote: post #95
bognition wrote: post #88

Earlier quoted context omitted.

It’s likely that the memcached install is so large that the underlying instances themselves are failing. When you have hundreds or thousands of instances, failures in the instances themselves become pretty regular.

I can say with certainty this isn't strictly true. The failures should be relatively rare; when I say relatively I mean on the level of natural node failure. If natural node failure isn't survivable without special systems to quickly replace downed nodes you don't actually have an N+1 redundancy system. Thus, the pools aren't large enough :) Or, in this case, if they really are failing this much then having them alwa…

> The failures should be relatively rare; when I say relatively I mean on the level of natural node failure.

And exactly how rare do you believe this to be?

In my experience, node failures at scale of hundreds to thousands of nodes are monthly to weekly, if not daily. Generally speaking, stability is a normal distribution. Young, new instances experience similar failure rates as old instances. If you have any sort of maximum node lifetime (for example, a week) or scale dynamically on a daily basis then you'll see a lot of failures.

Re: Slack’s Incident on 2-22-22

#128

Post by muglug »

ransom1538 wrote: post #124
dormando wrote: post #44

Now a more philosoraptor style comment: I see Mcrib is a service built to quickly detect and replace memcached's. I treat memcached in infrastructure as a very stable service. Meaning it is infrequently necessary to upgrade it, and it will generally not fail on its own. If it does it will be highly infrequent compared to services with higher churn or more complexity/dependencies. This means if they're failing often e…

"I treat memcached in infrastructure as a very stable service." I run memcached at a large scale. You are totally right. Every other year we will find ONE bad memcached node down. We use nutcraker instead of mcrouter for consistent hashing to each memcache node. Once i read "We also run a control plane for the cache tier, called Mcrib. Mcrib’s role is to generate up-to-date Mcrouter configurations" -- I was like oooo…

> I run memcached at a large scale

I don't believe you run it at the scale Slack does.

The people at Slack who decided to use Mcrouter (and created Mcrib) have experience running Memcached, Mcrouter and Nutcracker in production at two of the biggest web properties in the world.

Trust that they know whereof they speak.

Re: Slack’s Incident on 2-22-22

#129

Post by perydell »

This is very transparent and a good write-up. I wonder if someone at Slack could explain how they calculate their downtime on their status page. This outage was for 3 hours and 14 minutes but they claim 99.79% uptime for the month of February.

Re: Slack’s Incident on 2-22-22

#130

Post by selcuka »

lilyball wrote: post #72
ascar wrote: post #57

Earlier quoted context omitted.

> Europeans only got it a little better. There is a reasonable argument for little endian dates (as in the least significant information is usually the most relevant as it changes most often), but apart from the "it has been like this forever" I don't see any reasonable argument for middle endian date formats. Then again, the US is notoriously resistant to the metric system too.

It's because it matches the way we speak dates aloud. When intended for human consumption, sortability and big-endianness doesn't matter, but matching the way we speak does. Maybe other cultures actually speak dates differently, I don't know, but I have never seen a native English speaker habitually speak dates any differently than "January 1st, 2001". All that said, I definitely agree with the original complaint, m-…

I don't think how we speak dates is relevant. We say "hundred and fifty dollars" but still write it as "$150".
Post reply on HN