Live data from Hacker News

Slack’s Incident on 2-22-22

slack.engineering

151–160 of 183 posts

Re: Slack’s Incident on 2-22-22

#151
post #120
post #70

Earlier quoted context omitted.

The fact that the current top comment thread is quibbling about the date format in the title seems to agree with this assessment, if there was anything real to complain about that’s what we’d be seeing, instead we get bikeshedding on the date format in the title of a post.

[adds "bikeshedding" to vocabulary.]

This is awesome because we had a bikeshedding image floating around Slack's Slack instance that was posted with enough frequency that I remember it.

Re: Slack’s Incident on 2-22-22

#152
post #118

Earlier quoted context omitted.

It's because it matches the way we speak dates aloud. When intended for human consumption, sortability and big-endianness doesn't matter, but matching the way we speak does. Maybe other cultures actually speak dates differently, I don't know, but I have never seen a native English speaker habitually speak dates any differently than "January 1st, 2001". All that said, I definitely agree with the original complaint, m-…

Given that so many (all?) other English speaking nations including the UK usually speak it the other way around and write it day-month-year, I wonder if writing (especially thinking about newspapers here) influenced the way you speak it and not the other way around. March 1st saves space and ink over 1st of March or some other rationale. Someone certainly has already investigated the origin of putting the month first…

Old documents from the UK use both formats. There could be a European influence, but it would be from 150 or more years ago.

http://anguline.co.uk/cert/birth.html

Re: Slack’s Incident on 2-22-22

#153
post #66

Just my personal take, I think this is a really well-written incident postmortem. It's specific, extensive, candid, and dare I say, entertaining? Many incident reports are fully lacking in any meaningful detail, or wholly unapologetic. I actually enjoyed learning tidbits about the author, in particular their mention of https://how.complexsystems.fail/ . Reading this boosted my confidence in Slack's teams, which shoul…

The named authors on the blog post are an amazing set of engineers and the type that would have added a lot of introspection and expertise to what happened while maintaining a high-level of lightheartedness. I spent a number of years at Slack and was involved in a postmortem in my first week as an Engineering Manager. I was impressed by how excellent practices were borrowed from Etsy in the early days and then magnified by Google's practices. As someone who once had to run these sessions and distill the learnings in to the wee hours of the morning this is great to hear.

Re: Slack’s Incident on 2-22-22

#154
post #28
post #23

Earlier quoted context omitted.

For the whole of Europe it would be 22.02.2022, how is all of Europe statistically insignificant?

The official EU rules say 22.02.2022, but nobody in Europe would have trouble parsing 22/2/22 or any variation thereof. And the / (or -) separator is indeed used in parts of the EU. It’s the ordering that’s significant, not the separator.

The separator is often a good clue. Dots and dashes strongly imply d-m-y, slashes imply an English date, which might be m/d/y if it is from North America.

A mixture is even more likely to be d-m-y, today is 27/4-2022 in Danish handwriting.

Re: Slack’s Incident on 2-22-22

#155
post #108

Earlier quoted context omitted.

European date format doesn’t make sense either. It still doesn’t sort and the little-endian date is composed of big-endian year, month, and day. Putting the day first isn’t actually a benefit because you still need the year and month for context. Any favoritism of the EU format is the same as the US format. Just familiarity. ISO8601 is the only way.

I agree that the European format is probably not more useful, and you probably convinced me to go change my settings to YYYY-MM-DD. But I _do_ think that the European format makes more _sense_, it being in chronological "magnitude" order.

I can understand that perspective, although I maintain the usage is only preferred because of familiarity.

Since you are already on the fence with ISO8601 I invite you to consider time of day. Would you use second:minute:hour? That is also in (reverse!) “chronological magnitude” order.

Re: Slack’s Incident on 2-22-22

#156

Earlier quoted context omitted.

I can say with certainty this isn't strictly true. The failures should be relatively rare; when I say relatively I mean on the level of natural node failure. If natural node failure isn't survivable without special systems to quickly replace downed nodes you don't actually have an N+1 redundancy system. Thus, the pools aren't large enough :) Or, in this case, if they really are failing this much then having them alwa…

> The failures should be relatively rare; when I say relatively I mean on the level of natural node failure. And exactly how rare do you believe this to be? In my experience, node failures at scale of hundreds to thousands of nodes are monthly to weekly, if not daily. Generally speaking, stability is a normal distribution. Young, new instances experience similar failure rates as old instances. If you have any sort of…

Which still means you could implement a hard limit of 1 fail per hour and only allow more replacements with manual intervention. With a thousand nodes, several or hundreds failing within a few hours is so unlikely that you're probably better off preventing automatic failover in these cases.

But that generally mirrors my experience that automatic failover for stable software tends to cause more issues than it solves. A good (i.e. redundant hardware and software) Postgresql server is also so unlikely to fail that wrong detection and cascading issues from automatic failover are more likely than its actual benefits.

Re: Slack’s Incident on 2-22-22

#157
post #66

Just my personal take, I think this is a really well-written incident postmortem. It's specific, extensive, candid, and dare I say, entertaining? Many incident reports are fully lacking in any meaningful detail, or wholly unapologetic. I actually enjoyed learning tidbits about the author, in particular their mention of https://how.complexsystems.fail/ . Reading this boosted my confidence in Slack's teams, which shoul…

I think a well written post-mortem is an excellent recruiting tool.

Re: Slack’s Incident on 2-22-22

#158

That date format is actually the worst I have ever encountered. m-d-y, with year in 2 digits, numbers not zero-padded, US "order" yet using dashes. It's like a moderator of /r/ISO8601 came up with the worst possible format on purpose . Am I missing something?

I know it's somewhat trivial but it does bother me a bit too, because I am used to the dashes being an indicator that ISO 8601 is being used. If you're going to use a nonstandard format, I'd much prefer it not look like a standard one.

Re: Slack’s Incident on 2-22-22

#159
Doesn't anyone learn anything. Youth is not an achievement. Experience is. Having to refer to a book, rather than good change management practices, highlights the madness of agile tosh, and ignorance in terms of capacity and performance management. It's also a data breach incident (denial of service, unavailability), I hope they reported this to the UK ICO, and each countries data protection regulator. Amateurish rubbish.

Re: Slack’s Incident on 2-22-22

#160
post #28
post #23

Earlier quoted context omitted.

For the whole of Europe it would be 22.02.2022, how is all of Europe statistically insignificant?

The official EU rules say 22.02.2022, but nobody in Europe would have trouble parsing 22/2/22 or any variation thereof. And the / (or -) separator is indeed used in parts of the EU. It’s the ordering that’s significant, not the separator.

> nobody in Europe would have trouble

Yeah, I guess if people look at it and parse it, they understand it. But what I notice is, that the IT-bubble I am in has no worries parsing these dates because we use them a lot in IT, even in Europe. But people outside of IT do seem confused about US-formatted dates from time to time, because they rarely encounter them.

More than once did I notice someone struggling to fill in their birthday into an online form because the people making the form decided to use M/D/Y instead of D.M.Y Of course most can help themselves, but its not like these date formats seem natural or normal to everyone in Europe.

Post reply on HN