Live data from Hacker News

Slack’s Incident on 2-22-22

slack.engineering

41–50 of 183 posts

Re: Slack’s Incident on 2-22-22

#41

Earlier quoted context omitted.

Apparently it generates configurations for Mcrouter. Could be MemCache-Router Instance Borker.

I think you meant Broker, but the misspelling is an act of genius since we are talking about downtime caused by an infrastructure failure.

And I misread it as McBorker and now I can't stop chuckling.

Re: Slack’s Incident on 2-22-22

#42
post #38

Earlier quoted context omitted.

looks unambiguous to me lol

Ok, what is the format string? a) m-dd-yy b) m-yy-dd I can't tell. How do you disambiguate?

It can't be either, 22 is not a valid month.

I agree with you though, the point of a date like yyyy-mm-dd is to avoid working out stuff like this. You don't pick a date format based on whether the current date is ambiguous or not.

Re: Slack’s Incident on 2-22-22

#43
Hi! I'd like to offer some hopefully useful information if any Slack folks end up reading this, or anyone else with a similar infrastructure. I'll start with some tech and make a separate philosophical comment.

Also caveat: I have no deep view into Slack's infrastructure so anything I say here may not even be relevant. YMMV.

First some self promotion: https://github.com/memcached/memcached/wiki/Proxy memcached itself is shipping router/proxy software. Mcrouter is difficult to manage and unsupported. This proxy is community developed, more flexible, likely faster, and will support more native features of memcached. We're currently in a stabilization round ensuring it won't eat pets but all of the basic features have been in for a while. Documentation and example libraries are still needed but community feedback help speed those up tremendously (or any kind of question/help request).

It's not clear to me why memcached is being managed like this; mcrouter seems to only be used to abstract the configuration from the clients. It has a lot of features for redundant pools and so on. Especially with what sounds like globally immutable data and the threat of cascading failures during rolling upgrades it sounds like it would be very helpful here.

If cost or pool sizes are the main reasons why the structure is flat, using Extstore (https://github.com/memcached/memcached/wiki/Extstore) can likely help. Even if object value sizes are in the realm of 500 bytes, using flash storage can still greatly reduce the amount of RAM necessary or reduce the pool size (granted the network can still keep up) with nearly identical performance. Extstore takes a lot of tradeoffs (ie; keeping keys in RAM) to ensure most operations don't actually write to flash or double-read. Extstore's in use in tons of places and everyone's immediately addicted.

Finally, the Meta Protocol (https://github.com/memcached/memcached/wiki/MetaCommands) can help with stampeding herds to help keep DB load from exploding without adding excess network roundtrips under normal conditions. I've seen lots of workarounds people build but this protocol extension gives a lot of flexibility you can use to help survive degraded states: anti-stampeding herd, serve-stale, better counter semantics, and so on.

Re: Slack’s Incident on 2-22-22

#44
Now a more philosoraptor style comment: I see Mcrib is a service built to quickly detect and replace memcached's. I treat memcached in infrastructure as a very stable service. Meaning it is infrequently necessary to upgrade it, and it will generally not fail on its own. If it does it will be highly infrequent compared to services with higher churn or more complexity/dependencies. This means if they're failing often enough that you need to rapidly detect and replace them you have a more fundamental problem.

From a structural standpoint I think my technical comment can be useful. If things really are failing this much A) you should figure out why and slow that down. B) if you have a generally stable system and understand the typical rate of failure, you can add tripwires into Mcrib to avoid over-culling services and loudly raise alarms. Then C) you can improve technical reliability with redundancy/extstore/etc.

I've also seen plenty of times where folks have a dependency of a service determine if that service is usable, which I disagree with quite strongly. Consul being down on a node should trigger something to consider if the service is dead. It's important both for reliability (don't kill perfectly working things because you end up having to design around it), and for maintainability as you've now made people afraid of upgrading Consul or other co-dependent services. Other similar failures are single-point-of-testing availability checking where instead you probably want two points of truth before shooting a service.

Now you risk people being afraid of upgrading probably anything, which means they will work around it, abstract it, or needlessly replace it with something they feel safer managing. The latter is at best a waste of time, at worst a time bomb until you find out what conditions this new thing breaks under.

This isn't advocating that you design without assuming anything can fail anywhere at any time; just pointing out that how often a service _should_ fail is extremely useful information when designing systems and designing fail safes, alerts, monitoring, etc.

Re: Slack’s Incident on 2-22-22

#45

Earlier quoted context omitted.

Quoted post unavailable.

This sort of handwavy conspiracy thinking is distressingly common. What basis do you have for your suspicion? Is it just “big company bad”?

No post body was provided.

Re: Slack’s Incident on 2-22-22

#46
post #42
post #38

Earlier quoted context omitted.

Ok, what is the format string? a) m-dd-yy b) m-yy-dd I can't tell. How do you disambiguate?

It can't be either, 22 is not a valid month. I agree with you though, the point of a date like yyyy-mm-dd is to avoid working out stuff like this. You don't pick a date format based on whether the current date is ambiguous or not.

Good catch. I updated my post. The question remains, how can this format be disambiguated?

Agreed, this is why ISO8601 exists.

Re: Slack’s Incident on 2-22-22

#47

Earlier quoted context omitted.

I mean, there is literally no way to confuse it with another date, unless you go back 100 years, when Slack didn't exist. There is no 22nd month, so we know the 22s are the day and the year, leaving only the 2 to be the month. Is it really that difficult to parse?

> leaving only the 2 to be the day I think you meant "to be the month" there. qed■

Beautiful.

Re: Slack’s Incident on 2-22-22

#48
post #34

Earlier quoted context omitted.

The complaint isn't about the particular other order, but the fact that the order is ambiguous. In this case that doesn't matter, but often it does. Americans memorize inches and yards, and often also memorize centimeters and meters, and working with either is fine, but we're not so often faced with numbers where it might be inches or centimeters and we have to figure out which (and when we are, it's sometimes a pain…

Context matters in your pasta scenario.

Since the text itself doesn't clarify, context is the only way of resolving any of the scenarios. In each case it's usually sufficient and often not all that hard. But it's always harder than if the system in use was made explicit, and I understand the complaint (even if my annoyance at the ambiguity is quite significantly below the level where I would have complained myself, particularly in this case).

Re: Slack’s Incident on 2-22-22

#49

That date format is actually the worst I have ever encountered. m-d-y, with year in 2 digits, numbers not zero-padded, US "order" yet using dashes. It's like a moderator of /r/ISO8601 came up with the worst possible format on purpose . Am I missing something?

I expected the top comment on hackernews to be something this pedantic and irrelevant to the content, and I was not disappointed
Post reply on HN