A terrible, horrible, no-good, very bad day at Slack • Hacker News Board

Re: A terrible, horrible, no-good, very bad day at Slack

#231

Post by 1e-9 » Wed, Jul 08, 2020, 2:04 AM UTC

SilasX wrote: post #204

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why ? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Al…

Those who have read it to insatiable toddlers --- sometimes 5 or 10 readings in a row --- get the reference.

Re: A terrible, horrible, no-good, very bad day at Slack

#232

Post by dahfizz » Wed, Jul 08, 2020, 2:05 AM UTC

vondur wrote: post #194

m12k wrote: post #26

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances i…

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

Microservices are really cool™

Re: A terrible, horrible, no-good, very bad day at Slack

#233

Post by RonanTheGrey » Wed, Jul 08, 2020, 2:43 AM UTC

matiasfernandez wrote: post #221

"I'm still not understanding why it's so hard to display the birthday date on the settings page. Why can't we get this done this quarter?" Look, I'm sorry, we've been over this. It's the design of our back-end. First there's this thing called the Bingo service. See, Bingo knows everyone's name-o, so we get the user's ID out of there. And from Bingo, we can call Papaya and MBS (Magic Baby Service) to get that user ID…

Complete with indefatigable acronyms and insane naming conventions.

This guy engineerings.

Re: A terrible, horrible, no-good, very bad day at Slack

#234

Post by [deleted] » Wed, Jul 08, 2020, 2:43 AM UTC

Nextgrid wrote: post #223

vondur wrote: post #194

Earlier quoted context omitted.

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

Which is what always happens when the company aims to build an "engineering playground" full of microservices and other moving parts without a clear technical justification and balancing the pros & cons and why I personally don't like working on such projects - it makes me feel uneasy not having a good understanding of the entire system. To be fair to Slack, at their scale, lots of moving parts might make sense, but…

[deleted]

Re: A terrible, horrible, no-good, very bad day at Slack

#235

Post by whatsmyusername » Wed, Jul 08, 2020, 2:52 AM UTC

jessaustin wrote: post #195

whatsmyusername wrote: post #178

Earlier quoted context omitted.

You could, and I'm sure some places have a business case for this. Most need stuff going back at least a little bit. If I was talking to you about something on Friday and wanted to reference the conversation Monday I'd be real mad if the convo was already deleted.

I think I'm confused... there are complaints above that Discord doesn't save transcripts. That seems equivalent to zero-day retention? I haven't used Discord so I'm probably misunderstanding something.

If I scroll back in discord I can go back to the first message I ever sent to the servers I hang out on.

In slack you can configure a retention period for files, messages, or both. Anything older than that gets wiped.

Re: A terrible, horrible, no-good, very bad day at Slack

#236

Post by peterangular » Wed, Jul 08, 2020, 2:55 AM UTC

aeyes wrote: post #123

ketzo wrote: post #11

Earlier quoted context omitted.

Honestly it’s a bit tough for me to parse, but the way I’m reading it, 1. Stale configs led to an overabundance of web apps, and then 2. Old instances of the web app couldn’t be removed because of the consul-template bug. so, yes, a combination (in sequence) of the two. Hard for me to be sure because I’m by no means knowledgeable on this stuff.

Even easier way to understand what happened: - slots full - to update slots with a new host you need an empty slot - hosts went away but updating config was impossible -> errors because config referenced non-existing hosts

Agree but one more:

- monitoring was broke so we didn't learn about it until it was too late

Re: A terrible, horrible, no-good, very bad day at Slack

#237

Post by pradn » Wed, Jul 08, 2020, 3:05 AM UTC

cuu508 wrote: post #27

m12k wrote: post #26

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances i…

That's an excellent TL;DR, thank you! Postmortems should start with a summary paragraph like the above, and then go into story and full details below.

The internal Google postmortem format does require a 1-2 sentence summary. Absolutely vital for being able to learn from mistakes. No one is going to parse walls of text when browsing through post-mortems.

Re: A terrible, horrible, no-good, very bad day at Slack

#238

Post by bigiain » Wed, Jul 08, 2020, 3:15 AM UTC

amedvednikov wrote: post #92

bigiain wrote: post #68

Earlier quoted context omitted.

"640K ought to be enough for anybody!" Eventually, seemingly sane assumptions become anachronistic laughing points. (Even if they're apocryphal...)

This is not a sane assumption, and he never said it, it's a myth.

I disagree, specifically depending on when that assumption might have been made. My first computer came with 1KB of ram, expandable to 16KB. I have no doubt the designers of that and most of their peers at the time made similar assumptions. My second computer has interesting "bank switching" to circumvent the 16 bit database limitations of it's 8bit cpu that could openly address 64KB, and managed to mostly usefully have 128KB of ram in it. I suspect it's designers would have also happily made an assumption about 640KB being "enough for anyone".

(Also, maybe you should look up the definition of "apocryphal? I know he never said it, and strongly alluded to that, and didn't attribute it to Bill for that reason...)

Re: A terrible, horrible, no-good, very bad day at Slack

#239

Post by bigiain » Wed, Jul 08, 2020, 3:28 AM UTC

jjoonathan wrote: post #80

bigiain wrote: post #63

Earlier quoted context omitted.

Slack's problem though had nothing to do with racks going down. I'm not convinced Amazon's team is immune from the sort of complex failure mode described here. 'll bet there's people with equivalent sorts of stories about where edge cases in service interactions (either their own set of Lambda services or the AWS ones behind them, or more likely both) lead to a similar unexpected failure cascade.

> I'm not convinced Amazon's team is immune from the sort of complex failure mode You're being way too kind. Not only is AWS not immune, their autoscalers are often absurdly primitive. Like, hourly cron job doubling / halving within narrow safety rails primitive, where it's not merely possible to find a load that trips it up, it's all but inevitable. This varies by service, but they always project an image of their i…

> You're being way too kind. Not only is AWS not immune, their autoscalers are often absurdly primitive.

Yep. Very much so. Mostly because I don't have enough personal Lambda-specific warstories to feel confident badmouthing it in the context of this discussion thread. But the bits of AWS I do use are certainly not all rainbows and roses...

I have one app/platform I run that basically sits at a few requests an hour for 11 months of the year, then ramps up to well over 100,000 requests a minute between 8am and 11pm for 14 days. Classic ELB (back in the day) needed quite a lot of preemptive poking and fake generated traffic to be able to ramp up capacity fast enough for the beginning of each day (aELB is somewhat better but still needs juggling). We never even got close to getting autoscaling working nicely on the web and app server plane to let it loose in prod with real credit card billing at risk, we just add significantly over provisioned spot instances for our best estimates of yearly growth (and app behaviour changes) for the two weeks instead, and cautiously babysit things for the duration.

It's nice we can do that. It'd be nicer if I didn't have to keep explaining to suits and C*Os why they can't boast own the golf course that they have an autoscaling backend...

Re: A terrible, horrible, no-good, very bad day at Slack

#240

Post by Polylactic_acid » Wed, Jul 08, 2020, 3:33 AM UTC

dahfizz wrote: post #232

vondur wrote: post #194

Earlier quoted context omitted.

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

Microservices are really cool™

This is far from a micro service problem. I work on a medium sized monolithic rails app and regularly I have issues due to not understanding the whole situation. Even when working in a small part of the app I know super well sometimes other devs change things so the way it works in my mind is not how it works anymore.