Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

Post reply on HN
Print view View original

211–220 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#211

Post by SilasX »

nikanj wrote: post #206
SilasX wrote: post #204

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why ? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Al…

This book sold well enough to score a tv series, a disney movie, a musical, and a theater play. How much more successful do you need it to be? Odds seem very high that a kid growing up in the past 50 years was exposed to this story and phrase

I wasn’t. Or if I was, it obviously didn’t make an impression, and properly so: combining random negative words isn’t creative.

Re: A terrible, horrible, no-good, very bad day at Slack

#212

Post by [deleted] »

staticassertion wrote: post #203
vondur wrote: post #194

Earlier quoted context omitted.

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

Feels like you've just described every single bug ever.

[deleted]

Re: A terrible, horrible, no-good, very bad day at Slack

#213

Post by mattyb678 »

SilasX wrote: post #204

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why ? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Al…

A couple points. Just because you personally don't remember or know the book does not mean that it is not memorable. Also, it could be that those who weren't aware of the book used it after seeing it used elsewhere.

Re: A terrible, horrible, no-good, very bad day at Slack

#214

Post by jgwil2 »

SilasX wrote: post #204

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why ? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Al…

That's just, like, your opinion man.

Re: A terrible, horrible, no-good, very bad day at Slack

#215

Post by debt »

SilasX wrote: post #204

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why ? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Al…

I think they think it's cute and they're attempting to trivialize their loss.

Re: A terrible, horrible, no-good, very bad day at Slack

#217

Post by debt »

m12k wrote: post #26

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances i…

TL;DR meta point is: "The reason that we haven’t been doing any significant work on this HAProxy stack is that we’re moving towards Envoy Proxy for all of our ingress load-balancing"

The legacy system supporting Slack in production was heavily resource-constrained as they were moving to a new fancy system. Slack admits here that the legacy system likely wasn't getting the attention it needed and lo-and-behold it started failing in mysterious ways.

Organizational failure by not properly calculating all the risks caused by rotating out part of their load-balancing system. They probably should've asked for more budget here to keep their existing system functional as they slowly transitioned to their new system.

They admit that COVID caused all their systems to become stressed, they probably had appropriately budgeted for the transition to Envoy whenever they asked management(probably pre-covid). The team likely was never meant to support both the load they're now seeing during COVID while transitioning to a new system.

Either way during any transition, there's a period where you must support both systems at full capacity until the legacy system can be gracefully decommissioned.

Re: A terrible, horrible, no-good, very bad day at Slack

#218

Post by rhizome »

jedberg wrote: post #29

Great writeup. It's cool that they were able to figure it out as quickly as they did, all things considered. If I were brought in as a consultant on this, my first question would be: why are you using a fleet of HAProxies instead of the ALB? I'm not saying that's a bad choice, but I'd want to know why that choice was made. The second question I would ask is what kind of Chaos Engineering they are doing. Are they doin…

I hope all of these questions would be asked only after everything was working again!

Re: A terrible, horrible, no-good, very bad day at Slack

#219

Post by narag »

SilasX wrote: post #211
nikanj wrote: post #206

Earlier quoted context omitted.

This book sold well enough to score a tv series, a disney movie, a musical, and a theater play. How much more successful do you need it to be? Odds seem very high that a kid growing up in the past 50 years was exposed to this story and phrase

I wasn’t. Or if I was, it obviously didn’t make an impression, and properly so: combining random negative words isn’t creative.

It seems there's some kind of "all your base are belong to us" weird resonance for it. I had my own explanation why it works, but the only proof that's really needed is that it became a meme.

Re: A terrible, horrible, no-good, very bad day at Slack

#220

Post by yawaramin »

cuu508 wrote: post #27
m12k wrote: post #26

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances i…

That's an excellent TL;DR, thank you! Postmortems should start with a summary paragraph like the above, and then go into story and full details below.

BLUF style–Bottom Line Up Front. Often used by the military for field action reports.
Post reply on HN