Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

Post reply on HN
Print view View original

221–230 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#221

Post by matiasfernandez »

"I'm still not understanding why it's so hard to display the birthday date on the settings page. Why can't we get this done this quarter?"

Look, I'm sorry, we've been over this. It's the design of our back-end. First there's this thing called the Bingo service. See, Bingo knows everyone's name-o, so we get the user's ID out of there. And from Bingo, we can call Papaya and MBS (Magic Baby Service) to get that user ID and turn it into a user session token. We can validate those with LNMOP. And then once we have that we can finally pull the users info down from Raccoon.

Re: A terrible, horrible, no-good, very bad day at Slack

#222

Post by yawaramin »

SilasX wrote: post #204

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why ? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Al…

I immediately recognized the reference. There's no law that says pop culture references need to be to something with X amount of popularity. Chill.

Re: A terrible, horrible, no-good, very bad day at Slack

#223

Post by Nextgrid »

vondur wrote: post #194
m12k wrote: post #26

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances i…

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

Which is what always happens when the company aims to build an "engineering playground" full of microservices and other moving parts without a clear technical justification and balancing the pros & cons and why I personally don't like working on such projects - it makes me feel uneasy not having a good understanding of the entire system.

To be fair to Slack, at their scale, lots of moving parts might make sense, but I see a lot of companies (including startups with very few customers) going down the microservices route and exposing themselves to such a risk when there is no major upside beyond giving engineers lots of toys to play with and slapping the "microservices" and related buzzwords on their careers page.

Re: A terrible, horrible, no-good, very bad day at Slack

#224

Post by Nextgrid »

cutemonster wrote: post #192
ublaze wrote: post #14

Earlier quoted context omitted.

Cloud load balancers can be sneakily expensive. Few months ago, we spent a few weeks replacing an ELB with naive client side load balancing via round robin, which saves us > 200k/year. ELBs charge per byte transmitted, which seems reasonable, but can end up really expensive.

> client side load balancing In the browser? Or a mobile app? They send 1 api req to server 1, then 1 to server 2 and so on? What about any session cookies maybe tied to a specific server?

Presumably round-robin DNS. A DNS response would only return a handful of servers, of which the client will itself only pick one at random for the duration of the session.

Now this approach has drawbacks (DNS responses are cached, and the DNS record picked initially by the client will typically be cached until the app/browser is restarted) but if they are acceptable to you then it's an easy, proven solution.

Re: A terrible, horrible, no-good, very bad day at Slack

#225

Post by Nextgrid »

aflag wrote: post #66
jwr wrote: post #36

Earlier quoted context omitted.

> because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours This is based on faith — there might, or might not be a specialized 24/7 devops team who runs these things better than you. My rational mind has trouble accepting things based on faith, which is also…

Though they do have good availability, they are certainly not infallible and have been down for hours or even days. And, when that happens, all you can do is pray they get to it soon. You have no control over when they will make riskier changes or how fast they will be able to respond. If they fail to respect the SLA, the maximum they are going to do is giving your money back for your services. Obviously, there are g…

This is an underrated point. The problem with AWS and those massive services is that they are "alive" and continuously evolving. In most cases it's fine and you don't notice, but when it goes wrong it can affect you and you have no control over when that is likely to happen.

In contrast with in-house infrastructure, you can make your stack as simple or as complex as you'd like depending on your needs (a lot of projects can get away with a handful of physical machines all configured manually, no Terraform/Kubernetes/etc) and you control when you make drastic changes that risk breaking things so you can plan them during a time when downtime would be the least damaging to your business.

Re: A terrible, horrible, no-good, very bad day at Slack

#226

Post by fireflux_ »

matiasfernandez wrote: post #221

"I'm still not understanding why it's so hard to display the birthday date on the settings page. Why can't we get this done this quarter?" Look, I'm sorry, we've been over this. It's the design of our back-end. First there's this thing called the Bingo service. See, Bingo knows everyone's name-o, so we get the user's ID out of there. And from Bingo, we can call Papaya and MBS (Magic Baby Service) to get that user ID…

Reference: https://www.youtube.com/watch?v=y8OnoxKotPQ

I revisit this video every now and then.

Re: A terrible, horrible, no-good, very bad day at Slack

#227

Post by jedberg »

rhizome wrote: post #218
jedberg wrote: post #29

Great writeup. It's cool that they were able to figure it out as quickly as they did, all things considered. If I were brought in as a consultant on this, my first question would be: why are you using a fleet of HAProxies instead of the ALB? I'm not saying that's a bad choice, but I'd want to know why that choice was made. The second question I would ask is what kind of Chaos Engineering they are doing. Are they doin…

I hope all of these questions would be asked only after everything was working again!

Of course. :) These are things I would ask during a post-mortem.

Re: A terrible, horrible, no-good, very bad day at Slack

#229

Post by phaedryx »

SilasX wrote: post #204

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why ? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Al…

Because sometimes things go wrong and you just want to move to Australia.

Re: A terrible, horrible, no-good, very bad day at Slack

#230

Post by SilasX »

yawaramin wrote: post #222
SilasX wrote: post #204

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why ? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Al…

I immediately recognized the reference. There's no law that says pop culture references need to be to something with X amount of popularity. Chill.

There is a general practice of preferring to repeat stuff that's good, though, or which enhances what it's added to.
Post reply on HN