Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

Post reply on HN
Print view View original

251–260 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#251

Post by bstar77 »

Nextgrid wrote: post #223
vondur wrote: post #194

Earlier quoted context omitted.

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

Which is what always happens when the company aims to build an "engineering playground" full of microservices and other moving parts without a clear technical justification and balancing the pros & cons and why I personally don't like working on such projects - it makes me feel uneasy not having a good understanding of the entire system. To be fair to Slack, at their scale, lots of moving parts might make sense, but…

I think you are drawing the wrong conclusions here... Microservices is not the boogyman here. It more likely has to do with speed of development, developer turnover and a plethora of other things that result in insufficient knowledge transfer.

Microservices (like just about anything) can be implemented well or poorly. There's a reason we have sophisticated orchestration solutions like Kubernetes... it exists to tame large scale deployments that has sensible failover processes.

The benefits you get are services that can be scaled independently, deployments that only affect isolated pieces of code, horizontal scaling, dockerized environemnts, etc. All of these advantages should exist in well designed systems, but systems that have been executed hastily will likely have critical problems crop up at some point.

Re: A terrible, horrible, no-good, very bad day at Slack

#252

Post by onefuncman »

SilasX wrote: post #204

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why ? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Al…

Congratulations, you're one of the lucky 10,000!

Don't be a sourpuss because you were exposed to a new meme, that's how echo chambers form.

Re: A terrible, horrible, no-good, very bad day at Slack

#253

Post by rclayton »

jzoch wrote: post #242
vondur wrote: post #194

Earlier quoted context omitted.

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

Its a load balancer bug. Thats it. Everything else functioned totally normally from what I can see. I think people are blowing this out of proportion saying everything is so complex or its all microservices oh no! This exact bug could have bit a large monolithic app running on a VM.

My sentiments exactly.

Re: A terrible, horrible, no-good, very bad day at Slack

#254

Post by edude03 »

harpratap wrote: post #111
jrockway wrote: post #88

I had trouble getting through this article because my internal monologue was screaming "Envoy and xDS wouldn't have this problem". But that's exactly what they decided ;) HAProxy is a little behind the state of the art on "hey I could just ask some server where the backends are", and it shows in this case. (The "slots" are particularly alarming, as is having to restart when backends come and go.) xDS lets you give yo…

Isn't Ambassador doing the same thing? Btw not sure if you read till the end, they are actually in the process of migrating to Envoy.

Ambassador is a Envoy control plane, as in it uses Envoy for doing the data proxying, but it sets it up.

So yes, it is :)

Re: A terrible, horrible, no-good, very bad day at Slack

#255

Post by please-reread »

bstar77 wrote: post #251
Nextgrid wrote: post #223

Earlier quoted context omitted.

Which is what always happens when the company aims to build an "engineering playground" full of microservices and other moving parts without a clear technical justification and balancing the pros & cons and why I personally don't like working on such projects - it makes me feel uneasy not having a good understanding of the entire system. To be fair to Slack, at their scale, lots of moving parts might make sense, but…

I think you are drawing the wrong conclusions here... Microservices is not the boogyman here. It more likely has to do with speed of development, developer turnover and a plethora of other things that result in insufficient knowledge transfer. Microservices (like just about anything) can be implemented well or poorly. There's a reason we have sophisticated orchestration solutions like Kubernetes... it exists to tame…

Did you mean boogeyman? https://www.dictionary.com/browse/boogeyman

Re: A terrible, horrible, no-good, very bad day at Slack

#256

Post by lloydatkinson »

vondur wrote: post #194
m12k wrote: post #26

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances i…

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

Possibly, but I think it's equally if not more likely a result of "agile" and teams not communicating due to product owners and the like not wanting other teams to work together etc.

Re: A terrible, horrible, no-good, very bad day at Slack

#257

Post by arthurcolle »

please-reread wrote: post #255
bstar77 wrote: post #251

Earlier quoted context omitted.

I think you are drawing the wrong conclusions here... Microservices is not the boogyman here. It more likely has to do with speed of development, developer turnover and a plethora of other things that result in insufficient knowledge transfer. Microservices (like just about anything) can be implemented well or poorly. There's a reason we have sophisticated orchestration solutions like Kubernetes... it exists to tame…

Did you mean boogeyman? https://www.dictionary.com/browse/boogeyman

Obviously he means boogeyman. Even on your page, it suggests another spelling. What a useless comment.

Re: A terrible, horrible, no-good, very bad day at Slack

#258

Post by cutemonster »

peterwwillis wrote: post #247
cutemonster wrote: post #187

Earlier quoted context omitted.

What is, according to you, and the SRE book, a good time during the day/week/year to do these real test outages? How much downtime could be ok? For, let's say, a b2b saas They notify the customers first? Like, "we'll a little bit sabotage our, well, your, servers this weekend, to find out if they fail and shutdown completely and cannot start again" :-)

It doesn't have to be a customer impact to bring down a service or parts of a service (if it's designed correctly). But even if it was a customer impact, you can do it at the same time you'd schedule regular maintenance (there's always something that needs maintenance, eventually) and throw up a maintenance page. It's up to the business to define SLAs and SLOs that the customer will be satisfied with.

> do it at the same time you'd schedule regular maintenance ... SLA ... SLO

I'll do that (some day), thanks

Re: A terrible, horrible, no-good, very bad day at Slack

#259

Post by cutemonster »

Nextgrid wrote: post #224
cutemonster wrote: post #192

Earlier quoted context omitted.

> client side load balancing In the browser? Or a mobile app? They send 1 api req to server 1, then 1 to server 2 and so on? What about any session cookies maybe tied to a specific server?

Presumably round-robin DNS. A DNS response would only return a handful of servers, of which the client will itself only pick one at random for the duration of the session. Now this approach has drawbacks (DNS responses are cached, and the DNS record picked initially by the client will typically be cached until the app/browser is restarted) but if they are acceptable to you then it's an easy, proven solution.

Hmm I'd guess they have DNS cnames like api1.x.com and api2 and 3, 4

And then the client picks one, and if that server is offline, picks another

Seems as simple as DNS based? And works with broken server(s)

Re: A terrible, horrible, no-good, very bad day at Slack

#260

Post by boring_twenties »

cutemonster wrote: post #259
Nextgrid wrote: post #224

Earlier quoted context omitted.

Presumably round-robin DNS. A DNS response would only return a handful of servers, of which the client will itself only pick one at random for the duration of the session. Now this approach has drawbacks (DNS responses are cached, and the DNS record picked initially by the client will typically be cached until the app/browser is restarted) but if they are acceptable to you then it's an easy, proven solution.

Hmm I'd guess they have DNS cnames like api1.x.com and api2 and 3, 4 And then the client picks one, and if that server is offline, picks another Seems as simple as DNS based? And works with broken server(s)

Except you don't need that because you can just return all four IP addresses for one record, e.g. api.x.com
Post reply on HN