A terrible, horrible, no-good, very bad day at Slack • Hacker News Board

Re: A terrible, horrible, no-good, very bad day at Slack

#261

Post by Nextgrid » Wed, Jul 08, 2020, 11:27 AM UTC

bstar77 wrote: post #251

Nextgrid wrote: post #223

Earlier quoted context omitted.

Which is what always happens when the company aims to build an "engineering playground" full of microservices and other moving parts without a clear technical justification and balancing the pros & cons and why I personally don't like working on such projects - it makes me feel uneasy not having a good understanding of the entire system. To be fair to Slack, at their scale, lots of moving parts might make sense, but…

I think you are drawing the wrong conclusions here... Microservices is not the boogyman here. It more likely has to do with speed of development, developer turnover and a plethora of other things that result in insufficient knowledge transfer. Microservices (like just about anything) can be implemented well or poorly. There's a reason we have sophisticated orchestration solutions like Kubernetes... it exists to tame…

I think that all other things considered equal, a microservices architecture has more moving parts and is more complex to learn than a monolith. The same applies for Kubernetes, it's one extra layer of abstraction and moving parts that you need to understand and keep into account when developing or making changes to an existing system.

I am not saying that microservices is a problem for Slack (at their scale it can make sense), but I am expressing my overall concern about smaller companies going down the same route when their scale or the problem being solved doesn't justify it and they end up having to deal with the (self-inflicted) problems of a massively distributed system with no major upside as well as my personal opinion of why I feel uneasy working on systems where I don't have a full overview of how it works and its potential failure modes.

When it comes to the benefits of microservices I am not sure whether those are all worthwhile considering the overhead and extra complexity of development on a microservices architecture.

In a monolithic application, most data and functions you might need are just a function call away and you typically have one or a handful of databases to interact with, often abstracted away by an ORM. In a microservices architecture, you suddenly need to worry about serialization, authentication and communication between services (and its failure modes, etc) and might require coordinated changes across several services, each of which might use a different language/framework and deployment process.

In terms of getting started, it has always been easier for me to work on a monolith where the codebase makes up for bad/no documentation because my IDE can resolve the majority of the symbols and allows me to see where the data I need lives and where it's being used. In a microservices architecture all of that goes out the window and you need to do a lot more manual "discovery" work searching through the documentation (if there is documentation, which is not a given) and manually figuring out the RPC calls because IDEs typically can't resolve cross-service communications.

Running a monolithic application locally is a lot easier than a microservices architecture. For the former you can typically get away with just a database and cache server all running natively. The latter pretty much imposes a container-based stack where you are now running 10 databases, caches, reverse proxies and everything involved around service discovery, which adds yet another layer of abstraction and makes you spend more time on this useless plumbing than actually getting work done and delivering business value.

Re: A terrible, horrible, no-good, very bad day at Slack

#262

Post by ben509 » Wed, Jul 08, 2020, 11:41 AM UTC

SilasX wrote: post #204

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why ? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Al…

It's unusually memorable because it's from a children's book and little kids will demand their parents read the thing a hundred times. It's either been drilled into you as a child, or as a long suffering parent.

Re: A terrible, horrible, no-good, very bad day at Slack

#263

Post by SilasX » Wed, Jul 08, 2020, 12:09 PM UTC

onefuncman wrote: post #252

SilasX wrote: post #204

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why ? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Al…

Congratulations, you're one of the lucky 10,000! Don't be a sourpuss because you were exposed to a new meme, that's how echo chambers form.

I wasn't exposed to a new meme, I've seen it a hundred times before; the question is why people are so dazzled by it that they keep repeating it when it's not really clever and it doesn't enhance what they're adding it to. It's just a clumsy, extended way of cobbling together negative words to say "very bad".

Re: A terrible, horrible, no-good, very bad day at Slack

#264

Post by devit » Wed, Jul 08, 2020, 12:13 PM UTC

m12k wrote: post #26

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances i…

Seems the root cause is that instead of using an established system like Kubernetes, they rolled their own and didn't engineer and test it as well as such an endeavor would have required.

Re: A terrible, horrible, no-good, very bad day at Slack

#265

Post by astral303 » Wed, Jul 08, 2020, 12:39 PM UTC

Another insight for me was coding the original system to have N slots. This should’ve been a red flag—why is there an arbitrary constraint there? Why allow the system to have a fixed limit there?

If you choose to go down the slots road, then you need to put in alerts and discovery for reaching slot limits—-which means monitoring and tracking them, then setting up alerts.

Re: A terrible, horrible, no-good, very bad day at Slack

#266

Post by a2tech » Wed, Jul 08, 2020, 1:09 PM UTC

jzoch wrote: post #242

vondur wrote: post #194

Earlier quoted context omitted.

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

Its a load balancer bug. Thats it. Everything else functioned totally normally from what I can see. I think people are blowing this out of proportion saying everything is so complex or its all microservices oh no! This exact bug could have bit a large monolithic app running on a VM.

No, its not a load balancer bug. It was a bug with the home built program they built for managing config load ins TO the load balancer. The load balancer did exactly what it was instructed to do--but the slack developed app for helping the load balancer deal with their overly complicated back end discovery problems failed to keep the load balancers updated with accurate information on backend servers available for serving.

Re: A terrible, horrible, no-good, very bad day at Slack

#267

Post by Wheaties466 » Wed, Jul 08, 2020, 1:09 PM UTC

bstar77 wrote: post #251

Nextgrid wrote: post #223

Earlier quoted context omitted.

Which is what always happens when the company aims to build an "engineering playground" full of microservices and other moving parts without a clear technical justification and balancing the pros & cons and why I personally don't like working on such projects - it makes me feel uneasy not having a good understanding of the entire system. To be fair to Slack, at their scale, lots of moving parts might make sense, but…

I think you are drawing the wrong conclusions here... Microservices is not the boogyman here. It more likely has to do with speed of development, developer turnover and a plethora of other things that result in insufficient knowledge transfer. Microservices (like just about anything) can be implemented well or poorly. There's a reason we have sophisticated orchestration solutions like Kubernetes... it exists to tame…

I like your statement but I don't think it has to do with the speed of development either.

In a monolithic architecture the devs that deal with it, have to deal with the program as a whole. So if something doesn't work, its their problem. Where as in a micro service architecture it can be easy to spin up a service and not know the systems that integrate with it.

The problem here is with documentation and understanding of architecture. Its just the nature of the beast that the monolithic dev knows how thing communicate with the monolithic program because he needs to know, in order to do his job. In this instance the problem isn't with micro services, its with the execution. And that execution is a very easy trap to fall into with micro services.

Re: A terrible, horrible, no-good, very bad day at Slack

#268

Post by a2tech » Wed, Jul 08, 2020, 1:22 PM UTC

ketzo wrote: post #11

nik_0_0 wrote: post #7

Super interesting post. Following blog links, the timeline in https://slack.engineering/all-hands-on-deck-91d6986c3ee also offers a look at the play by play. However, as far as I can read it, they have somewhat different views on the root cause? "Soon, it became clear we had stale HAProxy configuration files, as a result of linting errors preventing re-rendering of the configuration." vs. "The program which synced th…

Honestly it’s a bit tough for me to parse, but the way I’m reading it, 1. Stale configs led to an overabundance of web apps, and then 2. Old instances of the web app couldn’t be removed because of the consul-template bug. so, yes, a combination (in sequence) of the two. Hard for me to be sure because I’m by no means knowledgeable on this stuff.

Real root cause was a poorly written home built tool to manage haproxy configs. The tool did not handle the slots being full and crashed. Respawn, rinse and repeat. Haproxy config got stale and when their automated tools removed servers it started with the machines haproxy actually knew about, and then the service died.

Re: A terrible, horrible, no-good, very bad day at Slack

#269

Post by thebean11 » Wed, Jul 08, 2020, 2:23 PM UTC

Wheaties466 wrote: post #267

bstar77 wrote: post #251

Earlier quoted context omitted.

I think you are drawing the wrong conclusions here... Microservices is not the boogyman here. It more likely has to do with speed of development, developer turnover and a plethora of other things that result in insufficient knowledge transfer. Microservices (like just about anything) can be implemented well or poorly. There's a reason we have sophisticated orchestration solutions like Kubernetes... it exists to tame…

I like your statement but I don't think it has to do with the speed of development either. In a monolithic architecture the devs that deal with it, have to deal with the program as a whole. So if something doesn't work, its their problem. Where as in a micro service architecture it can be easy to spin up a service and not know the systems that integrate with it. The problem here is with documentation and understandin…

The same thing can happen in a monolith. Make a change, run tests, and make sure your new feature works, while breaking some untested behavior you didn't know about in some other part of the monolith.

Re: A terrible, horrible, no-good, very bad day at Slack

#270

Post by mikecoles » Wed, Jul 08, 2020, 4:55 PM UTC

StavrosK wrote: post #87

k__ wrote: post #85

After I used Discord in different contexts for months now (and Slack for years), I can't understand why someone willingly chooses Slack. It's the Atlassian of chat tools. Horrible performance and bad usabillity.

I use Zulip for the day to day (it's amazing, I can't recommend it enough), but sometimes use Slack because some open source communities use it, and I'm always amazed at how damn slow it is. I can consistently out-type it, it's terrible. I guess it was great when it started out, but they're slowly boiling the frog, who is us.

Zulip won my last bake-off for chat systems. Integration was easy and the topic method of providing threads was amazing. The only feature it was missing was federation. In the XMPP world, you could communicate with users on other XMPP instances. With Zulip, you can only communicate with local users. Do you know if this is still the case?