Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s migration to a cellular architecture

slack.engineering

Post reply on HN
Print view View original

201–210 of 265 posts

Re: Slack’s migration to a cellular architecture

#201

Post by pclmulqdq »

theamk wrote: post #189
catchnear4321 wrote: post #179

Earlier quoted context omitted.

i mean you could also just spin up a reeeeeally big compute node and just do it all there. fewer things to monitor. fewer things that can fail. just log in from time to time to update packages. see, cloud doesn’t have to be complex.

until you get "hardware failure" note from cloud provider. Or the person updating packages makes a typo and messes up a system. sure, use "pet" computers for experiments and dev.. but having produluction be a "cattle" makes your life so much less stressful.

Having a few computers doesn't imply a "pets" approach to managing them.

Re: Slack’s migration to a cellular architecture

#202

Post by jameshart »

skullone wrote: post #73
jameshart wrote: post #33

Earlier quoted context omitted.

Half the complaints on here on architecture posts are 'you don't need this kind of stuff unless you're at FAANG scale'. Now we have a write up of something that's accessible to businesses at non-FAAANG scale, and we have the new complaint, that this kind of stuff isn't worthy of FAANG-scale architecture.

Geo traffic distribution, multi regions/AZs with functionality to weight and drain traffic should be used in most SaaS services where a simple failure somewhere could cost users time and lose company money/goodwill. It's not terribly hard nor expensive.

How do you think people are going to learn this stuff if not by reading about it from architects who have done it?

This writeup seems like a useful contribution to spreading this knowledge that you think every engineer should, somehow, innately be born with, to those members of the development community who missed out on picking this stuff up in elementary school.

Re: Slack’s migration to a cellular architecture

#203

Post by WinLychee »

pm wrote: post #197
WinLychee wrote: post #108

Earlier quoted context omitted.

PHP has some excellent ideas that other languages can't replicate, while at the same time having terrible ideas that other languages don't have to think about. Overall a huge fan of modern PHP, thanks for this writeup.

Which excellent ideas does it have that other languages can't replicate?

Perhaps more precisely: the defacto Apache-as-runtime + PHP model simplifies a ton of things. Namely your request state is created and destroyed all within the context of a single process, and you don't have to reason about shared state with other in-flight requests (unless you explicitly choose to go this route). It makes some bad programming patterns workable, because your state doesn't linger over a long-running period. Deploys are also super fast, you just have to swap the application code on disk and it'll get picked up on the next request (in-flight requests will keep processing with the old version IIRC). It's productive if not necessarily pretty. Also it has a type system now!

As a related thought, a lot of the modern serverless stuff feels like it's reinventing the ideas of Apache + PHP, or perhaps CGI?

Re: Slack’s migration to a cellular architecture

#204

Post by esprehn »

purpleturtle22 wrote: post #79

Can someone ELI5 the difference between using AWS availability zone affinity and then simply dropping the downed AZ at the top most routing point? Wouldn't that be the same thing, with the obvious caveat you are t using the routing technology Slack is using (We don't - We use vanilla AWS offerings)

Cells are not about guarding against AZ failure, but about partitioning the production infra to protect against bad deploys and configuration changes. Every AZ is split into many different cells.

Re: Slack’s migration to a cellular architecture

#205

Post by zx8080 »

skullone wrote: post #76
danielovichdk wrote: post #53

"A single Slack API request from a user (for example, loading messages in a channel) may fan out into hundreds of RPCs to service backends, each of which must complete to return a correct response to the user." Not being a dick here but is this not a fairly obvious flaw? I mean why not keep a structured "message log" of all channels of all time ? For every write the system updates the message log. I am guessing and m…

XMMP was extensible to support all this in the early 2000s. Slack reinvented simple services in the most obtuse way. I have to use Slack and I sideline quarterback all the ways things could have been better every day.

XMPP

Agree with this point of view. Except the Jabber/XMPP Cisco legal thing, there's just no tech answer on why on earth Slack did not use XMPP under the hood.

Re: Slack’s migration to a cellular architecture

#206

Post by esprehn »

deanCommie wrote: post #190
oceanplexian wrote: post #161

Earlier quoted context omitted.

AZs are buildings often times right next to each other on the same street. People who think this is a great failure domain for your entire business are deeply misguided. All it takes is a hurricane, a truck hitting a pole, a fire, or any number of extremely common situations and infra will be wiped off the map. Build stuff to be properly multi-region.

> AZs are buildings often times right next to each other on the same street. Not at AWS: https://aws.amazon.com/about-aws/global-infrastructure/regio... > An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. AZs give customers the ability to operate production applications and databases that are more highly available, fault tolerant, and s…

GCPs concept of Regions and Zones is different from AWS. For the same level of physical isolation as an AWS AZ you have to use different GCP Regions.

https://cloud.google.com/compute/docs/regions-zones

Re: Slack’s migration to a cellular architecture

#207

Post by alberth »

zx8080 wrote: post #205
skullone wrote: post #76

Earlier quoted context omitted.

XMMP was extensible to support all this in the early 2000s. Slack reinvented simple services in the most obtuse way. I have to use Slack and I sideline quarterback all the ways things could have been better every day.

XMPP Agree with this point of view. Except the Jabber/XMPP Cisco legal thing, there's just no tech answer on why on earth Slack did not use XMPP under the hood.

What’s even more interesting is … WhatsApp is XMPP/ejabberd based.

Slack would have known about WhatsApp architecture because it was widely talked about pre-FB acquisition (2014).

And Slack was founded in 2013.

Re: Slack’s migration to a cellular architecture

#208

Post by easton »

oceanplexian wrote: post #161
deanCommie wrote: post #52

Earlier quoted context omitted.

the "whole" of USE1 very rarely goes down [0], because unlike other cloud providers, Amazon's availability zones are actually independent and decoupled, and if you're running on EC2 in a zonal way it's highly unlikely an outage will affect multiple zones. [0] There are of course exceptions that come once every few years, but most instances people can think of in terms of widespread outages is one specific service goi…

AZs are buildings often times right next to each other on the same street. People who think this is a great failure domain for your entire business are deeply misguided. All it takes is a hurricane, a truck hitting a pole, a fire, or any number of extremely common situations and infra will be wiped off the map. Build stuff to be properly multi-region.

Is this true of AWS? I haven’t read that Wikileaks location document in a while, but I seem to recall in official docs the AZs being placed far enough away from each other to make sure a natural disaster won’t kill a whole region (different flood planes, etc). Of course, you go to Asburn and all the buildings are really close to each other.

Re: Slack’s migration to a cellular architecture

#209

Post by chrisweekly »

mrkeen wrote: post #68
chrisweekly wrote: post #37

I appreciate the clear explanation of the problem and the solution, which (as is so often the case) seems fairly simple or obvious in retrospect. Semi-related tangent: sometime around mid-2016, I came across a tool that helped visualize requests in near real-time, and showed what it "looks" like (ie, flow slows to trickle in service A during draining, while it ramps up in service B)... there was a really compelling d…

Vizceral

YES! Thank you!! :)

This kind of shared communal knowledge is one of many reasons I'm very grateful for the HN community.

Re: Slack’s migration to a cellular architecture

#210

Post by catchnear4321 »

ryukoposting wrote: post #193
catchnear4321 wrote: post #179

Earlier quoted context omitted.

i mean you could also just spin up a reeeeeally big compute node and just do it all there. fewer things to monitor. fewer things that can fail. just log in from time to time to update packages. see, cloud doesn’t have to be complex.

I'm not a cloud guy, but if you're going to put everything in one region, what's stopping you from shoving everything into a bunch of containers on a t2.2xlarge instance (or equivalent), and adding like one CloudWatch alarm (or equivalent) that reboots the instance if it stops responding?

will you settle for one big container?
Post reply on HN