Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s migration to a cellular architecture

slack.engineering

Post reply on HN
Print view View original

221–230 of 265 posts

Re: Slack’s migration to a cellular architecture

#221

Post by inopinatus »

oceanplexian wrote: post #161
deanCommie wrote: post #52

Earlier quoted context omitted.

the "whole" of USE1 very rarely goes down [0], because unlike other cloud providers, Amazon's availability zones are actually independent and decoupled, and if you're running on EC2 in a zonal way it's highly unlikely an outage will affect multiple zones. [0] There are of course exceptions that come once every few years, but most instances people can think of in terms of widespread outages is one specific service goi…

AZs are buildings often times right next to each other on the same street. People who think this is a great failure domain for your entire business are deeply misguided. All it takes is a hurricane, a truck hitting a pole, a fire, or any number of extremely common situations and infra will be wiped off the map. Build stuff to be properly multi-region.

When AWS is in immediately adjacent buildings, it’s for the same AZ.

Re: Slack’s migration to a cellular architecture

#222

Post by hliyan »

esprehn wrote: post #204
purpleturtle22 wrote: post #79

Can someone ELI5 the difference between using AWS availability zone affinity and then simply dropping the downed AZ at the top most routing point? Wouldn't that be the same thing, with the obvious caveat you are t using the routing technology Slack is using (We don't - We use vanilla AWS offerings)

Cells are not about guarding against AZ failure, but about partitioning the production infra to protect against bad deploys and configuration changes. Every AZ is split into many different cells.

So, guarding against human errors / process failures, and not hardware failures?

Re: Slack’s migration to a cellular architecture

#223

Post by hliyan »

Terretta wrote: post #106
madduci wrote: post #16

Cellular architecture? They've just rediscovered the art of redundancy systems

Indeed, for 20+ years of distributed data centers (remember AZs are generally separate DCs near a city but on different grids, regions are geographically disparate cities) we called it "shared nothing" architecture pattern. Here's AWS's 2019 guide for financial services in AWS, where the isolated stack concept is referenced under parallel resiliency section and called "shared nothing": https://d1.awsstatic.com/Financ…

I wish modern architecture writing could go back to being this straightforward:

"There are three dominent themes in building high transaction rate multiprocessor systems, namely shared memory (e.g. Synapse, IBM/AP configurations), shared disk (e.g. VAX/cluster, any multi-ported disk system), and shared nothing (e.g. Tandem, Tolerant). This paper argues that shared nothing is the pre- ferred approach."

https://dsf.berkeley.edu/papers/hpts85-nothing.pdf

Re: Slack’s migration to a cellular architecture

#224

Post by unethical_ban »

gumballindie wrote: post #88

"cellular architecture" What? Does amazon need to push for new sales points or are they simply making up architectures now?

Yeah, the idea was present at my bank employer 15 years ago. Drain a DC to do maintenance and load testing. It was called high availability.

This blog is writing about availability zones as if they're a new concept too.

Re: Slack’s migration to a cellular architecture

#226

Post by namdnay »

danielovichdk wrote: post #53

"A single Slack API request from a user (for example, loading messages in a channel) may fan out into hundreds of RPCs to service backends, each of which must complete to return a correct response to the user." Not being a dick here but is this not a fairly obvious flaw? I mean why not keep a structured "message log" of all channels of all time ? For every write the system updates the message log. I am guessing and m…

I imagine the base messages are in a single store. But then you have reactions, attachments, gifs, user profiles, and probably hundreds of custom integrations/plugins.

Having worked on other messaging apps, these are usually separated because their performance/scalability requirements are different

Re: Slack’s migration to a cellular architecture

#227

Post by lordgrenville »

memefrog wrote: post #158

"For example slack is an incredibly successful product. But it seems like every week I encounter a new bug that makes it completely unusable for me, from taking seconds per character when typing to being completely unable to render messages. (Discord on the other hand has always been reliable and snappy despite, judging by my highly scientific googling, having 1/3rd as many employees. So it's not like chat apps are j…

I really wouldn't judge the quality of some company's technical advice based on one person's experience with their UI. For almost any consumer software that gets mentioned here, you will find some people who love it and lots of others with gripes. And for e.g. Slack might have bad product/UI people but very good infra people. Better to look at TFA and judge it on its merits.

Re: Slack’s migration to a cellular architecture

#228

Post by DandyDev »

asankama wrote: post #214

Delighted to be part of this conversation on cell-based architecture. As the author of the cell-based reference architecture https://github.com/wso2/reference-architecture/blob/master/r... , I'm here to share insights on this exciting approach. Cell-based architecture introduces modular 'cells' into software systems, each with distinct APIs. This design fosters loose coupling and scalability – key for today's dynamic…

This seems written by AI? And as such it comes across as not genuine

Re: Slack’s migration to a cellular architecture

#229

Post by asah »

temp_praneshp wrote: post #87
asah wrote: post #78

Earlier quoted context omitted.

Am I missing something about us-east-1 reliability ? https://www.google.com/search?q=us-east-1+reliability https://www.google.com/search?q=us-east-1+outage

Yes. to put it a bit bluntly, you are using a very generic google search and being blind to nuance. us-east-1 does have more problems than other zones due to a variety of reasons, but it rarely (ie, once a few years) goes down as a whole. As long as you're in several AZs within us-east-1, the impact of most outages should not take you down completely. In the context of the comment you are replying to, your google sea…

thanks - jfyi this was from my personal experience spanning a decade with USE1, but again maybe my experience is out of date, so thanks for the update.

(p.s. the use of a google search vs direct results wasn't "lazy" - it's to allow readers to do their own research vs pasting one result and then getting accused of bias)

Re: Slack’s migration to a cellular architecture

#230

Post by tacker2000 »

muglug wrote: post #128
alberth wrote: post #111

Earlier quoted context omitted.

I like your question way better than mine :) Some follow-up … A. isn’t PHP on par perf wise to Hack these days? Re: “PHP is too slow” comment. B. have you ever looked into PHP-NGX? It’s perf looks impressive, though you lose the benefit of stateless https://github.com/rryqszq4/ngx-php https://www.techempower.com/benchmarks/#section=data-r21

> isn’t PHP on par perf wise to Hack these days? No. But I don't have any numbers, because it's been years since the two languages were directly comparable on anything but a teeny tiny example program. Facebook gets big cost savings from a 1% improvement in performance, so they make sure that performance is as good as it can possibly be. They have a team of engineers working on the problem. PHP doesn't have any engin…

Yes, Hack is for Google or FB scale stuff. But to be honest, Slack is probably up there also, so it makes sense
Post reply on HN