Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s migration to a cellular architecture

slack.engineering

Post reply on HN
Print view View original

251–260 of 265 posts

Re: Slack’s migration to a cellular architecture

#251

Post by deanCommie »

esprehn wrote: post #206
deanCommie wrote: post #190

Earlier quoted context omitted.

> AZs are buildings often times right next to each other on the same street. Not at AWS: https://aws.amazon.com/about-aws/global-infrastructure/regio... > An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. AZs give customers the ability to operate production applications and databases that are more highly available, fault tolerant, and s…

GCPs concept of Regions and Zones is different from AWS. For the same level of physical isolation as an AWS AZ you have to use different GCP Regions. https://cloud.google.com/compute/docs/regions-zones

That link says "Google designs zones to minimize the risk of correlated failures caused by physical infrastructure outages like power, cooling, or networking. Thus, if a zone becomes unavailable, you can transfer traffic to another zone in the same region to keep your services running."

Which is clearly false if a single flood can take out an entire region.

Re: Slack’s migration to a cellular architecture

#252

Post by robertlagrant »

zo1 wrote: post #238
grokys wrote: post #153

Earlier quoted context omitted.

Maybe because you value your employees being able to copy an image from your chat platform? (Teams still can't copy images, instead you get a massive base64 block of text iirc)

I don't like that either, but I get it, it breaks the privacy of the sent message as messages have permissions attached to them. Honestly, bugged me a few times before I just switched to using the snippet tool. I use it all the time anyways and this felt natural.

It's not base64 because of privacy, and using the snipping tool instead of just copying is not natural. Imagine opening a whole other program to copy some text you just highlighted.

Re: Slack’s migration to a cellular architecture

#253

Post by mox1 »

jasonwatkinspdx wrote: post #82
athoscouto wrote: post #35

Their siloing strategy, which I'll roughly refer as resolving a request from a single AZ, is a good way to keep operations and monitoring simple. A past team of mine managed services in a similar fashion. We had a couple (usually 2-4) single AZ clusters with a thin (Envoy) layer to balance traffic between clusters. We could detect incidents in a single cluster by comparing metrics across clusters. Mitigation was easy…

Yeah, I talked with a business that used a similar architecture for the same reasons. It can be really effective in multi-tenant apps where each customers data is fully independent and private. They also used multiple Amazon organizational accounts as a security partition. It made a few things more difficult but they felt the peace of mind was worth it.

My company has a pretty unique strategy where we have separate AWS accounts for each unit within the company. Each unit gets a prod and non-prod account.

We have ~150 accounts, so roughly 75 different department, with some having not much and others have a lot of resources.

Its complex, but provides a lot of nice security primitives. We have an overarching administrative account, but that doesnt get used (and lots of alarm bells go off when it is).

Re: Slack’s migration to a cellular architecture

#254

Post by zx8080 »

alberth wrote: post #207
zx8080 wrote: post #205

Earlier quoted context omitted.

XMPP Agree with this point of view. Except the Jabber/XMPP Cisco legal thing, there's just no tech answer on why on earth Slack did not use XMPP under the hood.

What’s even more interesting is … WhatsApp is XMPP/ejabberd based. Slack would have known about WhatsApp architecture because it was widely talked about pre-FB acquisition (2014). And Slack was founded in 2013.

Probably, it's the result of schizo-histerical tech decision process happening in some companies "we need fancy tech and NOT THAT XML".

Sometimes it's for OKRs and power/politics balance between departments and teams in an enterprise. "If an existing tech like XMPP is used then no serious development could be needed" fear (which is not really true). It can lead (and leads) to a huge waste of resources and overspending.

But it's a bit similar to building a luxury house. Not because an owner needs it. But because he can afford it.

Re: Slack’s migration to a cellular architecture

#255

Post by jamesfinlayson »

nijave wrote: post #235
jeremyjh wrote: post #198

Earlier quoted context omitted.

Actually I assumed AWS did it the same way as the others - I thought maybe they are in another building on a campus but I didn’t think that should be a factor in planning and that I should use regions for geographic redundancy anyway.

Afaik AWS AZs are physically separate. I think some maps exist. Around here, there's 3 AZs and they're multi building campuses about 10-20 miles apart situated in suburbs outside the city freeway belt.

Yeah I definitely thought they were physically separate - I thought that an AZ might span multiple data centres too (could be wrong) and I thought they had to be at least 10km apart.

Re: Slack’s migration to a cellular architecture

#256

Post by HumanOstrich »

asankama wrote: post #248
HumanOstrich wrote: post #239

Earlier quoted context omitted.

I'm familiar with the cell architecture promoted by the WSO2 papers (and other resources). I like it and I've used it in client projects. However, this article uses "cell" in a completely different way. It is not the cell-based architecture that you are promoting here without reading the article.

In the CBA paper, a 'Cell' is an architecture construct you can use in the design stage and take through the development and then to the deployment. So, it addresses both application and deployment architecture. Each cell has a boundary, a cell gateway, and components inside the cell.

Thanks for the follow-up and I apologize for my snark. It appears my knowledge was out-of-date and I stand corrected. I'll need to brush up on the current docs and research.

Re: Slack’s migration to a cellular architecture

#257

Post by philwelch »

danielovichdk wrote: post #53

"A single Slack API request from a user (for example, loading messages in a channel) may fan out into hundreds of RPCs to service backends, each of which must complete to return a correct response to the user." Not being a dick here but is this not a fairly obvious flaw? I mean why not keep a structured "message log" of all channels of all time ? For every write the system updates the message log. I am guessing and m…

> When companies create this microservices bog and then, when any problem comes up, they say, “distributed systems are hard” it reminds me of when my toddler throws food on the floor then says, “look, big mess”

https://x.com/telmudic/status/1684479894406025216

Re: Slack’s migration to a cellular architecture

#258

Post by qlkjwenf »

Thanks to EU, Microsoft Teams replaced Slack. GDPR makes it way too difficult to work with multiple software vendors, so companies usually only choose products from the absolute minimum number of vendors (even if there are better options). Also Slack asks too much money for what it does.

Re: Slack’s migration to a cellular architecture

#259

Post by gumballindie »

unethical_ban wrote: post #224
gumballindie wrote: post #88

"cellular architecture" What? Does amazon need to push for new sales points or are they simply making up architectures now?

Yeah, the idea was present at my bank employer 15 years ago. Drain a DC to do maintenance and load testing. It was called high availability. This blog is writing about availability zones as if they're a new concept too.

If amazon does something right then thats marketing.

Re: Slack’s migration to a cellular architecture

#260

Post by ec109685 »

sitkack wrote: post #170
ec109685 wrote: post #149

Earlier quoted context omitted.

66% isn’t low utilization. You’re always going to have micro spikes, and you never want to clip, so keeping some headroom around feels smart. Unless you co-mingle online and offline (batch) traffic on same hosts, flat response times and high utilization aren’t compatible.

High utilization means high variability and low resiliency and the last k-percentage of utilization causes highly non-linear effects.

Smart budgeted retries can help smooth those peaks.
Post reply on HN