Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s migration to a cellular architecture

slack.engineering

Post reply on HN
Print view View original

161–170 of 265 posts

Re: Slack’s migration to a cellular architecture

#161

Post by oceanplexian »

deanCommie wrote: post #52
ThePhysicist wrote: post #41

So they run everything in AWS USE1? That doesn't seem very redundant, but then I guess if the whole of USE1 goes down Slack won't be the only service that will be affected.

the "whole" of USE1 very rarely goes down [0], because unlike other cloud providers, Amazon's availability zones are actually independent and decoupled, and if you're running on EC2 in a zonal way it's highly unlikely an outage will affect multiple zones. [0] There are of course exceptions that come once every few years, but most instances people can think of in terms of widespread outages is one specific service goi…

AZs are buildings often times right next to each other on the same street. People who think this is a great failure domain for your entire business are deeply misguided. All it takes is a hurricane, a truck hitting a pole, a fire, or any number of extremely common situations and infra will be wiped off the map. Build stuff to be properly multi-region.

Re: Slack’s migration to a cellular architecture

#162

Post by slashdev »

athoscouto wrote: post #54
ComputerGuru wrote: post #47

Earlier quoted context omitted.

It sounds like you didn’t have persistent data, and were only offering compute? If there’s no need for a coherent master view accessible/writeable from all the clusters, there would be no reason to use multi-region cluster whatsoever.

We did. But the persisted data didn't live inside those ephemeral compute clusters though.

So your data store was still multi AZ? I’m a little confused how you’d serve the same user’s data consistently from multiple silos. Do you pin users to one AZ?

Re: Slack’s migration to a cellular architecture

#163

Post by lopkeny12ko »

eikenberry wrote: post #107
t0mas88 wrote: post #84

They got themselves into a mess here: > This turns out to have a lot of complexity lurking within. Slack does not share a common codebase or even runtime; services in the user-facing request path are written in Hack, Go, Java, and C++. This would necessitate a separate implementation in each language. This sounds crazy. I've seen several products where there is a core stack (e.g. Java) and then surrounding tools, ana…

What mess? That sounds like a healthy internal language ecosystem to me. You need at least 2 primary languages to avoid accidental lock-in and maintain good developer diversity. That very paragraph is a great example of how the diversity helped them avoid the trap of plumbing it through their RPCs.

You're suggesting that needing to reimplement the same thing 5 times for every single language in use is a hallmark of a "healthy internal language ecosystem"?

Re: Slack’s migration to a cellular architecture

#165

Post by gumballindie »

ignoramous wrote: post #98
gumballindie wrote: post #88

"cellular architecture" What? Does amazon need to push for new sales points or are they simply making up architectures now?

ex-AWS here May be marketing but it is an architecture born out of Amazon's (and AWS's) use of AWS: - Reliable scalability: How Amazon.com scales in the cloud , https://www.youtube.com/watch?v=QeW9wCB36ck&t=993 (2022) - How AWS minimizes the blast radius of failures , https://youtu.be/swQbA4zub20 (2018) For massive enterprise products like Slack that need close to 100% uptime across all their services, cells make sen…

Yeah that's what microservices were meant to achieve. Suppose the market is staturated with "microservices", so a new term was needed.

Re: Slack’s migration to a cellular architecture

#166

Post by ninkendo »

dr_kiszonka wrote: post #114

Nice write-up! If no new requests from users are arriving in a siloed AZ, internal services in that AZ will naturally quiesce as they have no new work to do. Not necessarily because, due to some bug, there may be resource-hungry jobs running indefinitely. (Slack's engineers must have considered this; I am just nitpicking this particular part of the text.)

If you replace "because" with "if", your comment makes more sense. "If" there are such bugs, you are right, but such bugs might not exist.

Re: Slack’s migration to a cellular architecture

#167

Post by endymi0n »

athoscouto wrote: post #35

Their siloing strategy, which I'll roughly refer as resolving a request from a single AZ, is a good way to keep operations and monitoring simple. A past team of mine managed services in a similar fashion. We had a couple (usually 2-4) single AZ clusters with a thin (Envoy) layer to balance traffic between clusters. We could detect incidents in a single cluster by comparing metrics across clusters. Mitigation was easy…

It took me some time to realize that Cloud Solution Architects are also just slightly more technical sales people in disguise whose only mission is upselling you onto more dependency. Same thing about their PR, every CxO these days says they need "multi-cloud", whatever that means and the costs are usually enormous, while complexity rises — with questionable benefit.

I did the math for our own stack and after a setback month in client revenue, and decided to put all our servers into a single AZ in a single region. The only multi-AZ, multi-region services are our backups. Surviving bad machines happens often enough that it's priced in via using Kubernetes, but losing a whole AZ is a freak accident that's just SO rare that calculating real business risk, it seemed apt to pretend it just doesn't happen (sorry, Google Cloud Paris customers).

Call me reckless, but I haven't looked back ever since and it saves us thousands of dollars in intra-AZ fees per month alone.

Re: Slack’s migration to a cellular architecture

#168

Post by fulladder »

memefrog wrote: post #157
fulladder wrote: post #142

Earlier quoted context omitted.

I haven't used Slack in a long time, but isn't this just the normal enshittification cycle that occurs with all Internet products? The founders got a nice exit several years back, I doubt they stuck around at Salesforce for long, so it's natural that the product would deteriorate over time. Slack IRC bridging in the 2014/2015 era was great. We had a lot of people who spent their whole workday in a terminal window and…

>isn't this just the normal enshittification cycle that occurs with all Internet products? No! Stop diluting this word.

> No! Stop diluting this word.

Yes, you're right, I'm misusing it.

However, I think that there is a phenomenon that happens to a lot of tech products that is more general than what Doctorow is talking about. There is a certain type of person who is attracted to building a new thing, and there is a different type of person who is attracted to a thing that is already successful. Pioneers and Settlers, as a former colleague of mine described it. In the context of Internet services, pioneers care a lot about attracting users initially so they tend to dwell on every minor detail. Settlers care a lot about stability, so gradual degradation over time (e.g., in performance, in other measures of quality) is tolerable as long as its rate is controllable and well-understood.

I think that Doctorow's thesis is a special case of this where greed is the driving factor behind the gradual erosion of quality.

Re: Slack’s migration to a cellular architecture

#169

Post by snoman »

quickthrower2 wrote: post #147
wasmitnetzen wrote: post #29

Earlier quoted context omitted.

My employer buys no Microsoft SaaS service, since we're mostly on Google services, so a stand-alone like Slack works quite well. And nobody uses Google Chat. And besides that, the UX of Teams is miles behind Slack.

Slack is not good UX in my opinion. It is often hard to see what generated a message notification - so yeah someone called me out, but who? where?. It shows me latest thread as being from last month when I know there have been more recent ones. It doesn't collapse those threads, so 100 reply incident threads dominate that view. Slack doesn't scale well (UX-wise) above say 30 people.

Opinions are valid, for sure. I can tell you that I’m a happy slack user at a company of just over a hundred thousand.

I haven’t regularly used teams in about a year, but I would legitimately consider passing on a job offer where they used it.

In a thread where many folks are talking about using the best tools for a job, teams is never the best tool for any form of digital communication.

Re: Slack’s migration to a cellular architecture

#170

Post by sitkack »

ec109685 wrote: post #149
kccqzy wrote: post #110

Earlier quoted context omitted.

That's another way of saying your typical utilization ratio is 66%. Which is on the low side honestly. That said, it's a trade off between efficiency and load spike tolerance. I trust that the trade off is made with informed decision.

66% isn’t low utilization. You’re always going to have micro spikes, and you never want to clip, so keeping some headroom around feels smart. Unless you co-mingle online and offline (batch) traffic on same hosts, flat response times and high utilization aren’t compatible.

High utilization means high variability and low resiliency and the last k-percentage of utilization causes highly non-linear effects.
Post reply on HN