Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s migration to a cellular architecture

slack.engineering

Post reply on HN
Print view View original

111–120 of 265 posts

Re: Slack’s migration to a cellular architecture

#111

Post by alberth »

muglug wrote: post #105
alberth wrote: post #61

Earlier quoted context omitted.

Hi Matt Thanks for Psalm! Curious, if Slack was built today from ground up - what tech stack do you think should/would be used?

That’s a simple question that’s hard to answer. A slightly different question that’s a bit easier to answer: “if I could wave a magic wand and X million lines of code were instantly rewritten and all developers were instantly trained on that language”. There the choice would be limited to languages that have similar or faster perf characteristics to Hack, without sacrificing developer productivity. Rust is out of the…

I like your question way better than mine :)

Some follow-up …

A. isn’t PHP on par perf wise to Hack these days? Re: “PHP is too slow” comment.

B. have you ever looked into PHP-NGX? It’s perf looks impressive, though you lose the benefit of stateless

https://github.com/rryqszq4/ngx-php

https://www.techempower.com/benchmarks/#section=data-r21

Re: Slack’s migration to a cellular architecture

#112

Post by mike_hock »

ignoramous wrote: post #98
gumballindie wrote: post #88

"cellular architecture" What? Does amazon need to push for new sales points or are they simply making up architectures now?

ex-AWS here May be marketing but it is an architecture born out of Amazon's (and AWS's) use of AWS: - Reliable scalability: How Amazon.com scales in the cloud , https://www.youtube.com/watch?v=QeW9wCB36ck&t=993 (2022) - How AWS minimizes the blast radius of failures , https://youtu.be/swQbA4zub20 (2018) For massive enterprise products like Slack that need close to 100% uptime across all their services, cells make sen…

Cells, interlinked.

Re: Slack’s migration to a cellular architecture

#113

Post by diarrhea »

gumballindie wrote: post #88

"cellular architecture" What? Does amazon need to push for new sales points or are they simply making up architectures now?

A big term for a simple design principle indeed.

But their implementation isn’t as grim as what I had initially envisioned when hearing that term. I immediately thought of Smalltalk and the idea of objects sitting next to each other, forming a graph (of no particular structure… just a graph), passing messages to neighbours. Like cells in an organism send hormones and whatnot. That makes for a huge mess that cannot be reasoned about, hence why we instead went with stricter structures like trees for (single) inheritance. That’s much closer to this silo approach, which seems nice and reasonable (although I get the impression considerable complexity was swept under the rug, like global DB consistence; the siloes cannot truly be siloed).

Re: Slack’s migration to a cellular architecture

#114

Post by dr_kiszonka »

Nice write-up!

    If no new requests from users are arriving in a siloed AZ, internal services in that AZ will naturally quiesce as they have no new work to do.
Not necessarily because, due to some bug, there may be resource-hungry jobs running indefinitely. (Slack's engineers must have considered this; I am just nitpicking this particular part of the text.)

Re: Slack’s migration to a cellular architecture

#115

Post by t0mas88 »

eikenberry wrote: post #107
t0mas88 wrote: post #84

They got themselves into a mess here: > This turns out to have a lot of complexity lurking within. Slack does not share a common codebase or even runtime; services in the user-facing request path are written in Hack, Go, Java, and C++. This would necessitate a separate implementation in each language. This sounds crazy. I've seen several products where there is a core stack (e.g. Java) and then surrounding tools, ana…

What mess? That sounds like a healthy internal language ecosystem to me. You need at least 2 primary languages to avoid accidental lock-in and maintain good developer diversity. That very paragraph is a great example of how the diversity helped them avoid the trap of plumbing it through their RPCs.

Since when is an "internal langue ecosystem" a good idea? Technology in a company like Slack exists to deliver useful features and good performance/stability to users faster than competitors can do it. For an app like theirs it doesn't sound like something that needs several disparate internal platforms that are slowing them down.

Re: Slack’s migration to a cellular architecture

#116

Post by nostrebored »

mynameisvlad wrote: post #70
messe wrote: post #49

Earlier quoted context omitted.

AWS also uses Slack internally, so add that to the list of shit that can hit the fan if us-east-1/IAD goes down.

Don’t they also use Chime? It wouldn’t be a single point of failure.

To contribute to the tangled ball of messaging, slack also uses chime sdk to handle huddles

Re: Slack’s migration to a cellular architecture

#117

Post by nostrebored »

hotnfresh wrote: post #81
skullone wrote: post #14

Earlier quoted context omitted.

Man what a mess. Meanwhile, everyone else can extend a library used by their common services in a common language trivially.

Meh. As long as you’ve got a good, typed interface for passing messages between them and for having a common understanding of (and versioning system for) key data structures, that’s fine for this sort of thing where it’s largely processing steams of small messages and events. … but it’s probably JSON and some JSON-Schema-based “now you have two problems” junk instead of what I described. In which case, yeah, ew, gros…

There are tons of approaches to align on service contracts for JSON based API calls. There’s also libraries like gRPC which help make contacts explicit. Neither are really uncommon

Re: Slack’s migration to a cellular architecture

#118

Post by rewmie »

kccqzy wrote: post #110
athoscouto wrote: post #63

Earlier quoted context omitted.

Even though each cluster was single AZ the whole system wasn't, so we weren't bound by the capacity of a single AZ. Most of the situations where we needed to drastically scale up were known ahead of time as well (e.g. campaign from customer), and we would preallocate instances or even more clusters. I may be forcing my memory, but if I'm not mistaken, our auto scaling was setup in a way that the system could handle s…

That's another way of saying your typical utilization ratio is 66%. Which is on the low side honestly. That said, it's a trade off between efficiency and load spike tolerance. I trust that the trade off is made with informed decision.

> That said, it's a trade off between efficiency and load spike tolerance. I trust that the trade off is made with informed decision.

I don't think that relatively low utilization rates is the scenario that requires "informed decision". The only tradeoff in low utilization rate scenarios is cost, which might be outright cheaper and irrelevant once you do the math on the tradeoffs of using reserved instances vs the cost of scaling up with on-demand instances.

You need to make a damn good case to chronically underprovision your system and expect it to autoscale your way into nickle-and-dime savings.

Re: Slack’s migration to a cellular architecture

#119

Post by nostrebored »

skullone wrote: post #89
t0mas88 wrote: post #84

They got themselves into a mess here: > This turns out to have a lot of complexity lurking within. Slack does not share a common codebase or even runtime; services in the user-facing request path are written in Hack, Go, Java, and C++. This would necessitate a separate implementation in each language. This sounds crazy. I've seen several products where there is a core stack (e.g. Java) and then surrounding tools, ana…

It even pains me to see they're suffering from so many own goals. And it's unfortunately reflected in the poor experience using the Slack client. Not to mention the multiple deprecated bot/integration APIs with such bad feature parity between all the different ways to integrate your own tooling into Slack.

What do you mean? Slack is one of the most responsive and reliable tools I touch every day.

Re: Slack’s migration to a cellular architecture

#120

Post by nostrebored »

t0mas88 wrote: post #115
eikenberry wrote: post #107

Earlier quoted context omitted.

What mess? That sounds like a healthy internal language ecosystem to me. You need at least 2 primary languages to avoid accidental lock-in and maintain good developer diversity. That very paragraph is a great example of how the diversity helped them avoid the trap of plumbing it through their RPCs.

Since when is an "internal langue ecosystem" a good idea? Technology in a company like Slack exists to deliver useful features and good performance/stability to users faster than competitors can do it. For an app like theirs it doesn't sound like something that needs several disparate internal platforms that are slowing them down.

How is choosing the right language for a task/team slowing them down?

For large scale, cross cutting initiatives you’ll have some pain. For feature velocity, you’ll see great results. Everything is a trade off.

Post reply on HN