Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s migration to a cellular architecture

slack.engineering

Post reply on HN
Print view View original

81–90 of 265 posts

Re: Slack’s migration to a cellular architecture

#81

Post by hotnfresh »

skullone wrote: post #14
aftbit wrote: post #9

Earlier quoted context omitted.

from the article: >Slack does not share a common codebase or even runtime; services in the user-facing request path are written in Hack, Go, Java, and C++.

Man what a mess. Meanwhile, everyone else can extend a library used by their common services in a common language trivially.

Meh. As long as you’ve got a good, typed interface for passing messages between them and for having a common understanding of (and versioning system for) key data structures, that’s fine for this sort of thing where it’s largely processing steams of small messages and events.

… but it’s probably JSON and some JSON-Schema-based “now you have two problems” junk instead of what I described. In which case, yeah, ew, gross. Unless they’ve made some unusually good choices.

Re: Slack’s migration to a cellular architecture

#82

Post by jasonwatkinspdx »

athoscouto wrote: post #35

Their siloing strategy, which I'll roughly refer as resolving a request from a single AZ, is a good way to keep operations and monitoring simple. A past team of mine managed services in a similar fashion. We had a couple (usually 2-4) single AZ clusters with a thin (Envoy) layer to balance traffic between clusters. We could detect incidents in a single cluster by comparing metrics across clusters. Mitigation was easy…

Yeah, I talked with a business that used a similar architecture for the same reasons. It can be really effective in multi-tenant apps where each customers data is fully independent and private. They also used multiple Amazon organizational accounts as a security partition. It made a few things more difficult but they felt the peace of mind was worth it.

Re: Slack’s migration to a cellular architecture

#83

Post by [deleted] »

skullone wrote: post #7
progbits wrote: post #4

Earlier quoted context omitted.

The other bit is separating the service into isolated cells so issues in one don't affect dependent services everywhere like they had experienced before. But yeah any good SRE could point this out years ago.

Just odd a company worth billions and billions of dollars is just now discovering HA models standard since the 90s. Can expand the Clos network architecture to these distributed service applications too. But judging by Slack's client quality, mature concepts such as those must be new to them.

[deleted]

Re: Slack’s migration to a cellular architecture

#84

Post by t0mas88 »

They got themselves into a mess here:

> This turns out to have a lot of complexity lurking within. Slack does not share a common codebase or even runtime; services in the user-facing request path are written in Hack, Go, Java, and C++. This would necessitate a separate implementation in each language.

This sounds crazy. I've seen several products where there is a core stack (e.g. Java) and then surrounding tools, analytics etc in Python, R and others. But why would you create such a mess for your primary user request path?

Sure, they're not "just a chat app" they have video, file sharing etc included and a lot of integrations. But still this sounds like a company that had too much money and too little sense while growing rapidly.

Re: Slack’s migration to a cellular architecture

#85

Post by jldugger »

bushbaba wrote: post #48
athoscouto wrote: post #35

Their siloing strategy, which I'll roughly refer as resolving a request from a single AZ, is a good way to keep operations and monitoring simple. A past team of mine managed services in a similar fashion. We had a couple (usually 2-4) single AZ clusters with a thin (Envoy) layer to balance traffic between clusters. We could detect incidents in a single cluster by comparing metrics across clusters. Mitigation was easy…

The downside of single AZ clusters is capacity. If you have a need to drastically scale up the compute might not be available in a single AZ.

Indeed, this is the main problem I run into. We have to scale up capacity before the traffic can be redirected or you basically double the scope of the outage briefly. Which involves multiple layers of capacity bringup -- ASG brings up new nodes, then HPA brings up the new pods.

Re: Slack’s migration to a cellular architecture

#86

Post by t0mas88 »

purpleturtle22 wrote: post #79

Can someone ELI5 the difference between using AWS availability zone affinity and then simply dropping the downed AZ at the top most routing point? Wouldn't that be the same thing, with the obvious caveat you are t using the routing technology Slack is using (We don't - We use vanilla AWS offerings)

They decided to use every routing tool available at least once in their setup, so they can't do this. But there is no explanation in the blog about why they use so many platforms and so many routing tools. Sounds to me like they got themselves into a mess and decided to continue on that path.

Re: Slack’s migration to a cellular architecture

#87

Post by temp_praneshp »

asah wrote: post #78
deanCommie wrote: post #52

Earlier quoted context omitted.

the "whole" of USE1 very rarely goes down [0], because unlike other cloud providers, Amazon's availability zones are actually independent and decoupled, and if you're running on EC2 in a zonal way it's highly unlikely an outage will affect multiple zones. [0] There are of course exceptions that come once every few years, but most instances people can think of in terms of widespread outages is one specific service goi…

Am I missing something about us-east-1 reliability ? https://www.google.com/search?q=us-east-1+reliability https://www.google.com/search?q=us-east-1+outage

Yes. to put it a bit bluntly, you are using a very generic google search and being blind to nuance.

us-east-1 does have more problems than other zones due to a variety of reasons, but it rarely (ie, once a few years) goes down as a whole. As long as you're in several AZs within us-east-1, the impact of most outages should not take you down completely. In the context of the comment you are replying to, your google search links are lazy and fail to see the big picture.

Re: Slack’s migration to a cellular architecture

#89

Post by skullone »

t0mas88 wrote: post #84

They got themselves into a mess here: > This turns out to have a lot of complexity lurking within. Slack does not share a common codebase or even runtime; services in the user-facing request path are written in Hack, Go, Java, and C++. This would necessitate a separate implementation in each language. This sounds crazy. I've seen several products where there is a core stack (e.g. Java) and then surrounding tools, ana…

It even pains me to see they're suffering from so many own goals. And it's unfortunately reflected in the poor experience using the Slack client. Not to mention the multiple deprecated bot/integration APIs with such bad feature parity between all the different ways to integrate your own tooling into Slack.
Post reply on HN