Live data from Hacker News

Slack’s migration to a cellular architecture

slack.engineering

121–130 of 265 posts

Re: Slack’s migration to a cellular architecture

#121
post #16

Cellular architecture? They've just rediscovered the art of redundancy systems

To me it seems without the art. The costs will be passed on to the customers. I think there must be good ways to do redundancy without having all services running at full blast in each Availability Zone.

It's a blunt tool, much like PHP. PHP does seem to be a good choice for them, but I wouldn't want to work there. It's all right, there are different ways to do stuff.

Re: Slack’s migration to a cellular architecture

#122
post #86

Earlier quoted context omitted.

They decided to use every routing tool available at least once in their setup, so they can't do this. But there is no explanation in the blog about why they use so many platforms and so many routing tools. Sounds to me like they got themselves into a mess and decided to continue on that path.

Somewhere, an engineering “leader” is going to point to this blog post and then say, “Well, that’s how Slack did it!” and promptly copy this overwrought system

I’m not sure if you’re being serious, but in any case; This will happen, as it always does, inevitably.

Re: Slack’s migration to a cellular architecture

#123

Earlier quoted context omitted.

Half the complaints on here on architecture posts are 'you don't need this kind of stuff unless you're at FAANG scale'. Now we have a write up of something that's accessible to businesses at non-FAAANG scale, and we have the new complaint, that this kind of stuff isn't worthy of FAANG-scale architecture.

Geo traffic distribution, multi regions/AZs with functionality to weight and drain traffic should be used in most SaaS services where a simple failure somewhere could cost users time and lose company money/goodwill. It's not terribly hard nor expensive.

Those are all much looser restrictions than routing traffic consistently to a cell

Re: Slack’s migration to a cellular architecture

#124

Earlier quoted context omitted.

It sounds like you didn’t have persistent data, and were only offering compute? If there’s no need for a coherent master view accessible/writeable from all the clusters, there would be no reason to use multi-region cluster whatsoever.

We did. But the persisted data didn't live inside those ephemeral compute clusters though.

Yeah, keep stateful stuff and stateless stuff separate; separate clusters, network spaces, cloud accounts, likely a mix of all that.

Clearly define boundaries and acceptable behavior within boundaries.

Setup up telemetry and observability to monitor for threshold violations.

Simple. Right?

Re: Slack’s migration to a cellular architecture

#125

Earlier quoted context omitted.

Depending on the size of the company, that value is absolutely insignificant.

Bigcos with robust sales truly can’t afford the organizational-attentional cost of walking across the street to pick up a $10,000 coin.

/s ?

Re: Slack’s migration to a cellular architecture

#126
post #7
post #4

Earlier quoted context omitted.

The other bit is separating the service into isolated cells so issues in one don't affect dependent services everywhere like they had experienced before. But yeah any good SRE could point this out years ago.

Just odd a company worth billions and billions of dollars is just now discovering HA models standard since the 90s. Can expand the Clos network architecture to these distributed service applications too. But judging by Slack's client quality, mature concepts such as those must be new to them.

The linked AWS article specifically explains that it’s not just the typical single load balancer for cross AZ routing. I frankly don’t know where you’re getting that this means that HA is new to them.

Re: Slack’s migration to a cellular architecture

#127
The thing I don't understand about Slack is how the core functionality seems to have continuously degraded since I started using it in ~2015. When I started using it, its core message sending features basically didn't have the issues with delayed messages or failure to send that I had experienced with competitors. Now, I routinely have to reset the app/clear the cache and go through various dances to get files to upload reliably (add the file to a message, wait five or ten seconds, then hit send). It's nice to see these technical write-ups about improving the infrastructure behind Slack, but I'd like to see fewer feature launches and more stability improvements to make the web, desktop and mobile apps feel like reliable software again. (nice to haves would be re-launching the XMPP and IRC bridges)

Re: Slack’s migration to a cellular architecture

#128
post #105

Earlier quoted context omitted.

That’s a simple question that’s hard to answer. A slightly different question that’s a bit easier to answer: “if I could wave a magic wand and X million lines of code were instantly rewritten and all developers were instantly trained on that language”. There the choice would be limited to languages that have similar or faster perf characteristics to Hack, without sacrificing developer productivity. Rust is out of the…

I like your question way better than mine :) Some follow-up … A. isn’t PHP on par perf wise to Hack these days? Re: “PHP is too slow” comment. B. have you ever looked into PHP-NGX? It’s perf looks impressive, though you lose the benefit of stateless https://github.com/rryqszq4/ngx-php https://www.techempower.com/benchmarks/#section=data-r21

> isn’t PHP on par perf wise to Hack these days?

No. But I don't have any numbers, because it's been years since the two languages were directly comparable on anything but a teeny tiny example program.

Facebook gets big cost savings from a 1% improvement in performance, so they make sure that performance is as good as it can possibly be. They have a team of engineers working on the problem.

PHP doesn't have any engineers working on performance full-time — it's impossible for the language to compete there. Hack has also removed a bunch of PHP constructs (e.g. magic methods) that are a drain on performance, so there's no way to close the gap.

But that should in no way make you choose Hack over PHP. Apart from anything else, the delta won't matter for 99.9% of websites.

Re: Slack’s migration to a cellular architecture

#129
post #99

Earlier quoted context omitted.

The only way you get Hack on that list of languages is that they had a policy of letting lead engineers starting a project to choose the language at will, and they hired enough lead engineers who previously worked at FB/Meta.

I think that Hack might’ve been on that list earlier than you think. Slack started as a PHP application.

Yeah. If they already had a large php codebase, moving to Hack makes complete sense.

Re: Slack’s migration to a cellular architecture

#130

"cellular architecture" What? Does amazon need to push for new sales points or are they simply making up architectures now?

Cell architecture goes way back, at least 10 years. Tumblr for example.

http://highscalability.com/blog/2012/5/9/cell-architectures....

Post reply on HN