Live data from Hacker News

Slack’s migration to a cellular architecture

slack.engineering

211–220 of 265 posts

Re: Slack’s migration to a cellular architecture

#211
post #189

Earlier quoted context omitted.

until you get "hardware failure" note from cloud provider. Or the person updating packages makes a typo and messes up a system. sure, use "pet" computers for experiments and dev.. but having produluction be a "cattle" makes your life so much less stressful.

Having a few computers doesn't imply a "pets" approach to managing them.

more like a zoo, right?

Re: Slack’s migration to a cellular architecture

#212
post #189

Earlier quoted context omitted.

i mean you could also just spin up a reeeeeally big compute node and just do it all there. fewer things to monitor. fewer things that can fail. just log in from time to time to update packages. see, cloud doesn’t have to be complex.

until you get "hardware failure" note from cloud provider. Or the person updating packages makes a typo and messes up a system. sure, use "pet" computers for experiments and dev.. but having produluction be a "cattle" makes your life so much less stressful.

I think everyone internalizes pets vs cattle a little differently, but I think people often don't realize that if you have cattle, then you now have a cattle ranch. I.e. you often create a new concept to manage your cattle, and you often run that like it's a pet.

E.g. you want to think about your machines/containers like cattle, so you put them into a kubernetes cluster, which has become your new pet. If all your infra fits on one machine, it's way easier to have that as your pet the same way it's easier to have a dog than run a livestock operation.

Re: Slack’s migration to a cellular architecture

#214
Delighted to be part of this conversation on cell-based architecture. As the author of the cell-based reference architecture https://github.com/wso2/reference-architecture/blob/master/r..., I'm here to share insights on this exciting approach.

Cell-based architecture introduces modular 'cells' into software systems, each with distinct APIs. This design fosters loose coupling and scalability – key for today's dynamic software landscape. Particularly, for those intrigued by microservices, cells align seamlessly with the independent, scalable components that power microservices architectures.

Curious to dive deeper? If you're keen to explore the nitty-gritty technicalities, I invite you to check out the architecture paper https://github.com/wso2/reference-architecture/blob/master/r... for an in-depth understanding. Let's kick-start this dialogue on the potential of cell-based architecture and its impact on modern software design. Feel free to join the conversation!

Re: Slack’s migration to a cellular architecture

#215

Their siloing strategy, which I'll roughly refer as resolving a request from a single AZ, is a good way to keep operations and monitoring simple. A past team of mine managed services in a similar fashion. We had a couple (usually 2-4) single AZ clusters with a thin (Envoy) layer to balance traffic between clusters. We could detect incidents in a single cluster by comparing metrics across clusters. Mitigation was easy…

I assume you were using AWS? I know some of the AZ of other cloud providers (Azure? Oracle? Google?) are not fully siloed. They might have independent power and networking, but be in the same physical location. I'm mentioning this for other people to be aware as one can easily make the assumption that an AZ is the same concept on all clouds, which is not true and painful to realise.

Azure's zones are "physically separate", but it's unclear whether zones could be in the same building. Especially since they don't guarantee distance between zones - they just aim for 300mi (483km)

Re: Slack’s migration to a cellular architecture

#216

Earlier quoted context omitted.

AZs are buildings often times right next to each other on the same street. People who think this is a great failure domain for your entire business are deeply misguided. All it takes is a hurricane, a truck hitting a pole, a fire, or any number of extremely common situations and infra will be wiped off the map. Build stuff to be properly multi-region.

> AZs are buildings often times right next to each other on the same street. Not at AWS: https://aws.amazon.com/about-aws/global-infrastructure/regio... > An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. AZs give customers the ability to operate production applications and databases that are more highly available, fault tolerant, and s…

> This is unique compared to Microsoft

Azure doesn't guarantee distance between zones, but they aim for 483km. So they can provide better isolation at the cost of higher inter-AZ latency. Depends on the region - you'd need an internal contact (and/or NDA?) to get approx numbers

Re: Slack’s migration to a cellular architecture

#217

Earlier quoted context omitted.

I noticed that the hack blog ( https://hhvm.com/blog/ ) basically stopped posting updates since the end of 2022. As downstream users of hacklang development have you folks noticed a change in development pace or ambition within the hack development team?

I too am super curious about this. Plus, it seems telling that Threads was developed in Python - not Hack. (I’m aware IG is Python & it’s the same team)

If anything from what I’ve heard hack is slowly taking over IG and WhatsApp. But it’s an incredibly large codebase to move

Re: Slack’s migration to a cellular architecture

#218

Earlier quoted context omitted.

i mean you could also just spin up a reeeeeally big compute node and just do it all there. fewer things to monitor. fewer things that can fail. just log in from time to time to update packages. see, cloud doesn’t have to be complex.

I'm not a cloud guy, but if you're going to put everything in one region, what's stopping you from shoving everything into a bunch of containers on a t2.2xlarge instance (or equivalent), and adding like one CloudWatch alarm (or equivalent) that reboots the instance if it stops responding?

At least three obvious reasons might be:

(1) Raw scale might mean you just plain can't fit everything on a single t2.2xlarge.

(2) Different services (containers) might have different performance profiles, so you may want a few different types of machines around.

(3) You probably still want N+2 redundancy even within your single AZ, so this scheme should at least be upgraded to three t2.2xlarge boxes. ;)

Re: Slack’s migration to a cellular architecture

#219

Earlier quoted context omitted.

What mess? That sounds like a healthy internal language ecosystem to me. You need at least 2 primary languages to avoid accidental lock-in and maintain good developer diversity. That very paragraph is a great example of how the diversity helped them avoid the trap of plumbing it through their RPCs.

You're suggesting that needing to reimplement the same thing 5 times for every single language in use is a hallmark of a "healthy internal language ecosystem"?

That's the red flag, the thing you are trying to avoid. You don't want to implement things in each language and you always have more than one language even if you standardize (over time). You don't want libraries, you want services. This is why things like Istio are way better than libraries for mesh networking. Using external services for common things keeps you from being locked into a single tech stack and the limitations that entails.

Re: Slack’s migration to a cellular architecture

#220

Earlier quoted context omitted.

Don’t they also use Chime? It wouldn’t be a single point of failure.

To contribute to the tangled ball of messaging, slack also uses chime sdk to handle huddles

Wonder why huddles sound better compared to chime?
Post reply on HN