Earlier quoted context omitted.
until you get "hardware failure" note from cloud provider. Or the person updating packages makes a typo and messes up a system. sure, use "pet" computers for experiments and dev.. but having produluction be a "cattle" makes your life so much less stressful.
Having a few computers doesn't imply a "pets" approach to managing them.
Slack’s migration to a cellular architecture
211–220 of 265 posts
Re: Slack’s migration to a cellular architecture
#212Earlier quoted context omitted.
i mean you could also just spin up a reeeeeally big compute node and just do it all there. fewer things to monitor. fewer things that can fail. just log in from time to time to update packages. see, cloud doesn’t have to be complex.
until you get "hardware failure" note from cloud provider. Or the person updating packages makes a typo and messes up a system. sure, use "pet" computers for experiments and dev.. but having produluction be a "cattle" makes your life so much less stressful.
E.g. you want to think about your machines/containers like cattle, so you put them into a kubernetes cluster, which has become your new pet. If all your infra fits on one machine, it's way easier to have that as your pet the same way it's easier to have a dog than run a livestock operation.
Re: Slack’s migration to a cellular architecture
#213Initially read this as: "Slack's Migration to Cellular Automata" and now I'm a little disappointed.
Re: Slack’s migration to a cellular architecture
#214Cell-based architecture introduces modular 'cells' into software systems, each with distinct APIs. This design fosters loose coupling and scalability – key for today's dynamic software landscape. Particularly, for those intrigued by microservices, cells align seamlessly with the independent, scalable components that power microservices architectures.
Curious to dive deeper? If you're keen to explore the nitty-gritty technicalities, I invite you to check out the architecture paper https://github.com/wso2/reference-architecture/blob/master/r... for an in-depth understanding. Let's kick-start this dialogue on the potential of cell-based architecture and its impact on modern software design. Feel free to join the conversation!
Re: Slack’s migration to a cellular architecture
#215Their siloing strategy, which I'll roughly refer as resolving a request from a single AZ, is a good way to keep operations and monitoring simple. A past team of mine managed services in a similar fashion. We had a couple (usually 2-4) single AZ clusters with a thin (Envoy) layer to balance traffic between clusters. We could detect incidents in a single cluster by comparing metrics across clusters. Mitigation was easy…
I assume you were using AWS? I know some of the AZ of other cloud providers (Azure? Oracle? Google?) are not fully siloed. They might have independent power and networking, but be in the same physical location. I'm mentioning this for other people to be aware as one can easily make the assumption that an AZ is the same concept on all clouds, which is not true and painful to realise.
Re: Slack’s migration to a cellular architecture
#216Earlier quoted context omitted.
AZs are buildings often times right next to each other on the same street. People who think this is a great failure domain for your entire business are deeply misguided. All it takes is a hurricane, a truck hitting a pole, a fire, or any number of extremely common situations and infra will be wiped off the map. Build stuff to be properly multi-region.
> AZs are buildings often times right next to each other on the same street. Not at AWS: https://aws.amazon.com/about-aws/global-infrastructure/regio... > An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. AZs give customers the ability to operate production applications and databases that are more highly available, fault tolerant, and s…
Azure doesn't guarantee distance between zones, but they aim for 483km. So they can provide better isolation at the cost of higher inter-AZ latency. Depends on the region - you'd need an internal contact (and/or NDA?) to get approx numbers
Re: Slack’s migration to a cellular architecture
#217Earlier quoted context omitted.
I noticed that the hack blog ( https://hhvm.com/blog/ ) basically stopped posting updates since the end of 2022. As downstream users of hacklang development have you folks noticed a change in development pace or ambition within the hack development team?
I too am super curious about this. Plus, it seems telling that Threads was developed in Python - not Hack. (I’m aware IG is Python & it’s the same team)
Re: Slack’s migration to a cellular architecture
#218Earlier quoted context omitted.
i mean you could also just spin up a reeeeeally big compute node and just do it all there. fewer things to monitor. fewer things that can fail. just log in from time to time to update packages. see, cloud doesn’t have to be complex.
I'm not a cloud guy, but if you're going to put everything in one region, what's stopping you from shoving everything into a bunch of containers on a t2.2xlarge instance (or equivalent), and adding like one CloudWatch alarm (or equivalent) that reboots the instance if it stops responding?
(1) Raw scale might mean you just plain can't fit everything on a single t2.2xlarge.
(2) Different services (containers) might have different performance profiles, so you may want a few different types of machines around.
(3) You probably still want N+2 redundancy even within your single AZ, so this scheme should at least be upgraded to three t2.2xlarge boxes. ;)
Re: Slack’s migration to a cellular architecture
#219Earlier quoted context omitted.
What mess? That sounds like a healthy internal language ecosystem to me. You need at least 2 primary languages to avoid accidental lock-in and maintain good developer diversity. That very paragraph is a great example of how the diversity helped them avoid the trap of plumbing it through their RPCs.
You're suggesting that needing to reimplement the same thing 5 times for every single language in use is a hallmark of a "healthy internal language ecosystem"?