So they run everything in AWS USE1? That doesn't seem very redundant, but then I guess if the whole of USE1 goes down Slack won't be the only service that will be affected.
Slack’s migration to a cellular architecture
41–50 of 265 posts
Re: Slack’s migration to a cellular architecture
#42[flagged]
Re: Slack’s migration to a cellular architecture
#43How can such an architecture function with respect to user data? If the DB instance primary handling your shard is in AZ-1 and AZ-1 gets drained, how can your writes continue to be serviced?
[deleted]
Re: Slack’s migration to a cellular architecture
#44I appreciate the clear explanation of the problem and the solution, which (as is so often the case) seems fairly simple or obvious in retrospect. Semi-related tangent: sometime around mid-2016, I came across a tool that helped visualize requests in near real-time, and showed what it "looks" like (ie, flow slows to trickle in service A during draining, while it ramps up in service B)... there was a really compelling d…
[deleted]
Re: Slack’s migration to a cellular architecture
#45So they run everything in AWS USE1? That doesn't seem very redundant, but then I guess if the whole of USE1 goes down Slack won't be the only service that will be affected.
Isn’t the point of the article that they don’t? And it describes how they implemented region drains to traffic shift between the different regions.
edit: Hmm or maybe not? I still sometimes confuse aws terminology. Perhaps it is all in us-east—1, just in different availability zones (buildings?)
Re: Slack’s migration to a cellular architecture
#46Earlier quoted context omitted.
from the article: >Slack does not share a common codebase or even runtime; services in the user-facing request path are written in Hack, Go, Java, and C++.
Man what a mess. Meanwhile, everyone else can extend a library used by their common services in a common language trivially.
Almost everyone embraced polyglotism and microservices together.
Re: Slack’s migration to a cellular architecture
#47Their siloing strategy, which I'll roughly refer as resolving a request from a single AZ, is a good way to keep operations and monitoring simple. A past team of mine managed services in a similar fashion. We had a couple (usually 2-4) single AZ clusters with a thin (Envoy) layer to balance traffic between clusters. We could detect incidents in a single cluster by comparing metrics across clusters. Mitigation was easy…
It sounds like you didn’t have persistent data, and were only offering compute? If there’s no need for a coherent master view accessible/writeable from all the clusters, there would be no reason to use multi-region cluster whatsoever.
Re: Slack’s migration to a cellular architecture
#48Their siloing strategy, which I'll roughly refer as resolving a request from a single AZ, is a good way to keep operations and monitoring simple. A past team of mine managed services in a similar fashion. We had a couple (usually 2-4) single AZ clusters with a thin (Envoy) layer to balance traffic between clusters. We could detect incidents in a single cluster by comparing metrics across clusters. Mitigation was easy…
The downside of single AZ clusters is capacity. If you have a need to drastically scale up the compute might not be available in a single AZ.
Re: Slack’s migration to a cellular architecture
#49So they run everything in AWS USE1? That doesn't seem very redundant, but then I guess if the whole of USE1 goes down Slack won't be the only service that will be affected.
AWS also uses Slack internally, so add that to the list of shit that can hit the fan if us-east-1/IAD goes down.
Re: Slack’s migration to a cellular architecture
#50So they run everything in AWS USE1? That doesn't seem very redundant, but then I guess if the whole of USE1 goes down Slack won't be the only service that will be affected.
Isn’t the point of the article that they don’t? And it describes how they implemented region drains to traffic shift between the different regions. edit: Hmm or maybe not? I still sometimes confuse aws terminology. Perhaps it is all in us-east—1, just in different availability zones (buildings?)
If I understand it correctly they have an edge network for ingress traffic but host all of their core services in a single AWS region (USE1) in multiple availability zones there.