Slack’s migration to a cellular architecture • Hacker News Board

Re: Slack’s migration to a cellular architecture

#181

Post by anon84873628 » Sat, Aug 26, 2023, 10:48 PM UTC

endymi0n wrote: post #167

athoscouto wrote: post #35

Their siloing strategy, which I'll roughly refer as resolving a request from a single AZ, is a good way to keep operations and monitoring simple. A past team of mine managed services in a similar fashion. We had a couple (usually 2-4) single AZ clusters with a thin (Envoy) layer to balance traffic between clusters. We could detect incidents in a single cluster by comparing metrics across clusters. Mitigation was easy…

It took me some time to realize that Cloud Solution Architects are also just slightly more technical sales people in disguise whose only mission is upselling you onto more dependency. Same thing about their PR, every CxO these days says they need "multi-cloud", whatever that means and the costs are usually enormous, while complexity rises — with questionable benefit. I did the math for our own stack and after a setba…

Yeah, for many businesses it probably isn't necessary to have crazy short RTO and RPO. Just restore the most recent backup in a new region and point at the cloud provider outage report...

Re: Slack’s migration to a cellular architecture

#182

Post by xwowsersx » Sat, Aug 26, 2023, 10:50 PM UTC

dcgudeman wrote: post #136

muglug wrote: post #17

Earlier quoted context omitted.

Yes — see my recent article https://slack.engineering/hakana-taking-hack-seriously/ We use a few languages to serve client requests, but by far the biggest codebase is written in Hack, which runs inside an interpreter called HHVM that’s also used at Facebook.

I noticed that the hack blog ( https://hhvm.com/blog/ ) basically stopped posting updates since the end of 2022. As downstream users of hacklang development have you folks noticed a change in development pace or ambition within the hack development team?

Kinda makes sense you would use PHP, even though I'm sure many people are shocked by it. PHP was pretty much born in a web context. The language was created with servers and request/response in mind and it shows.

Re: Slack’s migration to a cellular architecture

#183

Post by xwowsersx » Sat, Aug 26, 2023, 10:57 PM UTC

nostrebored wrote: post #117

hotnfresh wrote: post #81

Earlier quoted context omitted.

Meh. As long as you’ve got a good, typed interface for passing messages between them and for having a common understanding of (and versioning system for) key data structures, that’s fine for this sort of thing where it’s largely processing steams of small messages and events. … but it’s probably JSON and some JSON-Schema-based “now you have two problems” junk instead of what I described. In which case, yeah, ew, gros…

There are tons of approaches to align on service contracts for JSON based API calls. There’s also libraries like gRPC which help make contacts explicit. Neither are really uncommon

What are some of those approaches? Are there formal methods and/or tools for doing this?

Re: Slack’s migration to a cellular architecture

#184

Post by ladzoppelin » Sat, Aug 26, 2023, 10:57 PM UTC

lopkeny12ko wrote: post #159

nostrebored wrote: post #119

Earlier quoted context omitted.

What do you mean? Slack is one of the most responsive and reliable tools I touch every day.

I hope this is satire. Slack is one of the slowest work tools I've ever used. Every interaction and click visibly lags. It's a sad state of the world that almost every application now is written in Javascript and deployed with Electron, and massive memory usage and slow UIs have become accepted as the norm. Try any IRC client and tell me, with a straight face, that Slack is just as responsive.

So I only use Firefox and the Slack web client and don't experience any lag. I am surprised so many people use the Slack app over a web tab.

Re: Slack’s migration to a cellular architecture

#185

Post by [deleted] » Sat, Aug 26, 2023, 10:57 PM UTC

alberth wrote: post #61

muglug wrote: post #17

Earlier quoted context omitted.

Yes — see my recent article https://slack.engineering/hakana-taking-hack-seriously/ We use a few languages to serve client requests, but by far the biggest codebase is written in Hack, which runs inside an interpreter called HHVM that’s also used at Facebook.

Hi Matt Thanks for Psalm! Curious, if Slack was built today from ground up - what tech stack do you think should/would be used?

[deleted]

Re: Slack’s migration to a cellular architecture

#186

Post by xwowsersx » Sat, Aug 26, 2023, 10:59 PM UTC

mrkeen wrote: post #68

chrisweekly wrote: post #37

I appreciate the clear explanation of the problem and the solution, which (as is so often the case) seems fairly simple or obvious in retrospect. Semi-related tangent: sometime around mid-2016, I came across a tool that helped visualize requests in near real-time, and showed what it "looks" like (ie, flow slows to trickle in service A during draining, while it ramps up in service B)... there was a really compelling d…

Vizceral

Neat.

> If a graph of nodes and edges with data about traffic volume is provided, it will render a traffic graph animating the connection volume between nodes.

How would one go about providing such a graph? :)

Re: Slack’s migration to a cellular architecture

#187

Post by xwowsersx » Sat, Aug 26, 2023, 11:03 PM UTC

zo1 wrote: post #62

robertlagrant wrote: post #23

Earlier quoted context omitted.

Teams is doing well because it's often an IT department's simplest choice, but I don't find it's great for users.

Why would I choose Slack for my employees when Teams integrates so nicely with everything else in the "stack". Teams is leaps and bounds ahead already, and Slack really lost the boat many years ago. Speaking of which, I'm going now to buy more Microsoft shares.

The boat or the moat?

Re: Slack’s migration to a cellular architecture

#188

Post by lordofnorn » Sat, Aug 26, 2023, 11:04 PM UTC

catchnear4321 wrote: post #179

lordofnorn wrote: post #124

Earlier quoted context omitted.

Yeah, keep stateful stuff and stateless stuff separate; separate clusters, network spaces, cloud accounts, likely a mix of all that. Clearly define boundaries and acceptable behavior within boundaries. Setup up telemetry and observability to monitor for threshold violations. Simple. Right?

i mean you could also just spin up a reeeeeally big compute node and just do it all there. fewer things to monitor. fewer things that can fail. just log in from time to time to update packages. see, cloud doesn’t have to be complex.

Hey you know what; you do you.

I’m an EE by education. It’s electron state, silly leaky abstraction Stan’ing to my head.

Different babble for allocation of memory and algorithmic manipulation of the values stored within.

Correctness is important when it comes to results being mapped to human consumption and even then the subset of parameters to be be rigorous with can be made subj. personally I lean on a subset that includes biological health and well being and deprioritize religiosity

Re: Slack’s migration to a cellular architecture

#189

Post by theamk » Sat, Aug 26, 2023, 11:06 PM UTC

catchnear4321 wrote: post #179

lordofnorn wrote: post #124

Earlier quoted context omitted.

Yeah, keep stateful stuff and stateless stuff separate; separate clusters, network spaces, cloud accounts, likely a mix of all that. Clearly define boundaries and acceptable behavior within boundaries. Setup up telemetry and observability to monitor for threshold violations. Simple. Right?

i mean you could also just spin up a reeeeeally big compute node and just do it all there. fewer things to monitor. fewer things that can fail. just log in from time to time to update packages. see, cloud doesn’t have to be complex.

until you get "hardware failure" note from cloud provider. Or the person updating packages makes a typo and messes up a system.

sure, use "pet" computers for experiments and dev.. but having produluction be a "cattle" makes your life so much less stressful.

Re: Slack’s migration to a cellular architecture

#190

Post by deanCommie » Sat, Aug 26, 2023, 11:09 PM UTC

oceanplexian wrote: post #161

deanCommie wrote: post #52

Earlier quoted context omitted.

the "whole" of USE1 very rarely goes down [0], because unlike other cloud providers, Amazon's availability zones are actually independent and decoupled, and if you're running on EC2 in a zonal way it's highly unlikely an outage will affect multiple zones. [0] There are of course exceptions that come once every few years, but most instances people can think of in terms of widespread outages is one specific service goi…

AZs are buildings often times right next to each other on the same street. People who think this is a great failure domain for your entire business are deeply misguided. All it takes is a hurricane, a truck hitting a pole, a fire, or any number of extremely common situations and infra will be wiped off the map. Build stuff to be properly multi-region.

> AZs are buildings often times right next to each other on the same street.

Not at AWS: https://aws.amazon.com/about-aws/global-infrastructure/regio...

> An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. AZs give customers the ability to operate production applications and databases that are more highly available, fault tolerant, and scalable than would be possible from a single data center. All AZs in an AWS Region are interconnected with high-bandwidth, low-latency networking, over fully redundant, dedicated metro fiber providing high-throughput, low-latency networking between AZs. All traffic between AZs is encrypted. The network performance is sufficient to accomplish synchronous replication between AZs. AZs make partitioning applications for high availability easy. If an application is partitioned across AZs, companies are better isolated and protected from issues such as power outages, lightning strikes, tornadoes, earthquakes, and more. AZs are physically separated by a meaningful distance, many kilometers, from any other AZ, although all are within 100 km (60 miles) of each other.

This is unique compared to Microsoft, and Google (a single flood taking out multiple AZ's? Uh oh: https://www.theregister.com/2023/04/26/google_cloud_outage/)

Sure, a massive earthquake or a nuclear strike could probably take out several.