Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s migration to a cellular architecture

slack.engineering

Post reply on HN
Print view View original

191–200 of 265 posts

Re: Slack’s migration to a cellular architecture

#191

Post by FridgeSeal »

nostrebored wrote: post #119
skullone wrote: post #89

Earlier quoted context omitted.

It even pains me to see they're suffering from so many own goals. And it's unfortunately reflected in the poor experience using the Slack client. Not to mention the multiple deprecated bot/integration APIs with such bad feature parity between all the different ways to integrate your own tooling into Slack.

What do you mean? Slack is one of the most responsive and reliable tools I touch every day.

Ok genuine question: what other tools do you use?

Slack won’t seem that bad, until you use something that’s actually good.

The loss of performance on commonplace applications has been a real “boil the frog” situation; we’ve lost so much performance and responsiveness, but it’s happened so gradually that most people don’t notice.

Re: Slack’s migration to a cellular architecture

#192

Post by zomglings »

heywhatupboys wrote: post #10

Is Slack dead? unironically. Does it have a future? With Teams, etc. coming out, it seems most companies do not want to go the Slack route

I started using Slack in 2015 and thought it was a great product.

Since then, they have hit 2 home runs on top of their basic chat functionality:

1. Slack Connect: Being able to share channels between workspaces is simply amazing. Most of our customers are on Slack, and having a Slack connection to them makes it much easier to communicate with them and get their feedback as we improve our product. I don't know any other tool that even comes close to how important Slack Connect has been to product development in my startup.

2. Canvas: They rolled this out last year as notes or something, and I was pretty underwhelmed with the experience at first. But very recently (within the past month I think) they reintroduced this as "canvas" with really tight integration with threads. We have moved all of our planning and synchronization activity to a canvas that we set up every week.

Although these features are not difficult for Google or Microsoft to implement on a purely technical level, their product organizations don't seem to understand the network effects of chat the way that Slack's product organization does.

Slack is certainly not dead today and they are showing the savvy to stay alive well into the future.

Re: Slack’s migration to a cellular architecture

#193

Post by ryukoposting »

catchnear4321 wrote: post #179
lordofnorn wrote: post #124

Earlier quoted context omitted.

Yeah, keep stateful stuff and stateless stuff separate; separate clusters, network spaces, cloud accounts, likely a mix of all that. Clearly define boundaries and acceptable behavior within boundaries. Setup up telemetry and observability to monitor for threshold violations. Simple. Right?

i mean you could also just spin up a reeeeeally big compute node and just do it all there. fewer things to monitor. fewer things that can fail. just log in from time to time to update packages. see, cloud doesn’t have to be complex.

I'm not a cloud guy, but if you're going to put everything in one region, what's stopping you from shoving everything into a bunch of containers on a t2.2xlarge instance (or equivalent), and adding like one CloudWatch alarm (or equivalent) that reboots the instance if it stops responding?

Re: Slack’s migration to a cellular architecture

#194

Post by ak007 »

AugustoCAS wrote: post #177
athoscouto wrote: post #35

Their siloing strategy, which I'll roughly refer as resolving a request from a single AZ, is a good way to keep operations and monitoring simple. A past team of mine managed services in a similar fashion. We had a couple (usually 2-4) single AZ clusters with a thin (Envoy) layer to balance traffic between clusters. We could detect incidents in a single cluster by comparing metrics across clusters. Mitigation was easy…

I assume you were using AWS? I know some of the AZ of other cloud providers (Azure? Oracle? Google?) are not fully siloed. They might have independent power and networking, but be in the same physical location. I'm mentioning this for other people to be aware as one can easily make the assumption that an AZ is the same concept on all clouds, which is not true and painful to realise.

Thanks for highlighting this. Indeed all CSPs are not the same

Re: Slack’s migration to a cellular architecture

#195

Post by jldugger »

ec109685 wrote: post #151
jldugger wrote: post #85

Earlier quoted context omitted.

Indeed, this is the main problem I run into. We have to scale up capacity before the traffic can be redirected or you basically double the scope of the outage briefly. Which involves multiple layers of capacity bringup -- ASG brings up new nodes, then HPA brings up the new pods.

If there’s uncorrelated load you can also run on your hosts, then you can share their spare capacity, with the hope they don’t spike at same time. AWS does that with their lambda arch to reduce waste.

Maybe, but the cost accounting is already a nightmare.

Re: Slack’s migration to a cellular architecture

#196

Post by jldugger »

nostrebored wrote: post #119
skullone wrote: post #89

Earlier quoted context omitted.

It even pains me to see they're suffering from so many own goals. And it's unfortunately reflected in the poor experience using the Slack client. Not to mention the multiple deprecated bot/integration APIs with such bad feature parity between all the different ways to integrate your own tooling into Slack.

What do you mean? Slack is one of the most responsive and reliable tools I touch every day.

Well, status.slack.com says they're currently having an outage, that has been ongoing for multiple days.

Re: Slack’s migration to a cellular architecture

#197

Post by pm »

WinLychee wrote: post #108
muglug wrote: post #17

Earlier quoted context omitted.

Yes — see my recent article https://slack.engineering/hakana-taking-hack-seriously/ We use a few languages to serve client requests, but by far the biggest codebase is written in Hack, which runs inside an interpreter called HHVM that’s also used at Facebook.

PHP has some excellent ideas that other languages can't replicate, while at the same time having terrible ideas that other languages don't have to think about. Overall a huge fan of modern PHP, thanks for this writeup.

Which excellent ideas does it have that other languages can't replicate?

Re: Slack’s migration to a cellular architecture

#198

Post by jeremyjh »

AugustoCAS wrote: post #177
athoscouto wrote: post #35

Their siloing strategy, which I'll roughly refer as resolving a request from a single AZ, is a good way to keep operations and monitoring simple. A past team of mine managed services in a similar fashion. We had a couple (usually 2-4) single AZ clusters with a thin (Envoy) layer to balance traffic between clusters. We could detect incidents in a single cluster by comparing metrics across clusters. Mitigation was easy…

I assume you were using AWS? I know some of the AZ of other cloud providers (Azure? Oracle? Google?) are not fully siloed. They might have independent power and networking, but be in the same physical location. I'm mentioning this for other people to be aware as one can easily make the assumption that an AZ is the same concept on all clouds, which is not true and painful to realise.

Actually I assumed AWS did it the same way as the others - I thought maybe they are in another building on a campus but I didn’t think that should be a factor in planning and that I should use regions for geographic redundancy anyway.

Re: Slack’s migration to a cellular architecture

#199

Post by kccqzy »

ec109685 wrote: post #149
kccqzy wrote: post #110

Earlier quoted context omitted.

That's another way of saying your typical utilization ratio is 66%. Which is on the low side honestly. That said, it's a trade off between efficiency and load spike tolerance. I trust that the trade off is made with informed decision.

66% isn’t low utilization. You’re always going to have micro spikes, and you never want to clip, so keeping some headroom around feels smart. Unless you co-mingle online and offline (batch) traffic on same hosts, flat response times and high utilization aren’t compatible.

While I agree there's always going to be micro spikes and keeping some headroom is smart, 33% may be too much of a headroom for all but the most latency sensitive RPC services. Personally I aim for only 20% headroom.

Re: Slack’s migration to a cellular architecture

#200

Post by calrueb »

fiddlerwoaroof wrote: post #144
tmpX7dMeXU wrote: post #134

Earlier quoted context omitted.

Not to “works on my machine” you, but I…genuinely do not have these problems. I’ve never heard it from my team either. So we could at the very least say it’s not a widespread global issue. Even the percentage of nerds that would want IRC or XMPP bridges back would have to be vanishingly small. I’d be annoyed if Slack reimplemented such functionality because it no doubt slows down future development. Slack has a numbe…

So, it's workspace and user/device specific: two of the workspaces I interact with regularly have these problems and the problems also show up intermittently for some users and not others. (Anecdotally, my experience is that Matrix/Element used to be annoying compared to the Slack experience and now I mostly prefer it to Slack) I would be fine with the understanding that the IRC bridge was missing functionality (and…

I’m gonna go to bat for Slack on this one and say the “Later” feature they added recently has completely changed my workflow for the better. It’s so simple, but removes all the cognitive overhead of feeling pressured to deal with specific messages real time, else I forget about them. Now I just throw the message in Later and get back to it when I’m free.
Post reply on HN