Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

Post reply on HN
Print view View original

131–140 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#131

Post by peterwwillis »

hyperman1 wrote: post #4

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

Google goes over this scenario in their SRE books. You literally plan to take down your service so you never have 100% uptime, and literally step through a real failure and recovery.

In that process you would go, "Ok, the service is failed and I don't know why. How does the monitoring look?", and then you'd notice it was broken. This is best performed by the person on the team with the least experience, so they ask the most questions, which reveals more.

What's really funny is when the recovery you expected to work doesn't work, and then you have bigger problems... But at least you planned for it :)

Re: A terrible, horrible, no-good, very bad day at Slack

#132

Post by user5994461 »

MapleWalnut wrote: post #122

> It’s worth noting that HAProxy can integrate with Consul’s DNS interface, but this adds lag due to the DNS TTL, it limits the ability to use Consul tags, and managing very large DNS responses often seems to lead to hitting painful edge-cases and bugs. I was surprised how they dismissed HAProxy integration with Consul using SRV DNS records. Can anyone confirm the problems they highlight? It seems like their service…

Dynamic DNS support is only since haproxy 1.8, maybe 2 years old give or take. Slack infrastructure must be older than that so they improvised.

There were few bugs in the first implementations, should be good now. Slack don't mention their scale but I can imagine some UDP/DNS edge cases if there are hundreds of instances behind one domain.

Re: A terrible, horrible, no-good, very bad day at Slack

#133

Post by yjftsjthsd-h »

ryan_lane wrote: post #116
yjftsjthsd-h wrote: post #108

Earlier quoted context omitted.

I'd disagree pretty strongly. You may evaluate the tradeoffs and decide to go all-in on a single cloud (and you should do that analysis!), but it's far from a given. You're not wasting engineering time if the costs get out of hand and bankrupt you. Managed services may accelerate your time to market, but they do so at the expense of lock-in and your bill. Context: I work at a startup that benefits enormously by avoid…

Just because you're using managed services doesn't mean it's going to be considerably more expensive, when you consider labor costs, and general reliability. Your service will likely be more reliable if you use DynamoDB or AuroraDB. Your service will probably be more reliable if you build it in a way that assumes nodes will die at any point, will automatically come back in, and can scale up/down. It'll likely be more…

(Prices based on https://calculator.aws/#/createCalculator and https://www.hetzner.com/dedicated-rootserver?country=us)

An AuroraDB db.r5.xlarge with 10TB of storage, reserved instances 1Y term but no up-front, costs 1,301.40 USD per month.

Take a Hetzner AX161 with 4x3.84 TB SATA SSD, using RAIDZ for 11.52TB usable storage (and 4 times the RAM), at €297.00 per month... so 335.88 USD per month.

That's a difference of 965 $/mo = 11,580 $/yr. If you have 10 of these, they'll pay for a full time sysadmin. Now, that's leaving out a lot of details (bandwith costs and application hosting come to mind), and assumes truly massive databases. On the other hand, as that sysadmin, I promise our databases don't take anything like my full attention, and you really should have some sort of sysadmin/ops team anyways (please do not make devs run your AWS infrastructure; it will end in tears for everyone). Every time this argument comes up, people do mention reliability and time spent on maintenance, but... it's really not bad. Hardware doesn't actually fail that much, postgres isn't that complicated to configure, OS patches aren't that hard to apply. Your mileage will vary, but sometimes it's just not worth using AWS. (And sometimes, it really is; if we didn't need to run oversized databases, I'd push us to use AWS in a heartbeat)

Re: A terrible, horrible, no-good, very bad day at Slack

#134

Post by k__ »

rpdillon wrote: post #110
k__ wrote: post #96

Earlier quoted context omitted.

I'm not saying Discord is perfect or "always up", it's just that their client UI is better structured and more responsive in most cases. I am on multiple OSS Discord servers with thousands of users, and it works just fine most of the time. I am on multiple Slack servers with just 10-20 users and it is unbearably slow.

I admined a multi-hundred user Slack org and now am part of a multi-thousand user slack org and I've never experienced (or heard about) the slowness that you're describing. Do you have a feel for whether it was client-side or server-side?

More client-side.

Server-side scaling was only an issue with OSS communities and for hackathons, where >50k people were online.

Re: A terrible, horrible, no-good, very bad day at Slack

#135

Post by bobobob420 »

k__ wrote: post #96
jrockway wrote: post #89

Earlier quoted context omitted.

As someone who uses Discord, I can tell you it goes down too. My favorite part is looking at their status page, seeing that API response time is exceedingly high, and getting no updates from the team about whether or not they're fixing it.

I'm not saying Discord is perfect or "always up", it's just that their client UI is better structured and more responsive in most cases. I am on multiple OSS Discord servers with thousands of users, and it works just fine most of the time. I am on multiple Slack servers with just 10-20 users and it is unbearably slow.

What are some open source discord servers you are a part of? I would like to know about some open source communities if you do not mind.

Re: A terrible, horrible, no-good, very bad day at Slack

#136

Post by codonaut »

iampims wrote: post #125
sudhirj wrote: post #114

Earlier quoted context omitted.

So this is where 12factor comes in - unless the application is operating at lower than layer 7 (and with some of the newer offerings even layer 3) there’s not much technology centric stuff going on. Everything Slack is doing is happening at the HTTP / gRPC / Websocket level, and it’s hard to make the case for self managed hosting. The idea of requests going to an application server on a TCP or HTTP connection with th…

If I had To guess, I’d say that Slack’s infrastructure footprint is probably as big if not bigger than any popular PaaS.

That seems... Questionable.

Re: A terrible, horrible, no-good, very bad day at Slack

#137

Post by awill »

Jgrubb wrote: post #103
sudhirj wrote: post #31

This is one of the biggest arguments I see for serverless (AWS Lambda + DynamoDB) or at least managed PaaS systems (Google App Engine, Heroku with RDS or CloudSQL). These systems may seem to cost more for some workload curves (or might even be cheaper for your curve), but the difference is worth it because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and b…

I work for a PaaS. - The ideal fit for any hosting PaaS is a company who has a large hosting and infra footprint but for whom the technology is _not_ the core competency of the business. Slack is very much better off running their own systems with their own people. - As someone who deals with customers every day I can tell you that yes - we know our platform specifically and how the internet works generally better th…

>>"but we do not know _your application_ at all."

You're missing the point. A PaaS, or serverless service doesn't need to know your application. That's the whole point. They're just API calls, and they need to succeed with consistently low latency.

Re: A terrible, horrible, no-good, very bad day at Slack

#138

Post by deeteecee »

hyperman1 wrote: post #4

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

haha, curious what the actual context behind that statement is. one thing id ask them is does anyone on that team understand how that system works? also, what's the reason it was broken?

Re: A terrible, horrible, no-good, very bad day at Slack

#139

Post by qppo »

k__ wrote: post #85

After I used Discord in different contexts for months now (and Slack for years), I can't understand why someone willingly chooses Slack. It's the Atlassian of chat tools. Horrible performance and bad usabillity.

Saved chat history is the big one. Discord is not suitable for business comms without it.

Other than that:

- shared channels between workspaces

- threads

- private messages within a workspace, instead of globally

- decoupled accounts from workspaces, so I can use my personal email and work email associated slack workspaces at the same time

- much tighter integration with third party tools eg zoom, webex, etc

Discord is great for casual chats with friends or open source communities. Insufficient for business.

Re: A terrible, horrible, no-good, very bad day at Slack

#140

Post by tidepod12 »

freehunter wrote: post #91
bmurphy1976 wrote: post #83

Earlier quoted context omitted.

> I do sometimes wonder what the point is for half the stuff they release. I'm convinced they mvp every possible idea because it makes their platform more sticky. The more services you use, the harder it is to leave them for something better. The problem with that is you get 100 half dead zombie services and it's really hard to know which services are actually supported and which aren't. It's the Amazon equivalent of…

It almost seems to me (as an outsider) that Amazon has a policy that any tool they build internally or for a specific customer also has to be offered to the public on AWS. Like their managed satellite service or any number of things that make you say “I’ll bet some critical AWS thing was the first customer of this hyper-specific product”.

AFAIK this is pretty spot on. I don't know if it's "policy" or what, but based on talking with AWS account managers, AWS does not often (or ever) create custom AWS services/products for individual customers. They do however let customers submit feature requests, and then let customers lean heavily on AWS to implement those feature requests in the public AWS services.

I think this is what leads to a lot of MVP-type services. A large customer clamors for some individual niche feature, AWS implements a MVP version of it, and then the team moves on to the next feature request (which might be for a completely different service, leaving the MVP in a perpetual MVP state).

Post reply on HN