Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

Post reply on HN
Print view View original

151–160 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#151

Post by k__ »

qppo wrote: post #139
k__ wrote: post #85

After I used Discord in different contexts for months now (and Slack for years), I can't understand why someone willingly chooses Slack. It's the Atlassian of chat tools. Horrible performance and bad usabillity.

Saved chat history is the big one. Discord is not suitable for business comms without it. Other than that: - shared channels between workspaces - threads - private messages within a workspace, instead of globally - decoupled accounts from workspaces, so I can use my personal email and work email associated slack workspaces at the same time - much tighter integration with third party tools eg zoom, webex, etc Discord…

How come? Me phoning or talking with a customer/co-worker directly is also "suitable for business".

Re: A terrible, horrible, no-good, very bad day at Slack

#152

Post by jiveturkey »

sudhirj wrote: post #31

This is one of the biggest arguments I see for serverless (AWS Lambda + DynamoDB) or at least managed PaaS systems (Google App Engine, Heroku with RDS or CloudSQL). These systems may seem to cost more for some workload curves (or might even be cheaper for your curve), but the difference is worth it because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and b…

> you get their full benefit

and their full downside.

what, prey tell, downside?

they operate at a scale (since you mentioned it) much larger than yours. you don't benefit from scale beyond what you need, you "only" benefit from the SLA.

you don't get to tell them what to do, to set their priorities on features vs bugs vs performance vs meaningless metric of the day.

you don't get to interact consistently with the same person or same set of staff, to understand their foibles and to nudge effectively.

you don't get to decide what features are critical to you and cannot, ever, ever be cut no matter how otherwise impactful they are on the environment.

you don't get to set the timetable for "events".

it's not the absolute no-brainer you are making it out to be.

that said, i agree that the value is solidly there for those in the fat part of the bell curve.

Re: A terrible, horrible, no-good, very bad day at Slack

#153

Post by qppo »

k__ wrote: post #151
qppo wrote: post #139

Earlier quoted context omitted.

Saved chat history is the big one. Discord is not suitable for business comms without it. Other than that: - shared channels between workspaces - threads - private messages within a workspace, instead of globally - decoupled accounts from workspaces, so I can use my personal email and work email associated slack workspaces at the same time - much tighter integration with third party tools eg zoom, webex, etc Discord…

How come? Me phoning or talking with a customer/co-worker directly is also "suitable for business".

Chat history is pretty important for your HR and general counsel if there's instances of misconduct or abuse. Discord doesn't support the same level of chat history/logging that businesses (of all sizes) care about.

Re: A terrible, horrible, no-good, very bad day at Slack

#154

Post by kevmo314 »

Pxtl wrote: post #149

Not being involved in this kind of scaled-up devops kind of stuff, my read of this article was "HAProxy sounds awful "... but they noted in the article 1) Migrating to better software, and 2) Newer versions of HAProxy are also better software. But yeah, the first half where they're discussing "we don't do X, Y, or Z clean, idempotent way of doing config because the HAProxy doesn't perform right if you do it that way.…

I don't think that's fair. HAProxy is good and reliable. It's just old for the use cases we need today. Even the article has this opinion:

> While HAProxy has served us well and reliably for many years, it also has some operational sharp edges

Saying it sounds awful is like saying Vanilla JS sounds awful because it doesn't scale to the demanding webapp use cases of today.

And, as you mentioned, it's really compounded by the fact that they're using an outdated version of HAProxy, but that doesn't make it awful software.

Re: A terrible, horrible, no-good, very bad day at Slack

#155

Post by soamv »

awill wrote: post #137
Jgrubb wrote: post #103

Earlier quoted context omitted.

I work for a PaaS. - The ideal fit for any hosting PaaS is a company who has a large hosting and infra footprint but for whom the technology is _not_ the core competency of the business. Slack is very much better off running their own systems with their own people. - As someone who deals with customers every day I can tell you that yes - we know our platform specifically and how the internet works generally better th…

>>"but we do not know _your application_ at all." You're missing the point. A PaaS, or serverless service doesn't need to know your application. That's the whole point. They're just API calls, and they need to succeed with consistently low latency.

> consistently low latency

That's (a) just one objective and (b) too vaguely stated. What's an acceptable p95 latency for a FaaS? For a database write? For a message queue? The answer is a very big "it depends". And why are we only talking about latency?

There are tradeoffs literally everywhere you look in this space, and not knowing enough about the application's performance, reliability, and efficiency goals can sometimes be a real hindrance to being good at running that application.

Re: A terrible, horrible, no-good, very bad day at Slack

#156

Post by johannes1234321 »

clemParis wrote: post #148
johannes1234321 wrote: post #145

Earlier quoted context omitted.

Question on how they measure - I'm in 7 slack workspaces. Do I count 7 times?

Well, I have to create a new account every time I join a new group, so I'd say yes.

Two if my workspaces are tied to the same enterprise account, they should be the same, but who knows how they measure. My base assumption is "in a way giving nicer numbers"

Re: A terrible, horrible, no-good, very bad day at Slack

#157

Post by EB66 »

yjftsjthsd-h wrote: post #133
ryan_lane wrote: post #116

Earlier quoted context omitted.

Just because you're using managed services doesn't mean it's going to be considerably more expensive, when you consider labor costs, and general reliability. Your service will likely be more reliable if you use DynamoDB or AuroraDB. Your service will probably be more reliable if you build it in a way that assumes nodes will die at any point, will automatically come back in, and can scale up/down. It'll likely be more…

(Prices based on https://calculator.aws/#/createCalculator and https://www.hetzner.com/dedicated-rootserver?country=us ) An AuroraDB db.r5.xlarge with 10TB of storage, reserved instances 1Y term but no up-front, costs 1,301.40 USD per month. Take a Hetzner AX161 with 4x3.84 TB SATA SSD, using RAIDZ for 11.52TB usable storage (and 4 times the RAM), at €297.00 per month... so 335.88 USD per month. That's a difference o…

100% this. Cloud vendor lock-in is absolutely a relevant factor for consideration -- especially when the non-cloud alternatives are ~10x cheaper.

At my company we made the decision to stick primarily with managed dedicated servers over AWS in our very early days. Now we're a decent size a few years later (25 employees) and the cost savings we're realizing are tremendous. We did the math and found that if we had gone with AWS in the early days then we would now conservatively be paying an extra $165,000 on our hosting bill annually.

We still use AWS for some specialized services (e.g., Lex) but the bulk of our stack runs on gear that we now colocate for a fraction of the cost.

Re: A terrible, horrible, no-good, very bad day at Slack

#158

Post by jessaustin »

k__ wrote: post #151
qppo wrote: post #139

Earlier quoted context omitted.

Saved chat history is the big one. Discord is not suitable for business comms without it. Other than that: - shared channels between workspaces - threads - private messages within a workspace, instead of globally - decoupled accounts from workspaces, so I can use my personal email and work email associated slack workspaces at the same time - much tighter integration with third party tools eg zoom, webex, etc Discord…

How come? Me phoning or talking with a customer/co-worker directly is also "suitable for business".

I think this is why we often get that "this call may be recorded for training purposes" message while waiting in the call center queue? Even if they're not actually keeping all those recordings, they can pass the legal test by saying that they do.

Re: A terrible, horrible, no-good, very bad day at Slack

#159

Post by xorcist »

harpratap wrote: post #105
xorcist wrote: post #56

Autoscaling is hard. Never ever use one that you don't thoroughly understand. An autoscaler that keeps chugging when deploys aren't green is outright dangerous.

I think this one is more of a service discovery bug than auto-scaling.

Perhaps you know more than is in this blog post, but it sounds more like a rather standard load balancer.

Deployment broke. Yeah, that probably should have been caught. Even if it didn't, monitoring should have caught the stale load balancer config. It didn't, for some reason unknown to us.

These things happen. Things break. The autoscaler then proceeded to kill customer traffic. That was the part that worked as designed, so another design would have avoided escalating the situation (if you forgive some armchair engineering here).

Re: A terrible, horrible, no-good, very bad day at Slack

#160

Post by waprin »

iampims wrote: post #125
sudhirj wrote: post #114

Earlier quoted context omitted.

So this is where 12factor comes in - unless the application is operating at lower than layer 7 (and with some of the newer offerings even layer 3) there’s not much technology centric stuff going on. Everything Slack is doing is happening at the HTTP / gRPC / Websocket level, and it’s hard to make the case for self managed hosting. The idea of requests going to an application server on a TCP or HTTP connection with th…

If I had To guess, I’d say that Slack’s infrastructure footprint is probably as big if not bigger than any popular PaaS.

Snapchat runs on App Engine (and other stuff runs on App Engine too) so App Engine is almost certainly bigger than Slack just based on that.
Post reply on HN