Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

Post reply on HN
Print view View original

171–180 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#171

Post by scaryclam »

StavrosK wrote: post #87
k__ wrote: post #85

After I used Discord in different contexts for months now (and Slack for years), I can't understand why someone willingly chooses Slack. It's the Atlassian of chat tools. Horrible performance and bad usabillity.

I use Zulip for the day to day (it's amazing, I can't recommend it enough), but sometimes use Slack because some open source communities use it, and I'm always amazed at how damn slow it is. I can consistently out-type it, it's terrible. I guess it was great when it started out, but they're slowly boiling the frog, who is us.

That's one thing that makes me quite sad, to see open source communities adopt a closed chat tool. I know that it's easy and zero-maintenance for them, but I can't help but feel there are better options.

Re: A terrible, horrible, no-good, very bad day at Slack

#172

Post by lewis1028282 »

yjftsjthsd-h wrote: post #133
ryan_lane wrote: post #116

Earlier quoted context omitted.

Just because you're using managed services doesn't mean it's going to be considerably more expensive, when you consider labor costs, and general reliability. Your service will likely be more reliable if you use DynamoDB or AuroraDB. Your service will probably be more reliable if you build it in a way that assumes nodes will die at any point, will automatically come back in, and can scale up/down. It'll likely be more…

(Prices based on https://calculator.aws/#/createCalculator and https://www.hetzner.com/dedicated-rootserver?country=us ) An AuroraDB db.r5.xlarge with 10TB of storage, reserved instances 1Y term but no up-front, costs 1,301.40 USD per month. Take a Hetzner AX161 with 4x3.84 TB SATA SSD, using RAIDZ for 11.52TB usable storage (and 4 times the RAM), at €297.00 per month... so 335.88 USD per month. That's a difference o…

Yeah but why pay someone to look after databases when I could get them do something else that'll have way more value. Using managed services just reduces how much money I have to spend upfront.

Re: A terrible, horrible, no-good, very bad day at Slack

#173

Post by StavrosK »

scaryclam wrote: post #171
StavrosK wrote: post #87

Earlier quoted context omitted.

I use Zulip for the day to day (it's amazing, I can't recommend it enough), but sometimes use Slack because some open source communities use it, and I'm always amazed at how damn slow it is. I can consistently out-type it, it's terrible. I guess it was great when it started out, but they're slowly boiling the frog, who is us.

That's one thing that makes me quite sad, to see open source communities adopt a closed chat tool. I know that it's easy and zero-maintenance for them, but I can't help but feel there are better options.

There certainly are, Zulip is OSS and so is Matrix.

Re: A terrible, horrible, no-good, very bad day at Slack

#174

Post by aflag »

saurik wrote: post #72
aflag wrote: post #66

Earlier quoted context omitted.

Though they do have good availability, they are certainly not infallible and have been down for hours or even days. And, when that happens, all you can do is pray they get to it soon. You have no control over when they will make riskier changes or how fast they will be able to respond. If they fail to respect the SLA, the maximum they are going to do is giving your money back for your services. Obviously, there are g…

FWIW, that time AWS had a massive massive failure it was actually really convenient being on AWS as so much of the Internet was down at the same time that pretty much no one was upset at me for my one service also being offline.

I think it depends on your competitors and what your software does. If your competitors are still running their services but not you, you're bound to lose some customers. Also, regardless of that, you can easily be losing money while the service is down. For instance, if you have a food delivery app and it's down, while your competitor is up, people will just use the other app and you're going to lose money while AWS is down. Even worse, they may actually end up liking the competitor's app better, and you can even lose customers that way.

However, if you're looking from the point of view of an employee trying to justify to your boss why the service is down. I'm sure they'll be more understanding that's not your fault in this case. It's a risk they decided to take.

Re: A terrible, horrible, no-good, very bad day at Slack

#175

Post by jacquesm »

marcosdumay wrote: post #126
tinco wrote: post #40

Earlier quoted context omitted.

Would you consider it evil to use production backups for the QA step in CI/CD? It would catch two birds with one stone, continuously verifying the backups, and ensuring the new code works on real world data. We don't have any personal information in our production database, but even if we did, as long as the QA is thoroughly prevented from interacting with the outside world, it can't hurt to use production data right…

If you have a procedure for deriving QA data from production, change it to derive from the backup. Using it directly is full of dangers, not only of leaking information, but also of corrupting your backups. (Otherwise, why are you testing anything?) And deriving test data from production looks like a good thing to me, but make sure to restrict the access to the test environment and mask your data.

You should not use production data for QA, period. Use generated data.

If you really must use real data make sure that (1) you have explicit direction to do this from those higher up in the food chain so that if there ever is a breach of your test systems you don't end up holding the bag and (2) anonymize a copy of the data before it leaves the production system. That way you minimize the risk. But better: simply don't do it, generate your test data.

Re: A terrible, horrible, no-good, very bad day at Slack

#176

Post by dang »

chrismorgan wrote: post #6
kd5bjo wrote: post #2

The title is an allusion to a popular children’s book[1]. I’m assuming that an automated algorithm pulled the “very;” hopefully the mods will consider restoring it. [1] https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...

HN automatically mangles titles in various ways I’m not fond of, dropping words that might or might not be significant, fiddling with capitalisation, &c. , but the submitter can go back and edit the title back to what it was supposed to be, and the one time I’ve done that the system didn’t mangulate it again.

That's by design, because the software is obviously imperfect. It does more good than harm, though, so we keep it.

Re: A terrible, horrible, no-good, very bad day at Slack

#177

Post by dang »

kd5bjo wrote: post #2

The title is an allusion to a popular children’s book[1]. I’m assuming that an automated algorithm pulled the “very;” hopefully the mods will consider restoring it. [1] https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...

Ok, we've re-veried the title above.

Re: A terrible, horrible, no-good, very bad day at Slack

#178

Post by whatsmyusername »

jessaustin wrote: post #161
whatsmyusername wrote: post #117

Earlier quoted context omitted.

Because it's the Atlassian of chat tools. It's the lowest common denominator. Discord also doesn't let you set message retention that I'm aware of which is an immediate nonstarter. End up in one lawsuit where the other party demands a fishing expedition and you'll be real happy that you've got retention limited to 90 days by policy and in practice.

If it's legal to set this to 90 days, why can't a firm set it to one or zero days?

You could, and I'm sure some places have a business case for this.

Most need stuff going back at least a little bit. If I was talking to you about something on Friday and wanted to reference the conversation Monday I'd be real mad if the convo was already deleted.

Re: A terrible, horrible, no-good, very bad day at Slack

#179

Post by buzzert »

carterklein13 wrote: post #167
k__ wrote: post #96

Earlier quoted context omitted.

I'm not saying Discord is perfect or "always up", it's just that their client UI is better structured and more responsive in most cases. I am on multiple OSS Discord servers with thousands of users, and it works just fine most of the time. I am on multiple Slack servers with just 10-20 users and it is unbearably slow.

I think Discord's Achilles' heel for enterprise is its UI unfortunately. There's no way that my company would adopt a platform so "fun" in the way Discord tries to be. Animated characters, a logo which looks like a gamepad, etc... It sucks, because I use Slack for work and mostly Discord for personal use (mostly dev communities for different companies), and Discord is far and away a better experience. If Discord prov…

I log on to Slack at work and see pages of emoji, animated gifs, and animated reactions. What’s the difference?

Re: A terrible, horrible, no-good, very bad day at Slack

#180

Post by jacksonpollock »

cuu508 wrote: post #27
m12k wrote: post #26

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances i…

That's an excellent TL;DR, thank you! Postmortems should start with a summary paragraph like the above, and then go into story and full details below.

agreed!
Post reply on HN