A terrible, horrible, no-good, very bad day at Slack • Hacker News Board

Re: A terrible, horrible, no-good, very bad day at Slack

#271

Post by StavrosK » Wed, Jul 08, 2020, 5:01 PM UTC

mikecoles wrote: post #270

StavrosK wrote: post #87

Earlier quoted context omitted.

I use Zulip for the day to day (it's amazing, I can't recommend it enough), but sometimes use Slack because some open source communities use it, and I'm always amazed at how damn slow it is. I can consistently out-type it, it's terrible. I guess it was great when it started out, but they're slowly boiling the frog, who is us.

Zulip won my last bake-off for chat systems. Integration was easy and the topic method of providing threads was amazing. The only feature it was missing was federation. In the XMPP world, you could communicate with users on other XMPP instances. With Zulip, you can only communicate with local users. Do you know if this is still the case?

Yes, I doubt federation will ever be added.

Re: A terrible, horrible, no-good, very bad day at Slack

#272

Post by twizzerlora » Wed, Jul 08, 2020, 5:24 PM UTC

mikecoles wrote: post #270

StavrosK wrote: post #87

Earlier quoted context omitted.

I use Zulip for the day to day (it's amazing, I can't recommend it enough), but sometimes use Slack because some open source communities use it, and I'm always amazed at how damn slow it is. I can consistently out-type it, it's terrible. I guess it was great when it started out, but they're slowly boiling the frog, who is us.

Zulip won my last bake-off for chat systems. Integration was easy and the topic method of providing threads was amazing. The only feature it was missing was federation. In the XMPP world, you could communicate with users on other XMPP instances. With Zulip, you can only communicate with local users. Do you know if this is still the case?

Zulip has a cross server bridge using a bot that you might be interested in:

https://github.com/zulip/python-zulip-api/tree/f47559e68dfcc...

Re: A terrible, horrible, no-good, very bad day at Slack

#273

Post by tabbott » Wed, Jul 08, 2020, 6:47 PM UTC

mikecoles wrote: post #270

StavrosK wrote: post #87

Earlier quoted context omitted.

I use Zulip for the day to day (it's amazing, I can't recommend it enough), but sometimes use Slack because some open source communities use it, and I'm always amazed at how damn slow it is. I can consistently out-type it, it's terrible. I guess it was great when it started out, but they're slowly boiling the frog, who is us.

Zulip won my last bake-off for chat systems. Integration was easy and the topic method of providing threads was amazing. The only feature it was missing was federation. In the XMPP world, you could communicate with users on other XMPP instances. With Zulip, you can only communicate with local users. Do you know if this is still the case?

As other folks have mentioned, Zulip has a number of cross-server integrations with both the Zulip protocol and other protocols like XMPP. There's a few we document here as well as Matterbridge:

* https://zulipchat.com/integrations/communication * https://github.com/42wim/matterbridge

We'll eventually add a more native way of connecting a stream between two Zulip servers; we just want to be sure we do that right; federation done sloppily is asking for a lot of spam/abuse problems down the line.

(I'm the Zulip lead developer)

Re: A terrible, horrible, no-good, very bad day at Slack

#274

Post by ueli » Wed, Jul 08, 2020, 7:00 PM UTC

vondur wrote: post #194

m12k wrote: post #26

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances i…

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

Woods' Theorem: As the complexity of a system increases, the accuracy of any single agent's own model of that system decreases rapidly.

from: https://snafucatchers.github.io/

Re: A terrible, horrible, no-good, very bad day at Slack

#275

Post by aagha » Wed, Jul 08, 2020, 9:24 PM UTC

m12k wrote: post #26

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances i…

I used to work for an Independent System Operator (ISO)[0] and we used to have a sometimes painful process for rolling out any changes to production. It's been a while and I don't remember it all, but it went something like this:

1. Fill out out a change request (CR) form and print it.

2. Have it signed by your manager and the managers of every system it touched, including business owners.

3. Attend the 2x a week meeting and explain your CR. In this meeting, explain what was happening, why, who authorized it, what to do if it failed, what to do if it initially worked but failed later (e.g, on a weekend).

4. Hope your CR passes the vote.

5. Implement your roll-out plan.

This is a robust process. Where it breaks down--I felt--is when you need to fix a typo on the public-facing website that's managed by a CMS.

0 - https://www.eesi.org/files/070913_Jay_Caspary.pdf

Re: A terrible, horrible, no-good, very bad day at Slack

#276

Post by antoinealb » Wed, Jul 08, 2020, 10:32 PM UTC

pradn wrote: post #237

cuu508 wrote: post #27

Earlier quoted context omitted.

That's an excellent TL;DR, thank you! Postmortems should start with a summary paragraph like the above, and then go into story and full details below.

The internal Google postmortem format does require a 1-2 sentence summary. Absolutely vital for being able to learn from mistakes. No one is going to parse walls of text when browsing through post-mortems.

Here is an example of such a postmortem format if people wonder: https://landing.google.com/sre/sre-book/chapters/postmortem/

Re: A terrible, horrible, no-good, very bad day at Slack

#277

Post by gutino » Thu, Jul 09, 2020, 8:44 AM UTC

bstar77 wrote: post #251

Nextgrid wrote: post #223

Earlier quoted context omitted.

Which is what always happens when the company aims to build an "engineering playground" full of microservices and other moving parts without a clear technical justification and balancing the pros & cons and why I personally don't like working on such projects - it makes me feel uneasy not having a good understanding of the entire system. To be fair to Slack, at their scale, lots of moving parts might make sense, but…

I think you are drawing the wrong conclusions here... Microservices is not the boogyman here. It more likely has to do with speed of development, developer turnover and a plethora of other things that result in insufficient knowledge transfer. Microservices (like just about anything) can be implemented well or poorly. There's a reason we have sophisticated orchestration solutions like Kubernetes... it exists to tame…

Kubernetes is for scaling, the microservice complexity is about the spaghetti relation graph between them, the mis-consistency effects and so on. Believe me, you better have a infrastructure/development process that justify them.

Re: A terrible, horrible, no-good, very bad day at Slack

#278

Post by cutemonster » Thu, Jul 09, 2020, 1:11 PM UTC

boring_twenties wrote: post #260

cutemonster wrote: post #259

Earlier quoted context omitted.

Hmm I'd guess they have DNS cnames like api1.x.com and api2 and 3, 4 And then the client picks one, and if that server is offline, picks another Seems as simple as DNS based? And works with broken server(s)

Except you don't need that because you can just return all four IP addresses for one record, e.g. api.x.com

I think if such a DNS/ip based round robin server is down, or replies 500 error, the client won't try another server

Unless there's a way to get all ip addrs in js? By custom client code that queries the DNS system?

Re: A terrible, horrible, no-good, very bad day at Slack

#279

Post by zomglings » Thu, Jul 09, 2020, 2:58 PM UTC

harpratap wrote: post #15

hyperman1 wrote: post #4

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

Chaos Engineering comes into play here. Deliberately break your platform to see if everything works as expected.

To catch this particular issue, they would have had to overprovision webapp instances beyond their N*M limit, take down a subset of the originally provisioned webapp instances and reprovision new ones in order to have observed this particular issue with their HAProxy setup.

It seems like it would take a superhuman level of foresight to catch this scenario for chaos engineering.

Re: A terrible, horrible, no-good, very bad day at Slack

#280

Post by xtracto » Thu, Jul 09, 2020, 8:33 PM UTC

k__ wrote: post #101

xtracto wrote: post #99

Earlier quoted context omitted.

Threaded comments. I tried using Discord with a group of 4 people for real job and we missed threaded comments. Also the ability to draw on screen while screen sharing. So simple yet so useful.

Interesting. I personally didn't like threads and this is the first time someone mentions them favorably.

Funny that I found it easy to see your comment in HackerNews because it was separated in ... a thread!

Imagine how HackerNews, Reddit or any other discussion board would be without threading. Now, if you see a #general Slack room of a 100 person company, you'll quickly see that it would be a mess without threading. That's what happens in Discord.