Slack outage: Connectivity issues affecting all workspaces • Hacker News Board

Re: Slack outage: Connectivity issues affecting all workspaces

#151

Post by londons_explore » Wed, Jun 27, 2018, 3:37 PM UTC

nojvek wrote: post #121

anonu wrote: post #52

Previous outages: https://news.ycombinator.com/item?id=16108912 - 5 months ago (longer discussion) https://news.ycombinator.com/item?id=15597461 - 7 months ago https://news.ycombinator.com/item?id=15597431 - 8 months ago https://news.ycombinator.com/item?id=13811815 - 1 year ago https://news.ycombinator.com/item?id=10616743 - 3 years ago

In light of how Slack and other companies haven't been able to get a decent level of uptime, I have to say, the company known to make huge web applications that don't go down in shame every couple of months is probably Google. I can't remember the last time Gmail was down. It just works! If google is down, probably your internet is down. Their expertise and discipline in distributed applications is unrivaled. I'm gue…

Google is expert at designing services which you won't notice when there is downtime.

Take Google Search for example. When there is downtime, results might be slightly less accurate, or the parcel tracking box might not appear, or the page won't say the "last visited" time beside search results.

The SRE's are running around fixing whatever subsystem is down or broken, but you the user probably don't notice.

Re: Slack outage: Connectivity issues affecting all workspaces

#152

Post by Jgrubb » Wed, Jun 27, 2018, 3:38 PM UTC

raphaelj wrote: post #98

Does anyone know about any Slack alternatives that support bots, GIFYs and emojis ? And above all that is not ridiculously slow.

We just spun up Mattermost, seems ok for the first hour.

Re: Slack outage: Connectivity issues affecting all workspaces

#153

Post by dannyrosen » Wed, Jun 27, 2018, 3:38 PM UTC

Hope Slack considers doing a post-mortem similar to Gitlab[1]. Sharing what they learned and giving customers context is appreciated.

[1]: https://about.gitlab.com/2017/02/10/postmortem-of-database-o...

Re: Slack outage: Connectivity issues affecting all workspaces

#154

Post by mikec3010 » Wed, Jun 27, 2018, 3:40 PM UTC

FooBarWidget wrote: post #90

sarreph wrote: post #33

You know how much of the community uses one messaging system when 15 minutes after it going down, it has over 40 points on the front page! This says a lot about how it's a single point of failure in modern company comms. It's even worrying to think about how some users probably have production-dependent (dare I postulate it) workflows in Slack that get crippled by its outage... ITT: Chat about decentralisation that w…

Yes it's a single point of failure, but so what? I don't particularly care whether other organizations fail at the same time as I do, I just care whether I fail. Hosting my own chat system does not solve that problem. In fact, it may make it worse because then I have to worry about system administration, and Slack probably has more expertise on that. It's likely that they can fix this problem for all customers faster…

I think it's inexcusable for a chat program to go down in 2018.

* your hdd failed? Use a raid

* your power went out? Use a UPS

* your DNS went down? Use a fallback (slack2)

* your whole datacenter flooded? Good thing you have multiple replicated cloud instances that seamlessly take over

See, these are the issues that "the cloud" was supposed to solve. Not give us the same problems as before, just with a recurring bill for "chat as a service".

And inb4 "chill Mike it's just a chat server not life support firmware" yeah but slack is the most trivial software you can think of: send text from one computer to another. I see no reason this service can't be nearly as reliable as life support firmware in 2018. We've had over 30 years to get this right. Raise the fricking bar.

Re: Slack outage: Connectivity issues affecting all workspaces

#155

Post by thomastjeffery » Wed, Jun 27, 2018, 3:40 PM UTC

sarreph wrote: post #33

You know how much of the community uses one messaging system when 15 minutes after it going down, it has over 40 points on the front page! This says a lot about how it's a single point of failure in modern company comms. It's even worrying to think about how some users probably have production-dependent (dare I postulate it) workflows in Slack that get crippled by its outage... ITT: Chat about decentralisation that w…

My company has a customer support system that relies on Slack chatops.

It's an interesting morning, to say the least.

Re: Slack outage: Connectivity issues affecting all workspaces

#156

Post by drb91 » Wed, Jun 27, 2018, 3:40 PM UTC

FooBarWidget wrote: post #90

sarreph wrote: post #33

You know how much of the community uses one messaging system when 15 minutes after it going down, it has over 40 points on the front page! This says a lot about how it's a single point of failure in modern company comms. It's even worrying to think about how some users probably have production-dependent (dare I postulate it) workflows in Slack that get crippled by its outage... ITT: Chat about decentralisation that w…

Yes it's a single point of failure, but so what? I don't particularly care whether other organizations fail at the same time as I do, I just care whether I fail. Hosting my own chat system does not solve that problem. In fact, it may make it worse because then I have to worry about system administration, and Slack probably has more expertise on that. It's likely that they can fix this problem for all customers faster…

> It's likely that they can fix this problem for all customers faster than I can fix my problem for myself. And it's not like I'm crippled when Slack is down.

Well, you can probably infer the former from the dependency on the latter. You use these tools because they can reduce the scrambling when shit does hit the fan, not because they are necessary.

Re: Slack outage: Connectivity issues affecting all workspaces

#157

Post by pi-squared » Wed, Jun 27, 2018, 3:41 PM UTC

Telegram was having an outage at about the same time, probably unrelated but still: http://downdetector.com/status/telegram

Re: Slack outage: Connectivity issues affecting all workspaces

#158

Post by sulami » Wed, Jun 27, 2018, 3:42 PM UTC

My team jumped onto IRC, which is working pretty well. No custom emoji, but that's about it.

Re: Slack outage: Connectivity issues affecting all workspaces

#159

Post by tapoxi » Wed, Jun 27, 2018, 3:43 PM UTC

sanderjd wrote: post #128

tapoxi wrote: post #3

It's times like this I wish there was a solid decentralized standard to pick from, but there's no clear choice between XMPP and Matrix.

It's not about the protocols, it's about having a client with a user experience that is acceptable to an entire company rather than just a team of engineers. Which decentralized protocol has such a client? (Speaking as someone who got burned trying to advocate for IRC at a company that eventually and inevitably switched to Slack.)

There's plenty of decent XMPP clients, like Spark (https://igniterealtime.org/projects/spark/) but they'd take an IT team to configure.

Matrix has Riot (https://riot.im/app) but personally I find it incredibly confusing.

Re: Slack outage: Connectivity issues affecting all workspaces

#160

Post by kirbypineapple » Wed, Jun 27, 2018, 3:46 PM UTC

tapoxi wrote: post #3

It's times like this I wish there was a solid decentralized standard to pick from, but there's no clear choice between XMPP and Matrix.

We use Slack for everyday company wide communication/ announcements and Riot for encrypted secure communications (you can host Riot yourself): https://about.riot.im/