Slack is down • Hacker News Board

Re: Slack is down

#451

Post by aphextron » Mon, Jan 04, 2021, 5:37 PM UTC

MuffinFlavored wrote: post #307

ryanSrich wrote: post #266

This is actually impressive, in a bad way. I just have become so used to being able to run highly resilient cross region infrastructure for millions of users with just a handful of people that I forget what real downtime looks like. For their app to just go completely offline is unacceptable. Bugs and degraded services I get. But this is catastrophic.

I can't even begin to guess what went wrong. What are your guesses? How many screaming executives are there at Slack saying "just roll it back"?

>I can't even begin to guess what went wrong. What are your guesses? How many screaming executives are there at Slack saying "just roll it back"?

Doubtful it's a code issue causing a total system outage. I'm assuming they have a bunch of auto scaling infrastructure that wound down over the holidays and couldn't take the spike this morning.

Re: Slack is down

#452

Post by jolmg » Mon, Jan 04, 2021, 5:37 PM UTC

minitoar wrote: post #433

orthecreedence wrote: post #414

"It will replace email."

I mean...wasn't gmail (which effectively IS email for many, many people) down recently?

> which effectively IS email for many, many people

Doesn't have to be, though. One person doesn't even have to tie their address to a single provider, and seeing past received messages doesn't even need internet connectivity.

Re: Slack is down

#453

Post by harikb » Mon, Jan 04, 2021, 5:37 PM UTC

Can we please trade in centralized Slack and single-sign-on and get back the netsplit of IRC :) ? At least I can chat with half of my colleagues :)

Re: Slack is down

#454

Post by kuon » Mon, Jan 04, 2021, 5:37 PM UTC

This is like the first time in a year I need slack, call that bad luck.

I hope matrix/element will rise more.

Re: Slack is down

#455

Post by NyxWulf » Mon, Jan 04, 2021, 5:38 PM UTC

cozzyd wrote: post #415

ARandomerDude wrote: post #281

> Customers may have trouble connecting or using Slack I can't stand how marketing speak pervades every sphere of the world. Their entire system is offline (inconvenient certainly, but it happens) and they can't bring themselves to say "Slack is down. We're working on it and will be back ASAP." or something similar. Instead we may have trouble.

I did manage to receive a message a few minutes ago, so it might be just mostly dead.

If it's all dead, there is only one thing you can do.

Re: Slack is down

#456

Post by jennyyang » Mon, Jan 04, 2021, 5:39 PM UTC

Could this be some sort of data corruption? I find it hard to believe that Slack could be down for this long without something that is exceedingly hard to rollback. Even if some services are completely overwhelmed with traffic, they could block a certain percentage of traffic to decrease load, and then force servers up across their datacenters and then unblock traffic. It has the hallmarks to me of some sort of datastore is down, but obviously just a random guess.

Re: Slack is down

#457

Post by sergiotapia » Mon, Jan 04, 2021, 5:39 PM UTC

sytse wrote: post #258

At GitLab our fallback from Slack is Zoom https://about.gitlab.com/handbook/communication/#emergency-c... I'm posting this because I found a lot of people don't know that Zoom includes a complete chat client that includes channels. And #HugOps to the engineers at Slack working on this. I appreciate that they posted a periodic update even when there was no news to report: "There are no changes to report as of yet. We'…

At Papa we use Discord as backup.

Re: Slack is down

#458

Post by jaywalk » Mon, Jan 04, 2021, 5:39 PM UTC

mey wrote: post #264

Why does the slack client not show connection issues instead of just hard locking up?

When it went down fully and I had the Windows client open, it went to a page that basically said "Slack is down, we don't know why, try restarting and see if that fixes it. Here's the status page."

It would be nice if they could fix it so that a fresh start also goes to that page, at the very least.

Re: Slack is down

#459

Post by welterde » Mon, Jan 04, 2021, 5:40 PM UTC

JMTQp8lwXL wrote: post #289

dkdk8283 wrote: post #248

Earlier quoted context omitted.

Lack of logs and history is a feature not a bug.

For business users, there are regulatory requirements. You need to keep information around for some period of time, but not forever. History and searching is useful for spreading tribal knowledge throughout an organization.

Does that actually extend to Slack/slack-like things though?

Since I would see Slack more of a replacement for phone calls or hallway discussions. Neither of which typically has any logs or recordings (and I wouldn't want to work somewhere that did keep such logs).

Re: Slack is down

#460

Post by ourcat » Mon, Jan 04, 2021, 5:40 PM UTC

ekianjo wrote: post #440

ourcat wrote: post #425

Earlier quoted context omitted.

I'm quite surprised that you don't use Mattermost as a 'Slack fallback' at GitLab.

Indeed, if you champion FOSS, why would you recommend a proprietary piece of software as fallback?

My point was really that it's GitLab's own product. "GitLab Mattermost" [https://docs.gitlab.com/omnibus/gitlab-mattermost/]

I'm amazed they use Slack at all. Let alone as a fallback.