Slack is down • Hacker News Board

Re: Slack is down

#381

Post by JadoJodo » Mon, Jan 04, 2021, 5:22 PM UTC

blntechie wrote: post #332

I have always had this fantasy thinking of what happens when outages of one of these major service never come back online i.e. in this outage Slack loses info of all the accounts, users, messages etc. How would people react? What would engineers do to recover? I always found that idea fascinating. Imagine Google saying tomorrow that they lost all the accounts and emails. What kind of impact the world will have?

My tangential thought in that regard is what if this is a really bad outage that causes Slack to tank (i.e. A large number of companies switch to Microsoft, Zulip, etc). Equally interesting a thought.

Re: Slack is down

#382

Post by exabrial » Mon, Jan 04, 2021, 5:22 PM UTC

we run a copy of this as a super duper backup: https://github.com/shazow/ssh-chat

Re: Slack is down

#383

Post by cratermoon » Mon, Jan 04, 2021, 5:22 PM UTC

savo92 wrote: post #366

cratermoon wrote: post #213

Earlier quoted context omitted.

I have definitely worked in places where the times right before and right after a change freeze were the most unstable, so that could be it. However, as others have mentioned, it's pretty early on the west coast of the US. Unless some engineer was up extra early (perhaps at the behest of an anxious project manager) it seems unlikely to be a release. What it could be is some engineer somewhere coming in after the holi…

Or unless that engineer was not in the US

Very possible. I don't know what Slack's workforce distribution is. In places I've worked there have definitely been some incidents in US off-hours triggered by someone on the other side of the world.

Re: Slack is down

#384

Post by codingdave » Mon, Jan 04, 2021, 5:22 PM UTC

blntechie wrote: post #332

I have always had this fantasy thinking of what happens when outages of one of these major service never come back online i.e. in this outage Slack loses info of all the accounts, users, messages etc. How would people react? What would engineers do to recover? I always found that idea fascinating. Imagine Google saying tomorrow that they lost all the accounts and emails. What kind of impact the world will have?

That scenario is what Disaster Recovery plans are for. Every large company I've worked for has had recovery plans in place, including scenarios as disturbing as "All data centers and offices explode simultaneously, and all staff who know how it all works are killed in the blasts."

You not only have backups in place, you have documentation in place, including a back-up vendor who has copies of the documentation and can staff up workers to get it up and running again without any help from existing staff.

And we tested those scenarios. I'm not sure which dry runs were less fun - when you got paged at 3 AM to go to the DR site and restore the entire infrastructure from scratch... or when you got paged at 3 AM and were instructed to stay home and not communicate with anyone for 24 hours to prove it can be done with out you. (OK, so staying home was definitely more fun, but disturbing.)

Re: Slack is down

#385

Post by tvorm » Mon, Jan 04, 2021, 5:23 PM UTC

remyp wrote: post #318

ARandomerDude wrote: post #281

> Customers may have trouble connecting or using Slack I can't stand how marketing speak pervades every sphere of the world. Their entire system is offline (inconvenient certainly, but it happens) and they can't bring themselves to say "Slack is down. We're working on it and will be back ASAP." or something similar. Instead we may have trouble.

I'm willing to bet this is influenced more by SLAs and Slack's lawyers than marketing speak.

As someone in marketing, it's a little bit of this, and a little bit of determining what the most default, catch-all statement could be well ahead of time to make "crisis comms" that much smoother.

Re: Slack is down

#386

Post by michaelt » Mon, Jan 04, 2021, 5:23 PM UTC

ajkjk wrote: post #233

ghostpepper wrote: post #224

Earlier quoted context omitted.

and yet they're still proudly proclaiming: "Uptime for the current quarter: 100%"

Every time this kind of thing happens HNers love to grip about how the status pages aren't correct yet. It's so weird -- like the people freaking out about the outage are going to be updating their uptime trackers right now or something. Who cares? It'll be fixed later.

Well, who consults a service's status page when it isn't down? During an outage is literally the only time a service status page has any function.

A status page that doesn't get updated during an outage is about as much use as a solar-powered flashlight (without built in power storage).

Re: Slack is down

#387

Post by lwedel » Mon, Jan 04, 2021, 5:23 PM UTC

as an alternative - Cisco has - webex, less known, but does the work: https://status.webex.com/service/status?lang=en_US

Re: Slack is down

#388

Post by tclancy » Mon, Jan 04, 2021, 5:23 PM UTC

blntechie wrote: post #332

I have always had this fantasy thinking of what happens when outages of one of these major service never come back online i.e. in this outage Slack loses info of all the accounts, users, messages etc. How would people react? What would engineers do to recover? I always found that idea fascinating. Imagine Google saying tomorrow that they lost all the accounts and emails. What kind of impact the world will have?

We might start to see actual legislation around implied SLAs in the US which would cause Google to rethink everyone's 20% project being rolled out for 2 years.

Re: Slack is down

#389

Post by ab_io » Mon, Jan 04, 2021, 5:24 PM UTC

blntechie wrote: post #332

I have always had this fantasy thinking of what happens when outages of one of these major service never come back online i.e. in this outage Slack loses info of all the accounts, users, messages etc. How would people react? What would engineers do to recover? I always found that idea fascinating. Imagine Google saying tomorrow that they lost all the accounts and emails. What kind of impact the world will have?

It would be a mass customer extinction event for said service, and would effectively result in a windfall for competing services

Re: Slack is down

#390

Post by criley2 » Mon, Jan 04, 2021, 5:24 PM UTC

_6fmb wrote: post #343

bww wrote: post #314

Earlier quoted context omitted.

Just a reminder that it's probably not a wise idea for anyone to get further in bed with Zoom than they already are. https://www.washingtonpost.com/technology/2020/12/18/zoom-he...

What an overwrought headline, the employee in question has already been fired.

It's weird that you describe the headline as "overwrought" and call the person an "employee" when the headline is more accurate than you.

This was an executive, not just an employee. That's a huge distinction and I can't help but think you intentionally downgraded his position to cover-up his behavior. "Just an employee" "Not a big deal"

But when you read the allegations, they seem like a very big deal that an executive was spying on users, giving their information to the Chinese government explicitly for oppressive purposes, including folks who are not in China, and went out of his way to personally censor non-Chinese groups meeting to discuss the Massacre-Which-Cannot-Be-Mentioned.

I would say the headline understates the gravity (it's very much a 'by-the-books' headline that you KNOW went through ten levels of Legal), and that your hand waving here feels much more dishonest than the headline.