Slack is down • Hacker News Board

Re: Slack is down

#361

Post by ibraheemdev » Mon, Jan 04, 2021, 5:16 PM UTC

ibraheemdev wrote: post #20

> Customers may have trouble loading channels or connecting to Slack at this time. Our team is investigating and we will follow up with more information as soon as we have it. We apologize for any disruption caused. - Jan 4, 10:14 AM EST The status for messaging and connection services has been marked as [incident] https://status.slack.com/

All services have now been marked as [outages]:

> We're continuing to investigate connection issues for customers, and have upgraded the incident on our side to reflect an outage in service. All hands are on deck on our end to further investigate. We'll be back in a half hour to keep you posted.

- Jan 4, 11:20 AM EST

> There are no changes to report as of yet. We're still all hands on deck and continuing to dig in on our side. We'll continue to share updates every 30 minutes until the incident has been downgraded

- Jan 4, 11:52 AM EST

Re: Slack is down

#362

Post by TwoPhotons » Mon, Jan 04, 2021, 5:16 PM UTC

blntechie wrote: post #332

I have always had this fantasy thinking of what happens when outages of one of these major service never come back online i.e. in this outage Slack loses info of all the accounts, users, messages etc. How would people react? What would engineers do to recover? I always found that idea fascinating. Imagine Google saying tomorrow that they lost all the accounts and emails. What kind of impact the world will have?

That wouldn't be ideal.

Re: Slack is down

#363

Post by kmichler » Mon, Jan 04, 2021, 5:17 PM UTC

adrianpike wrote: post #310

kmichler wrote: post #292

We're using Google Chat again. Feels ancient. https://chat.google.com/

Same - and it's just... weird. The "everything must be in a thread" model feels really clunky.

That's a good point, however threads do tend to help with keeping things organized when in a channel.

It actually might be a good thing that everyone doesn't feel the need to look at slack every X minutes.

Re: Slack is down

#364

Post by unreal37 » Mon, Jan 04, 2021, 5:17 PM UTC

ARandomerDude wrote: post #281

> Customers may have trouble connecting or using Slack I can't stand how marketing speak pervades every sphere of the world. Their entire system is offline (inconvenient certainly, but it happens) and they can't bring themselves to say "Slack is down. We're working on it and will be back ASAP." or something similar. Instead we may have trouble.

It's not down for me...

Re: Slack is down

#365

Post by vlod » Mon, Jan 04, 2021, 5:18 PM UTC

Scandiravian wrote: post #244

lylo wrote: post #214

This is an excellent reminder of the danger of being locked into closed systems. I wonder how many companies (like mine) have literally ground to a halt because of this? Do other companies have a risk-documented backup plan B for times like this? Presumably the default is for everyone to resort to email? More worryingly is the number of ChatOps processes and alerting/observability systems that are in place around Sla…

I just finished an 5 hour debugging session on what turned out to be several cascading bugs in one of the older systems at my company Can't deploy the fix, because - developers trigger deployments through slack and I don't have access to the underlying deployment system - infrastructure guys who have access aren't responding to my emails

Sounds like a great opportunity to work on not having Slack as the only trigger mechanism. :)

Or a least document how to call the slack bot manually. (assuming it's just a http endpoint)

Re: Slack is down

#366

Post by savo92 » Mon, Jan 04, 2021, 5:18 PM UTC

cratermoon wrote: post #213

exhaze wrote: post #89

When I was at Uber, we noticed that most incidents are directly caused by human actions that modify the state of the system. Therefore, a large "backlog" of human actions that modify the system state have a much higher chance of causing an incident. My bet is that this incident is caused by a big release after a post-holiday "code freeze".

I have definitely worked in places where the times right before and right after a change freeze were the most unstable, so that could be it. However, as others have mentioned, it's pretty early on the west coast of the US. Unless some engineer was up extra early (perhaps at the behest of an anxious project manager) it seems unlikely to be a release. What it could be is some engineer somewhere coming in after the holi…

Or unless that engineer was not in the US

Re: Slack is down

#367

Post by ajkjk » Mon, Jan 04, 2021, 5:19 PM UTC

Merman_Mike wrote: post #279

ajkjk wrote: post #233

Earlier quoted context omitted.

Every time this kind of thing happens HNers love to grip about how the status pages aren't correct yet. It's so weird -- like the people freaking out about the outage are going to be updating their uptime trackers right now or something. Who cares? It'll be fixed later.

I think the point is that a "Status Page" should show the accurate, current status of the system. Not a place holder for "we'll fix it later". People look at a status page to know what's happening now .

I wasn't talking about the status page, I was talking about the uptime % tracker.

edit: oh, sorry, i did say 'status page' in the first part. But I kinda meant update % tracker like the parent.

Re: Slack is down

#368

Post by mey » Mon, Jan 04, 2021, 5:20 PM UTC

bengale wrote: post #288

mey wrote: post #264

Why does the slack client not show connection issues instead of just hard locking up?

The client on my Mac showed a page that said it was having issues connecting with a link to their status page.

The Windows desktop app is less fortunate.

Re: Slack is down

#369

Post by blntechie » Mon, Jan 04, 2021, 5:20 PM UTC

MattGaiser wrote: post #351

blntechie wrote: post #332

I have always had this fantasy thinking of what happens when outages of one of these major service never come back online i.e. in this outage Slack loses info of all the accounts, users, messages etc. How would people react? What would engineers do to recover? I always found that idea fascinating. Imagine Google saying tomorrow that they lost all the accounts and emails. What kind of impact the world will have?

Google would be catastrophic because so much is stored there. Slack is mostly real time communication, at least for me. There are a few bits and bobs that really should be documented that are in the messages though.

Yeah, Google would easily top the list of companies which can have catastrophic impact. Microsoft, Apple, Salesforce, Dropbox would be the next in the list I guess if we leave out the utility companies and internet providers etc.

Re: Slack is down

#370

Post by easton » Mon, Jan 04, 2021, 5:20 PM UTC

pwned1 wrote: post #245

This site seems to be lagging as well. Or is it just me?

It isn’t just you, but due to Hacker News’ right-sized[0] infrastructure, you should sign out unless you need to comment. That way you hit the caches instead of getting the server to make you a new page.

0: https://news.ycombinator.com/item?id=12911461