Slack is down • Hacker News Board

Re: Slack is down

#211

Post by blntechie » Mon, Jan 04, 2021, 4:47 PM UTC

kevindong wrote: post #82

adwww wrote: post #57

Earlier quoted context omitted.

Nice of you to ignore the majority of the world's population who have been up and working long before America woke up.

Slack makes ~61% of its revenue from US customers which only has 4 time zones compared to the remainder of their revenue being spread out across ~20 time zones. It's not an unreasonable hypothesis. See page 12 of the document (which is page 14 of the PDF) https://d18rn0p25nwr6d.cloudfront.net/CIK-0001764925/70df834...

Slack most likely has more US customers but

- Revenue is not same as users. Slack have tons of free users and some countries also has lower priced plans.

- Many companies like Amazon etc. probably is counted as US revenue for Slack but they have more than 30% of their employees outside the US. This should not be huge numbers but significant.

Re: Slack is down

#212

Post by hacky_engineer » Mon, Jan 04, 2021, 4:48 PM UTC

matsemann wrote: post #192

Thaxll wrote: post #156

Earlier quoted context omitted.

You just don't deploy something major the first day after a 2 weeks vacation, it does not makes any sense.

Why? I had a rewrite of some core logic the last day before Christmas that I didn'td deploy, as it wasn't time critical to get out and I didn't want to be disturbed during holidays. Today it was perfect to deploy, as I can watch it the whole week if needed.

Yeah, I do this all the time. I don't want to be bothered on the weekend, so I push releases at the beginning of the week when possible.

Re: Slack is down

#213

Post by cratermoon » Mon, Jan 04, 2021, 4:48 PM UTC

exhaze wrote: post #89

When I was at Uber, we noticed that most incidents are directly caused by human actions that modify the state of the system. Therefore, a large "backlog" of human actions that modify the system state have a much higher chance of causing an incident. My bet is that this incident is caused by a big release after a post-holiday "code freeze".

I have definitely worked in places where the times right before and right after a change freeze were the most unstable, so that could be it. However, as others have mentioned, it's pretty early on the west coast of the US. Unless some engineer was up extra early (perhaps at the behest of an anxious project manager) it seems unlikely to be a release.

What it could be is some engineer somewhere coming in after the holiday, noticing a slightly flaky thing, and thinking, "I'll reboot/redeploy/refresh this thing so the flakiness doesn't get worse". Only it turns out the flaky thing was a signal of something else falling over. Or maybe the redeploy was the wrong version because of bad CI/CD, or maybe the person just fat-fingered it.

Re: Slack is down

#214

Post by lylo » Mon, Jan 04, 2021, 4:48 PM UTC

This is an excellent reminder of the danger of being locked into closed systems.

I wonder how many companies (like mine) have literally ground to a halt because of this? Do other companies have a risk-documented backup plan B for times like this? Presumably the default is for everyone to resort to email?

More worryingly is the number of ChatOps processes and alerting/observability systems that are in place around Slack.

Not being able to chat with co-workers for an hour or two is fine, but not being able to safely manage CI/CD/deployments is a big risk.

Re: Slack is down

#215

Post by FerdSlav » Mon, Jan 04, 2021, 4:50 PM UTC

rexreed wrote: post #200

Any opinions on using Discord vs. Slack?

I personally have found that one of Discords major shortcomings is the lack of support for threaded message chains. For those times when you may have 2+ parallel conversations in a channel you end up dramatically reducing the ability to effectively communicate.

Re: Slack is down

#216

Post by radlad » Mon, Jan 04, 2021, 4:50 PM UTC

brundolf wrote: post #140

exhaze wrote: post #126

Earlier quoted context omitted.

To elaborate a bit more on this point, you have to think about it like any complex system failure - it's almost never one thing, but rather a combination of many different factors. The factors around post NYE releases: - high risk changes that weren't released pre-holidays get released. Depending on the company, this could mean a 1-week to 1-month delay between implementation and release. The greater that interval, t…

Sudden surge of traffic as all their users returns to work?

Yes - I wondered if they took some servers down prior to the break as a cost saving measure, and forgot to reinstate them.

Re: Slack is down

#217

Post by heroHACK17 » Mon, Jan 04, 2021, 4:50 PM UTC

Just wanted to come here and say, hey! How is everyone doing? How was your Holiday break?

Re: Slack is down

#218

Post by thismodernlife » Mon, Jan 04, 2021, 4:50 PM UTC

I wondered about getting credits for the outage but you can't view the SLA page because the app is down.

https://slack.com/intl/en-gb/terms/service-level-agreement

Re: Slack is down

#219

Post by spelunker » Mon, Jan 04, 2021, 4:50 PM UTC

nicioan wrote: post #195

Funny how status.slack.com has reported Incidents and Outages for a while now, but still the "Uptime for the current quarter" is reported at 100% on the bottom right of the status table.

I would think that number will be updated once the fire is put out.

Re: Slack is down

#220

Post by nikolay » Mon, Jan 04, 2021, 4:50 PM UTC

Can't handle the post-holiday surge, or people wanted to justify their long holiday and pushed something only to witness their holiday optimism head-crashing on the surface of the reality?