Slack is down. This is shaping up to be my most productive day in a long while.
Slack is down
301–310 of 840 posts
Re: Slack is down
#302I am genuinely surprised that Slack wasn't ready for people to come back from holiday, to view increased queues of unread messages, to have to manually login vs. having auth tokens or cookies, etc. Either that, or they had a cosmically coincidental outage on a really bad Monday to have it. It's bad enough team comms go over Slack so much now, at least we have email fallback. What scares me is for the teams that use S…
My coworker's theory was someone was waiting for the holiday's end to deploy something risky.
And I'm in that boat of depending on Slack for alerting... in fact my team was also waiting over the holidays to deploy more robust non-Slack-based alerting (in our defense the product is only a few months old and only now starting to scale to any real volume).
Re: Slack is down
#303Just a note, if your company uses G Suite, chat.google.com exists and is basically an entire Slack clone. We use it as a backup when Slack goes down (obviously doesn't help for bots and ChatOps we've set up, but works well for realtime work chat).
Calling it a clone is a stretch. There are a ton of features missing.
Re: Slack is down
#304I am genuinely surprised that Slack wasn't ready for people to come back from holiday, to view increased queues of unread messages, to have to manually login vs. having auth tokens or cookies, etc. Either that, or they had a cosmically coincidental outage on a really bad Monday to have it. It's bad enough team comms go over Slack so much now, at least we have email fallback. What scares me is for the teams that use S…
The two cliched sources of this problem are 1) someone pushed something out over the holidays that could have waited until January, or 2) peak capacity was negatively affected since the last time a spike happened, nobody had a way to monitor it, and so this has been broken since the end of May. On further reflection, someone will admit that they noticed a notch-up in response times and did not connect the dots.
Re: Slack is down
#305Earlier quoted context omitted.
You just don't deploy something major the first day after a 2 weeks vacation, it does not makes any sense.
Why? I had a rewrite of some core logic the last day before Christmas that I didn'td deploy, as it wasn't time critical to get out and I didn't want to be disturbed during holidays. Today it was perfect to deploy, as I can watch it the whole week if needed.
Well, I think it probably depends on where you work. At my work, people just took 2-3 weeks of time off. It takes a moment to get your head back in the game.
Re: Slack is down
#306Just a note, if your company uses G Suite, chat.google.com exists and is basically an entire Slack clone. We use it as a backup when Slack goes down (obviously doesn't help for bots and ChatOps we've set up, but works well for realtime work chat).
Our org just failed over to GChat as well. Piece of cake.
Quite glad we never moved any critical ops work into Slack bots, since we don't control Slack.
Re: Slack is down
#307This is actually impressive, in a bad way. I just have become so used to being able to run highly resilient cross region infrastructure for millions of users with just a handful of people that I forget what real downtime looks like. For their app to just go completely offline is unacceptable. Bugs and degraded services I get. But this is catastrophic.
I can't even begin to guess what went wrong. What are your guesses? How many screaming executives are there at Slack saying "just roll it back"?
Re: Slack is down
#308I am genuinely surprised that Slack wasn't ready for people to come back from holiday, to view increased queues of unread messages, to have to manually login vs. having auth tokens or cookies, etc. Either that, or they had a cosmically coincidental outage on a really bad Monday to have it. It's bad enough team comms go over Slack so much now, at least we have email fallback. What scares me is for the teams that use S…
It might have been the increased usage due to the pandemic (since it didn't happen from 2019 to 2020) + the sudden inflow of people at the same time.
Re: Slack is down
#309Earlier quoted context omitted.
I have tried to sell my organisation on a shared Google Chat doc for 90s style realtime ICQ chat in times like these, but there has been little uptake.
G Suite actually has an entire Slack clone, chat.google.com. I've been on G Suite (now annoyingly renamed to Google Workspace) for years and actually just recently found it existed from another comment on HN.
Yeah, this is what we actually use as a fallback, and I did push for this as an full time alternative given we'd get it free, but people dislike it for all sorts of frivolous reasons.
Re: Slack is down
#310We're using Google Chat again. Feels ancient. https://chat.google.com/
Same - and it's just... weird. The "everything must be in a thread" model feels really clunky.