Slack/Notion. Stop with the features already. You're killing yourself in slow motion. Focus on performance, robustness of your infrastructure.
Slack is down
481–490 of 840 posts
Re: Slack is down
#482Earlier quoted context omitted.
That scenario is what Disaster Recovery plans are for. Every large company I've worked for has had recovery plans in place, including scenarios as disturbing as "All data centers and offices explode simultaneously, and all staff who know how it all works are killed in the blasts." You not only have backups in place, you have documentation in place, including a back-up vendor who has copies of the documentation and ca…
That’s some really good thoughts on DR planning. I have never thought DR to be to such an extent. How many companies really plan for an event where their entire infrastructure goes offline and their entire team gets killed? Does even companies like Google plan for this kind of event?
The company I work for plans for that and it's definitely not FAANG. In fact, DR planning and testing is far more important than stuff like continuous integration, build pipelines, etc.
Re: Slack is down
#483For a product that is so simple, there are no good self-hosted alternatives. Mattermost and RocketChat are written very poorly, reliability and getting your data out is impossible. Slack goes down so often we're thinking of writing a very boring clone that uses ActiveMQ and MySQL, just because chat should be boring and needs to "just work".
I was just considering setting up a Mattermost instance for our company since I used it for a year at a previous job without any issues (I was just a user though, I didn't deploy or maintain it). Just curious, why do you think it's poorly written or unreliable?
We tried running it, so a lot of experience with it, and it wasn't great. It barely stayed online.
For something so simple, you have to run a massive server, like gigs of ram and multiple core, even with a very modest user load. Take a look at the codebase, it's also a mess and impossible to fix any bugs. Finally, if you want to get your data out or report on the message activity, good luck, you'd be better off passing paper notes around. The open source version is nerfed a bit too, no LDAP authentication for instance, so it creates a lot of problems there too.
Re: Slack is down
#484I am genuinely surprised that Slack wasn't ready for people to come back from holiday, to view increased queues of unread messages, to have to manually login vs. having auth tokens or cookies, etc. Either that, or they had a cosmically coincidental outage on a really bad Monday to have it. It's bad enough team comms go over Slack so much now, at least we have email fallback. What scares me is for the teams that use S…
It might have been the increased usage due to the pandemic (since it didn't happen from 2019 to 2020) + the sudden inflow of people at the same time.
This would be a good explanation for an outage in March or April 2020, not so much in January 2021.
Re: Slack is down
#485Just seems slightly disingenuous to me to have "100% uptime" on the same page that says there is a current major outage.
Re: Slack is down
#486This is actually impressive, in a bad way. I just have become so used to being able to run highly resilient cross region infrastructure for millions of users with just a handful of people that I forget what real downtime looks like. For their app to just go completely offline is unacceptable. Bugs and degraded services I get. But this is catastrophic.
I can't even begin to guess what went wrong. What are your guesses? How many screaming executives are there at Slack saying "just roll it back"?
Mass server migration?
Re: Slack is down
#487Earlier quoted context omitted.
Indeed, if you champion FOSS, why would you recommend a proprietary piece of software as fallback?
My point was really that it's GitLab's own product. "GitLab Mattermost" [ https://docs.gitlab.com/omnibus/gitlab-mattermost/ ] I'm amazed they use Slack at all. Let alone as a fallback.
No I'm pretty sure that's just a sort of 'integration', Mattermost shipped with GitLab?
https://about.gitlab.com/blog/2015/08/18/gitlab-loves-matter...
> Like many companies in the last year we've switched to using Slack to improve internal communication. [...] Since Slack doesn't offer an on-premises version, we searched for other options. We found Mattermost to be the leading open source Slack-alternative and suggested a collaboration to the Mattermost team.
I'm not really sure why it's 'GitLab Mattermost' and not (at your link) 'GitLab Nginx' et al. though.
Re: Slack is down
#488I am genuinely surprised that Slack wasn't ready for people to come back from holiday, to view increased queues of unread messages, to have to manually login vs. having auth tokens or cookies, etc. Either that, or they had a cosmically coincidental outage on a really bad Monday to have it. It's bad enough team comms go over Slack so much now, at least we have email fallback. What scares me is for the teams that use S…
Do you know that was the root cause or are you making an assumption and running with it?
Re: Slack is down
#489Re: Slack is down
#490Earlier quoted context omitted.
Just a reminder that it's probably not a wise idea for anyone to get further in bed with Zoom than they already are. https://www.washingtonpost.com/technology/2020/12/18/zoom-he...
What an overwrought headline, the employee in question has already been fired.
Sorry, but an executive is not just "an employee" and any alarms are rightfully justified. Took a little bit of cajoling in my company but we've successfully moved to self-hosted tools for the most part (Jitsi and Rocket.chat) with just a couple of projects with outside contractors using Slack.