801–810 of 840 posts
Earlier quoted context omitted.
In what companies?
Competent ones like those you'd hear about being down on HN. At least that how it worked at one FAANG
I have always had this fantasy thinking of what happens when outages of one of these major service never come back online i.e. in this outage Slack loses info of all the accounts, users, messages etc. How would people react? What would engineers do to recover? I always found that idea fascinating. Imagine Google saying tomorrow that they lost all the accounts and emails. What kind of impact the world will have?
That scenario is what Disaster Recovery plans are for. Every large company I've worked for has had recovery plans in place, including scenarios as disturbing as "All data centers and offices explode simultaneously, and all staff who know how it all works are killed in the blasts." You not only have backups in place, you have documentation in place, including a back-up vendor who has copies of the documentation and ca…
I sat in on a DR test where the moment one of the Auckland based ops team tried asking the Wellington lead, the boss stepped in and said "Wellington has been levelled by an earthquake. Everyone is dead or trying to get back to their family. They will not be helping you during the exercise."
Earlier quoted context omitted.
That scenario is what Disaster Recovery plans are for. Every large company I've worked for has had recovery plans in place, including scenarios as disturbing as "All data centers and offices explode simultaneously, and all staff who know how it all works are killed in the blasts." You not only have backups in place, you have documentation in place, including a back-up vendor who has copies of the documentation and ca…
That’s some really good thoughts on DR planning. I have never thought DR to be to such an extent. How many companies really plan for an event where their entire infrastructure goes offline and their entire team gets killed? Does even companies like Google plan for this kind of event?
It's pretty much part of the basic day-to-day life in some industries.
Earlier quoted context omitted.
I suggested RocketChat when the outage was announced and HN community downvoted it quite heavily. I'm not sure why. [0] We ended making the switch and committed to Discord. We're now looking at Rocket.chat as a backup in case Discord goes down. But Slack is now completely out of the picture for our team. [0] https://news.ycombinator.com/item?id=25633047
Just curious - why not use Matternost as a backup? (disclosure: I work at Mattermost, but really just want to know what you think) I’ve advocated for an idea where Mattermost is to be used as a “bunker” where it is hosted on a raspberry Pi (or somewhere else) and acts as a digital bunker if your critical infrastructure (slack, teams, exchange?) is compromised somehow.
I know it's a tough spot, but if it were usable from GitLab with zero config that would be great for fallback.
Earlier quoted context omitted.
Yes - I wondered if they took some servers down prior to the break as a cost saving measure, and forgot to reinstate them.
Doubtful. It's not impossible a company the size of Slack would be reliant on a specific engineer logging on in the morning before a traffic spike so the service can handle the spike in load, but that's a misuse of modern distributed cloud-based computing. Hate on the cloud all you want, but AWS has (several flavors of) load balancers and various ways to automatically scale up and down resources (and if you're conser…
In 2021, how does one keep track of resource starvation at the process, container, os, service, pod, cluster, availability zone and region levels?
Earlier quoted context omitted.
I'm not sure about that. I feel like I get more upvotes from sarcasm and jokes than from insight. In this instance, I think it's because when people hear something dumb said seriously in real life, they're not going to readily recognize online that it's a joke.
Yeah, Poe’s law applies here. That’s definitely something someone less informed might say in earnest.
For a product that is so simple, there are no good self-hosted alternatives. Mattermost and RocketChat are written very poorly, reliability and getting your data out is impossible. Slack goes down so often we're thinking of writing a very boring clone that uses ActiveMQ and MySQL, just because chat should be boring and needs to "just work".
Earlier quoted context omitted.
> your email program Reality check: Most people don't use email programs anymore. Also how do you get IRC to sync all conversation data, history, between your several desktops and phones, how do you send files, make calls, and thread conversations?
But clearly no one is saying that email is too hard to use and we should just use $something_else (or are they?). And you are starting to move goalposts here.. first it was uptime, then it was operations and now it's features... And what about those web based IRC solutions? They are even easier to use than slack, have combined history, file sharing, etc.
Earlier quoted context omitted.
This thread is about a Slack outage, which you have no control over. Mattermost and similar software is self-hosted, which of course doesn't mean you're getting 100% uptime, but you have (more) control over it.
In practice self hosted usually translates to more downtime and slower performance when it works. Unless your org has more expertise running a chat service than Microsoft or slack, your self hosted alternative is always going to suck more.
From my experience, when I self-host stuff it's a lot faster (more server resources) and never had any downtime (server doesn't simply go down for no reason).
I'd like to take this moment to mention self-hosted, open source, and federated alternatives like XMPP and Matrix. I'd like to, but unfortunately I don't feel like I can in good faith. Matrix is woefully immature, and suffers from a lot of issues, but I think is closer to being a functional Slack/Discord alternative. XMPP is much more mature, and works very well for chat, but doesn't have a nice package that does all…
RocketChat works pretty well for simple team comms. I have no idea if it can do XMPP and/or Matrix.
* You get notifications for channels for which you have suppressed those notifications
* Some channels are marked as having new notifications, when they haven't
* Notifications for new messages in threads you are involved in are quite hard to find (horrible UX)
* Some UX choices are very confusing (you get a column of options related to notifications, and for some, the left option is the one leading to more notifications, for some the right option)
* There are some overlapping features that lead to inconsistent usage (channels vs. discussions vs. threads)
* Threads are hard to read, because follow-ups in threads are shown in a smaller font size. You cannot increase the font size at all in the desktop application
.... and so on.
Also, I tried to submit some bugs, but for that I'd need to have some information which only the admins have that run this instance, and in the end it was too much effort to get all that information together, so I didn't even bother.