Earlier quoted context omitted.
I'm quite surprised that you don't use Mattermost as a 'Slack fallback' at GitLab.
Indeed, if you champion FOSS, why would you recommend a proprietary piece of software as fallback?
[deleted]
Slack is down
601–610 of 840 posts
Re: Slack is down
#602Earlier quoted context omitted.
> I'll take this moment to remind everyone of their human tendency to read meaning into random events. There's no evidence to suggest New Year traffic has caused this, and outages like this can happen in spite of professional and competent preparation. On the one hand, sure we don't specifically know what's going on. On the other hand, it's the first Monday in the new year and they went down shortly after the start o…
It seemingly worked ok in UTC-2 in the morning and early afternoon, then started having issues and is now a bit intermittent (or fixed, there's not much traffic on my channels, as it's evening already). Do they have that much more traffic on US east coast than in Europe?
Probably, but it was only 2-3pm UK time when it started falling over so there would be all the Europe traffic plus the East Coast traffic starting to sign in.
Re: Slack is down
#603These events seem to be happening almost on a monthly basis now. IRC was never this unreliable and at least with netsplits it was obvious what had happened because you'd see the clients disconnect. IME messages just fail to send with Slack, then you can retry but they're not properly idempotent and you end up sending the messages twice. It's really poor.
Its especially strange when you think about how unoriginal Slack's product domain is, and how comparable, and in some cases small, their userbase is. * iMessage, which likely handles something in the range of 750M-1B monthly actives. * WhatsApp, 2B users [1], though no clarity on "active" users. * Telegram, 400M monthly actives [2] * Discord, 100M monthly actives [3] * Slack, 12M daily actives [4] * Teams, which is c…
It isn't just # of users, though - SlackOps is probably unique to Slack in that list (minus Teams, I guess) - so # of messages per month is a better metric. Not that I'm letting Slack off the hook, it still may be that their codebase and/or dev process is just nasty.
Re: Slack is down
#604Earlier quoted context omitted.
That scenario is what Disaster Recovery plans are for. Every large company I've worked for has had recovery plans in place, including scenarios as disturbing as "All data centers and offices explode simultaneously, and all staff who know how it all works are killed in the blasts." You not only have backups in place, you have documentation in place, including a back-up vendor who has copies of the documentation and ca…
This scenario isn't as far fetched as people think. I was running a global deployment in 2012 when hurricane Sandy hit the east cost. The entire eastern seaboard went offline and was off for several days. Some data centers were down for weeks. Our plan had covered that contingency and we failed all of our US traffic to the two west coast regions of amazon. Our downtime on the east cost was around two minutes. Yet a s…
I worked for a regional company in the oil and gas industry and the HQ and both datacenters were in the same earthquake zone. A twice per century earthquake had a real risk of taking down both DCs and the HQ. The plan would have been for every gas station in the vertical to switch to a contingency plan distributing critical emergency supplies and selling non-essential supplies using off-grid procedures.
Re: Slack is down
#605Slack has been a uniquely iffy service. I wonder if there's a solid decentralized alternative.
I've had good experiences with ngircd. It's an IRC server that is very easy to self-host, and it can be installed via APT on any debian/ubuntu/raspbian etc system, and I'm sure on many others.
Re: Slack is down
#606Earlier quoted context omitted.
For those that don't know what this comment is about: https://eng.uber.com/multitenancy-microservice-architecture/
I'm actually more confused after reading that. I assumed that you meant that tested in production on purpose, but it sounds , at a skim, like they do non-prod testing environments - in fact, it looks like they've gone to having multiple beta environments of every service?
My understanding is that they have a "tenancy" variable in every service call which can take a different code path. They seem to only have one environment for everything and do tests/experiments at code level based on this variable.
Re: Slack is down
#607> Customers may have trouble connecting or using Slack I can't stand how marketing speak pervades every sphere of the world. Their entire system is offline (inconvenient certainly, but it happens) and they can't bring themselves to say "Slack is down. We're working on it and will be back ASAP." or something similar. Instead we may have trouble.
It never went down for me.
Re: Slack is down
#608Earlier quoted context omitted.
I wouldn't be surprised if it's actually a combination of a new feature being recently rolled out, along with the sudden spike in load this morning. The holidays are actually the perfect time for Slack to roll out a risky deployment, as it has to be their lowest usage time. So it would make sense if something was pushed out last week or the week before. And everything probably seemed fine. And then this morning they…
Holidays are a good time for a company to do a risky deployment, but a bad time for an individual employee to do a risky deployment, assuming one doesn't want to work overtime over the holiday fixing things.
Depends on how well compensated holiday overtime is. There are some employees happy to work overtime if their hourly pay is doubled or tripled. However there also those who wouldnt do that for any price.
Re: Slack is down
#609Earlier quoted context omitted.
The good thing about that is that if you want a fast client there are quite a few native clients to pick. https://matrix.org/clients/ For example, Mirage is Python/QT and quite fast in my experience. There are Rust clients, C++ clients, terminal based ones, etc.
Do you happen to know of any desktop clients that support encryption/cross-signing? I'd like to get off of Element desktop/web for a couple of reasons, but I need those features. I'd help implement them myself, but that's beyond my skill level. Edit: For anyone else wondering, matrix-commander [0] looks like it may be workable if a cli tool is acceptable for your usecase. [0] https://matrix.org/docs/projects/client/m…
Fluffychat [1] is built with Flutter and apparently supports e2e encryption.
Note: I wanted to try it out for a while, but haven't yet.
Re: Slack is down
#610At GitLab our fallback from Slack is Zoom https://about.gitlab.com/handbook/communication/#emergency-c... I'm posting this because I found a lot of people don't know that Zoom includes a complete chat client that includes channels. And #HugOps to the engineers at Slack working on this. I appreciate that they posted a periodic update even when there was no news to report: "There are no changes to report as of yet. We'…
As someone who has to use Zoom Chat to interact with a client on a daily basis, please, do not recommend Zoom Chat to anyone except as an example of how not to do chat software.
--
Though, I do agree wholeheartedly with your sentiment that the Slack team needs all the positive vibes they can get right now.