Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack is down

status.slack.com

Post reply on HN
Print view View original

661–670 of 840 posts

Re: Slack is down

#661

Post by dheera »

thesuitonym wrote: post #634

I'd like to take this moment to mention self-hosted, open source, and federated alternatives like XMPP and Matrix. I'd like to, but unfortunately I don't feel like I can in good faith. Matrix is woefully immature, and suffers from a lot of issues, but I think is closer to being a functional Slack/Discord alternative. XMPP is much more mature, and works very well for chat, but doesn't have a nice package that does all…

> self-hosted

How often is Slack/Discord down? I mean it's not perfect, but I really honestly don't think I could match their uptime by self-hosting, as well as more on-call rotations for something that's not core product.

I very much prefer that for something that isn't core product, if it goes down I need to do exactly nothing for it to come back up, and that the engineers at Slack will be starting to work on it likely before I even realize it's down.

Re: Slack is down

#663

Post by spicyramen »

thesuitonym wrote: post #634

I'd like to take this moment to mention self-hosted, open source, and federated alternatives like XMPP and Matrix. I'd like to, but unfortunately I don't feel like I can in good faith. Matrix is woefully immature, and suffers from a lot of issues, but I think is closer to being a functional Slack/Discord alternative. XMPP is much more mature, and works very well for chat, but doesn't have a nice package that does all…

Facebook and other vendors killed XMPP, we lived in a non federated world in Enterprise and consumer. No interest of companies to change this

Re: Slack is down

#664

Post by likpok »

kevinmchugh wrote: post #145
exhaze wrote: post #126

Earlier quoted context omitted.

To elaborate a bit more on this point, you have to think about it like any complex system failure - it's almost never one thing, but rather a combination of many different factors. The factors around post NYE releases: - high risk changes that weren't released pre-holidays get released. Depending on the company, this could mean a 1-week to 1-month delay between implementation and release. The greater that interval, t…

If new hires tends to break production, it's not in the first business day of the calendar year. December gets really quiet for recruiting, typically, as candidates get busy with their social lives, and scheduling interviews gets harder. January is busy for recruiting, but given a week or two of interviewing and negotiating, two weeks notice, it's probably February before new employees are starting, and they're not m…

You will also get a pause in new hires in late December for the same reason. I've certainly accepted an offer late in the year and then didn't start until the new year.

Probably not as big of a rush as the end of school year rush in summer though.

I also doubt that new people will be breaking production on day one. Even at a fast moving startup I'd expect it to take a bit to go through the onboarding paperwork, get a laptop and actually try pushing a change to production.

Re: Slack is down

#665

Post by MattJ100 »

thesuitonym wrote: post #634

I'd like to take this moment to mention self-hosted, open source, and federated alternatives like XMPP and Matrix. I'd like to, but unfortunately I don't feel like I can in good faith. Matrix is woefully immature, and suffers from a lot of issues, but I think is closer to being a functional Slack/Discord alternative. XMPP is much more mature, and works very well for chat, but doesn't have a nice package that does all…

The problem is that XMPP and Matrix are protocols, not products.

Element (the primary Matrix software) definitely has Slack and Discord in its sights.

I don't think there are any serious "self-hosted Slack-like" contenders that are XMPP-based right now. You can piece components together (yay, standards!) and I did exactly this for the IETF's XMPP deployment recently. But it's far from being a cohesive easy-to-deploy product. Simply because nobody is building that right now. It takes time and resources and there's no money in it.[1]

People who do set out to build Slack clones (projects like Mattermost and Rocket Chat) and earn money don't have features such as federation on their priority list and don't build on top of Matrix/XMPP. They roll their own custom protocols and as far as I can see they are fairly content with that decision.

[1] There's even less money it, but nevertheless I am currently working on such a self-hostable "package" for XMPP. However rather than focusing on the team chat use-case (Slack/etc.) I'm focusing on personal messaging (WhatsApp/etc.): https://snikket.org/ if you're interested. It's possible I will broaden the scope one day.

EDIT: typo

Re: Slack is down

#666

Post by [deleted] »

cgh wrote: post #632
sytse wrote: post #258

At GitLab our fallback from Slack is Zoom https://about.gitlab.com/handbook/communication/#emergency-c... I'm posting this because I found a lot of people don't know that Zoom includes a complete chat client that includes channels. And #HugOps to the engineers at Slack working on this. I appreciate that they posted a periodic update even when there was no news to report: "There are no changes to report as of yet. We'…

Unpopular opinion, but WebEx beats the pants off Zoom. Of course, it's neither free nor open. But it does support strong end to end encryption and authentication and has regulatory compliance to a bunch of things, if that's important to you. I get that there is WebEx hate because "enterprise" etc, but we use it around here and it works quite well.

[deleted]

Re: Slack is down

#667

Post by fragmede »

radlad wrote: post #216
brundolf wrote: post #140

Earlier quoted context omitted.

Sudden surge of traffic as all their users returns to work?

Yes - I wondered if they took some servers down prior to the break as a cost saving measure, and forgot to reinstate them.

Doubtful. It's not impossible a company the size of Slack would be reliant on a specific engineer logging on in the morning before a traffic spike so the service can handle the spike in load, but that's a misuse of modern distributed cloud-based computing.

Hate on the cloud all you want, but AWS has (several flavors of) load balancers and various ways to automatically scale up and down resources (and if you're conservative, you can disable the 'down' part). If you're operating a major SaaS company like Slack and not taking advantage of them, something's gone wrong.

Re: Slack is down

#668

Post by jhgg »

013a wrote: post #403
gruez wrote: post #354

Earlier quoted context omitted.

>I didn't realize that Discord has way more active users than Slack Keep in mind you're comparing daily active users vs monthly active users. I'd guess most slack users are online weekday for pretty much the entire day (because it's for work and your boss expects you to be online), whereas a good chunk of discord users are only logging in a few hours a week when they're gaming.

At 12:00pm EDT on a workday: Minecraft official server: 190k online users. | Fortnite official server: 180k online users. | Valorant official server: 170k online users. | Jet's Dream World (community): 130k online users. | CallMeCarson server (YouTuber): 100k online users. | Call of Duty official server: 90k online users. | Rust (the game) official discord: 80k online users. | League of Legends official server: 60k o…

Our secret sauce is Elixir/BEAM and Rust :)

Well for the real time side, I can't tell you how big a boon it's been to build our platform on top of Elixir/BEAM. Hands down the best runtime / VM for the job - and a big big secret to our success. Where we couldn't get BEAM fast enough - we lean on rust and embed it into the VM via NIFs.

2021 is the year of rust - with the async ecosystem continuing to mature (tokio 1.0 release) we will be investing heavily in moving a lot of our workloads from Python to Rust - and using Rust in more places, for example, as backend data services that sit in front of our DBs. We have already piloted this last year for our messages data store and have implemented such things as concurrency throttles and query coalescing to keep the upstream data layer stable. It has helped tremendously but we still have a lot of work to do!

To help scale those super large servers, in 2020 we invested heavily in making sure our distributed system can handle the load.

Did you know that all those mega servers you listed run within our distribution on the same hardware and clusters as every other discord server - with no special tenancy within our distribution. The largest servers are scheduled amongst the smallest servers and don't get any special treatment. As a server grows - it of course is able to consume a larger share of resources within our distribution - and automatically transitions to a mode built for large servers (we call this "relays" internally.) At any hour, over a hundred million BEAM processes are concurrently scheduled within our distributed system. Each with specific jobs within their respective clusters. A process may run your presence, websocket connection, session on discord, voice chat server, go live stream, your 1:1/group DM call, etc. We schedule/reschedule/terminate processes at a rate of a few hundred thousand per minute. We are able to scale by adding more nodes to each cluster - and processes are live migrated to the new nodes. This is an operation we perform regularly - and actually is how we deploy updates to our real time system.

I was responsible for building and architecting much of these systems. It's been super cool to work on - and - it's cool to see people acknowledge the scale we now run at! Thank you!! It's been a wild ride haha.

As for scale, our last public number perhaps comparable to Slack is ~650 billion messages sent in 2020, and a few trillion minutes of voice/video chat activity. However given the crazy growth that has happened last year due to COVID - the daily message send volumes are well over the 2 billion/day average.

Re: Slack is down

#669

Post by signed0 »

blntechie wrote: post #332

I have always had this fantasy thinking of what happens when outages of one of these major service never come back online i.e. in this outage Slack loses info of all the accounts, users, messages etc. How would people react? What would engineers do to recover? I always found that idea fascinating. Imagine Google saying tomorrow that they lost all the accounts and emails. What kind of impact the world will have?

In 2011 a small amount (0.02%) of Gmail users had all their emails deleted due to a bug: https://gmail.googleblog.com/2011/02/gmail-back-soon-for-eve... They ended up having to restore them from tape backup, which took several days. Affected users also had all their incoming mail bounce for 20 hours.

Re: Slack is down

#670

Post by brlewis »

VectorLock wrote: post #436
glouwbug wrote: post #193

Earlier quoted context omitted.

What's there to change in Slack, though? It's arguably a messaging system, and that feature is tried and tested. That, and giphys, to be honest. EDIT: Guys it was a joke, chill

HN's tolerance for jokes and sarcasm is extremely low.

I'm not sure about that. I feel like I get more upvotes from sarcasm and jokes than from insight. In this instance, I think it's because when people hear something dumb said seriously in real life, they're not going to readily recognize online that it's a joke.
Post reply on HN