Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Stores Billions of Messages (2017)

blog.discord.com

Post reply on HN
Print view View original

361–370 of 377 posts

Re: How Discord Stores Billions of Messages (2017)

#361

Post by caymanjim »

yobbo wrote: post #316
caymanjim wrote: post #60

Earlier quoted context omitted.

It's free for the same reason everything is free these days. VC funds anything that will attract a lot of users to mine data from so they can sell the data. Discord didn't do anything that was groundbreaking or even solve a problem that had no solution; they just came along during a time when investors are willing to fund a company operating at a loss for a decade until FANG buys them. Discord's a pretty good product…

> Same for Slack The ux of Slack is essentially screen+irc implemented in JS with emotes. It enabled technical and non-technical people to use the same tool. The key to success is not technical, it's that they tailored the product to a specific group that would then lock itself in. I didn't understand Discord's success, but comments here point that gamers couldn't find free group-voice apps at a critical time. Here a…

I wasn't referring to things like IRC. When Slack was initially released, it was no different from Campfire and a whole string of other web-based chat systems that came and went going all the way back to the dawn of AJAX in the late 90s. Slack's improved a lot since then, with app integrations and other features, but fundamentally it wasn't any different than its predecessors. It's easy to think that Slack did something groundbreaking, or figured out the magic solution to the problem that sank its predecessors, but just like Discord, the reason Slack won is because it came along at a time when companies can raise tens or hundreds of millions of dollars to float them for years while offering a free product. Then they can upsell later, and/or commoditize their users' personal information. Those business models weren't as easy to come by in the past, so a lot of products failed. None of this is to bash Slack; it's an adequate product for what it does.

Another big thing that the current crop of winners has going for it is that cloud hosting allows applications to launch literally for free and scale quite a bit without paying much of anything in infrastructure costs. That also wasn't an option 10-20 years ago.

Re: How Discord Stores Billions of Messages (2017)

#362

Post by AtlasBarfed »

imadeamistake wrote: post #200
PretzelPirate wrote: post #167

Earlier quoted context omitted.

The bigger catch is that when your partition grows too big and your nodes are hit by the OOMKiller, you have very few options other than create a new table and replay data, or use a cli tool to manually partition your data while the node is offline. Using Cassandra tends to mean pushing costs to your developers instead of spending more money on storage resources, and your devs will almost certainly spend a ton of tim…

Cassandra won't try and load a partition into memory. It doesn't work that way. The only way you would get behavior like that is by setting "allow filtering" to on. Allow filtering is a dedicated keyword for "I know I shouldn't do this but I'm going to anyway". If you're trying to run those types of queries, use a different database. If someone is making you use a transactional database without joins for analytical l…

I huge partition is often spread across multiple sstables, and often has tombstone issues if the large partition consists of a large number of column keys or any regular update cycle, which is often the case for hot rows.

In that case the overhead of processing and collecting all the parts of the data you need spread across different sstables and then do tombstones can lead to a lot of memory stress.

Re: How Discord Stores Billions of Messages (2017)

#363

Post by AtlasBarfed »

bsaul wrote: post #173
PretzelPirate wrote: post #167

Earlier quoted context omitted.

The bigger catch is that when your partition grows too big and your nodes are hit by the OOMKiller, you have very few options other than create a new table and replay data, or use a cli tool to manually partition your data while the node is offline. Using Cassandra tends to mean pushing costs to your developers instead of spending more money on storage resources, and your devs will almost certainly spend a ton of tim…

what would you recommend now instead ?

Kind of the million dollar question. People like to complain about cassandra, but that just brings up the adage about C++: people complain about the things they use.

But let's not pretend that cassandra isn't almost always a bear. The other problem is that cassandra keeps things up (and never gets the credit for it) but that creates a host of edge cases and management headaches (which makes management hate it).

Most competitors abandon AP for CP (HBase and Cockroach and I think FoundationDB) in order to get joins and SQL, but the BFD on cassandra is the AP design.

Scylla did a C++ rewrite to address tail latency due to JVM GC, but after an explosive release cycle, they basically stalled at partial 2.2 compatiblity. Rocksandra isn't in mainline and doesn't appear to be worked on anymore.

I follow the Jepsen tests a lot: they don't seem to have found a magic solution.

I think Cassandra stopped short of some key OSS deliverables, and I think they could simplify the management as well, both with a UI for admin and with some re-jiggering of how some things work on nodes. The devs are simply swamped with stability and features right now.

And Datastax won't help that much, what admin UI cassandra had was abandoned, and I half think the reason they acquired TLP was that TLP was producing/sponsoring useful admin tooling.

I would love to try something new. What appeals to me about cassandra is the fundamentals of the design, and the fair amount of tranparency there is (although there is still some marketing bullcrap that surrounds it like "CQL is like SQL" and other big lies).

So many other NoSQL's are bolt-on capabilities for handling distribution that Jepsen exposes (MongoDB famously) and have sooo much bullcrap in their claims. All the NoSQLs are desperate for market share, so they all lie about CAP and the edge cases.

Purely distributed databases are VERY HARD and are open to exaggeration, handwaving, and false demonstrations by the salesmen, but those people won't be around when you need a database like this to shine: when the shit hits the fan, you lose an entire datacenter, or similar things.

Re: How Discord Stores Billions of Messages (2017)

#364

Post by onkoe »

jgilias wrote: post #67

Can anyone share experiences with using Discord as a communications tool in a workplace? We're currently on Google Chat because it comes with the package that we pay for anyway, but it's pretty lame. So from time to time we consider jumping to Slack. But then, why not Discord?

It's everything you could want, but this a lot of asterisks. A lot of things are limited (check other comment), but most importantly, their policies force you to hand over any text you write on their platform. But frankly, I'm not sure if there's anything near a perfect solution. A lot of companies use a big clump of services including both self-hosted and "rented". I hope to one day see a better, more comprehensive solution in line with the future of online work.

Re: How Discord Stores Billions of Messages (2017)

#365

Post by vortico »

Ekaros wrote: post #335
vortico wrote: post #222

Earlier quoted context omitted.

I'd imagine Discord uses deduplication, but I bet it doesn't save them 5% storage space.

I think it might, spamming same meme images over and over is quite common in some servers. On other hand the bigger pictures might overhelm these just in size.

Yeah, that's why I assumed it wouldn't help that much. People re-upload 100kB memes all the time, but the bulk would probably be 5MB phone pictures that won't typically be re-uploaded.

Re: How Discord Stores Billions of Messages (2017)

#366

Post by Kaze404 »

SilverRed wrote: post #240
Kaze404 wrote: post #158

Earlier quoted context omitted.

It's definitely improved over the years, but every remotely populated server I'm in uses bots for basic moderation features like ban words, proper bans/kicks (for example, temporary bans), warns, etc. There's still a long way to go in my opinion.

I'm not aware of anything that does a better job than discord. So they can be doing a fantastic job relative to the competition while still leaving stuff to be desired. Although bots are not really a bad solution and they leave the tools in the hands of the users who can now do just about anything.

I disagree. In my opinion those tools are the bare minimum for effective moderation, and while I love that Discord gives developers an API that allows them to implement those systems, I think it's something that should be handled by Discord themselves.

Re: How Discord Stores Billions of Messages (2017)

#367

Post by porb121 »

mjevans wrote: post #137
porb121 wrote: post #86

Earlier quoted context omitted.

uh, how about the fact that vent/mumble/Skype were all awful?

The only bad part of mumble calls for voice come from: * Where the server is hosted / quality of server * Poor client UI The client UI issue is how easy it is to work-around bad audio from other users. It's possible to do, the UI just completely sucks. User interface and end user fulfillment just aren't great generally for OSS. I think it would take a commons improvement project with either government grants (infrast…

i totally disagree. the worst part about mumble is that it was a pain in the ass to set up. creating a discord server is trivial: all of my nontechnical friends have used the software just fine. mumble is terribly fiddly in comparison.

Re: How Discord Stores Billions of Messages (2017)

#368

Post by jsmith45 »

Saris wrote: post #122
Kaze404 wrote: post #82

Earlier quoted context omitted.

I can't think of many reasons Discord "ate the whole market" besides smart marketing, honestly. It does audio rooms incredibly well, but everything else (even their developer support team) is just terrible.

Everything else was/is much worse. Even years later it's still the only platform I know of that combines text chat rooms, voice chat rooms, and video streaming into one place, all accessible from your 'server' as they call it. It also has clients for many platforms, including a web client, all of which look and function the same. Any alternative out there does one of those things decently well, but either completely…

I think the most direct competitor to discord (for gaming and related communities) is guilded.gg which is basically a clone of discord that offers additional features as well, including offering most of the features that discord has paywalled behind "nitro" for free.

The big issue they have is building up a large enough network effect. I really can't see the discord communities I am a part of moving over there any time soon. Also, they were recently acquired by roblox, and nobody knows for sure what the new ownership will end up doing to the platform.

Re: How Discord Stores Billions of Messages (2017)

#369

Post by ljw1001 »

lmilcin wrote: post #228

Well... we have 3 node MongoDB cluster and are processing up to a million trades... per second. And a trade is way more complex than a chat message. Has tens to hundreds of fields, may require enriching with data from multiple external services and then requires to be stored, be searchable with unknown, arbitrary bitemporal queries and may need multiple downstream systems to be notified depending on a lot of factors…

Curious as to how many days of data you have in your cluster. It seems like it could be ~1/2 billion records per day, 125 billion per year-ish. In a few years your 3 node Mongo cluster would be getting towards volumes I associate with a 'big data' kind of solution like BigTable.

Re: How Discord Stores Billions of Messages (2017)

#370

Post by fao_ »

phailhaus wrote: post #359
fao_ wrote: post #255

Earlier quoted context omitted.

Are you not sure that financial data "with hundreds of fields" is more complex than chat data which has a relatively linear threading and only a handful of fields?

I'm asking about how your system scales to the number of queries, but you seem to be taking every question personally. You seem to really want to make sure everyone knows that you think Discord's problems are easy to solve. I'm not saying Discord is more complicated, but you're not really giving enough information to prove that Discord's problems are a subset of yours. Do you support more simultaneous queries than Di…

I think you need to recheck my username, you have mistaken me with another poster in the thread.
Post reply on HN