Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Stores Billions of Messages (2017)

blog.discord.com

Post reply on HN
Print view View original

281–290 of 377 posts

Re: How Discord Stores Billions of Messages (2017)

#281

Post by giancarlostoro »

lmilcin wrote: post #228

Well... we have 3 node MongoDB cluster and are processing up to a million trades... per second. And a trade is way more complex than a chat message. Has tens to hundreds of fields, may require enriching with data from multiple external services and then requires to be stored, be searchable with unknown, arbitrary bitemporal queries and may need multiple downstream systems to be notified depending on a lot of factors…

> -- if you are willing to spend a little bit of learning effort, it is easily possible to run millions of non trivial transactions per second on a single server,

I got into programming through the Private Server (gaming) scene. You learn that the more you optimize and refactor your code to be more efficient, the more you can handle on less hardware, including embedded systems. So yeah, it's amazing how much is wasted. I'm kind of holding hope that things like Rust and Go focus on letting you get more out of less hardware.

Re: How Discord Stores Billions of Messages (2017)

#282

Post by kerng »

throwdbaaway wrote: post #276
lmilcin wrote: post #262

Earlier quoted context omitted.

Data doesn't come or leave batched. This is just internal mechanism. Think in term of Discord, their database probably already queues and batches writes. Or maybe they could decide to fetch details of multiple users with a single query by noticing there are 10k concurrent asks for user details. So why have 10k queries when you could have 10 queries for 1k user objects? If you complain that my process is different bec…

Of course, cassandra/mongodb/etc can perform their own batching when writing to the commit log, and can also benefit from write combining by not flushing out the dirty data immediately. That's besides the point. Your use case allows you to perform batching for writes at the *application layer*, while discord's use case doesn't.

Why couldn't others with lots of traffic use a similar approach? I assume they do. Seems pretty genius idea to batch things like that, especially when qps is very high batching (maybe waiting for a few ms to fill a batch) makes a lot of sense.

Re: How Discord Stores Billions of Messages (2017)

#283

Post by PeterCorless »

Notanothertoo wrote: post #261

We use scylla for our IoT stream, bucket per day, with a date index for second resolution data. The current day is a hot spot of, but we throw that in redis. It's running one of the largest re insurance providers IoT deployments.

That sounds like one of my favorite use cases, Meshify / MunichRE! :)

A little history lesson is in order:

https://www.scylladb.com/2019/02/01/meshify-and-scylla-an-in...

Re: How Discord Stores Billions of Messages (2017)

#284

Post by tonymet »

lmilcin wrote: post #228

Well... we have 3 node MongoDB cluster and are processing up to a million trades... per second. And a trade is way more complex than a chat message. Has tens to hundreds of fields, may require enriching with data from multiple external services and then requires to be stored, be searchable with unknown, arbitrary bitemporal queries and may need multiple downstream systems to be notified depending on a lot of factors…

Considering that a cpu can do 3 billion things a second , and a typical laptop can store 16 billion things in memory , it shouldn’t take more than 5 of these to handle “billions of messages” . I agree with you that modern frameworks are inefficient

Re: How Discord Stores Billions of Messages (2017)

#285

Post by adamnemecek »

tonymet wrote: post #284
lmilcin wrote: post #228

Well... we have 3 node MongoDB cluster and are processing up to a million trades... per second. And a trade is way more complex than a chat message. Has tens to hundreds of fields, may require enriching with data from multiple external services and then requires to be stored, be searchable with unknown, arbitrary bitemporal queries and may need multiple downstream systems to be notified depending on a lot of factors…

Considering that a cpu can do 3 billion things a second , and a typical laptop can store 16 billion things in memory , it shouldn’t take more than 5 of these to handle “billions of messages” . I agree with you that modern frameworks are inefficient

By 16 billion things you mean 16 billion bytes? If you are talking about physical memory, then no, you can't occupy the entire memory. If you are talking about virtual memory, then you can store more data.

Re: How Discord Stores Billions of Messages (2017)

#286

Post by lmm »

Kaze404 wrote: post #134
porb121 wrote: post #86

Earlier quoted context omitted.

uh, how about the fact that vent/mumble/Skype were all awful?

I don't see how that's relevant? I don't even prefer those over Discord, but I don't think it's enough of an improvement to warrant the market share it has now.

Small amounts of friction make a big difference. Back when my gaming friends were using Mumble, half of the group wouldn't bother joining voicechat (and we were lucky to have someone technical enough to run the server in the first place); with Discord it's easy enough that everyone does it.

Re: How Discord Stores Billions of Messages (2017)

#287

Post by PeterCorless »

objektif wrote: post #220
PeterCorless wrote: post #216

Earlier quoted context omitted.

Yes. There are ways to group images that seem to be the same. TinEye and Google image search do that. So you'd have a collection of related hashes that equal "Bob's prom photo where he looks like a goofer."

Yes definitely I have seen it work in action but you cant just tell a user "here use this smaller and more pixelated version of your image that we think is kind of similar".

Oof! No, you can't. :D

Re: How Discord Stores Billions of Messages (2017)

#288

Post by knownjorbist »

dancemethis wrote: post #279
umvi wrote: post #23

Discord is so good. I just can't imagine it can stay this good forever. My fear is that eventually it will be bought out and aggressively monetized.

It's not good. It's user hostile software.

Only out-of-touch tech elitists on HN think this.

Re: How Discord Stores Billions of Messages (2017)

#289

Post by tomnipotent »

PeterCorless wrote: post #33
preetamjinka wrote: post #18

Looks like they migrated (at least partially) to Scylla: "Discord Chooses Scylla as Its Core Storage Layer" (2020) https://www.scylladb.com/press-release/discord-chooses-scyll...

Yes. They started experimenting with Scylla earlier, and made the switch in 2020. Here's more on their logic — they like "opinionated systems": https://www.scylladb.com/2019/03/20/discord-on-the-joy-of-op...

Scylla is really impressive. The only complaint I've heard is about the shard-per-cpu approach, causing some issues when data has bad distribution.

Re: How Discord Stores Billions of Messages (2017)

#290

Post by hyperhopper »

hardwaregeek wrote: post #252
disillusioned wrote: post #227

Earlier quoted context omitted.

There are plenty of reasons to do this, all of which have to do with, say, privacy, and Slack has managed their way into making it dramatically less annoying. 1) They send magic links. Pretty easy. 2) They make all known workspaces you've logged into before discoverable and allow for a one-click "add to desktop Slack" option, which makes dealing with the whole "different users" issue. And to the extent that I use dif…

That's true, but it doesn't change the user story going from "I click a link, I join the workspace" to "I click a link, I fill out yet another registration form, decide which email to use, add another password to my password safe, then join the workspace". Minor differences but friction does matter.

To add onto this, you also have no cross platform user consistency, so if I DM a user, if I want to search my DMs there, I might have to search my DMs in 10+ servers for the specific message I want. There are a million other problems with this model but this example is definitely one that I frequently ran into before people I knew switched to discord
Post reply on HN