Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Stores Billions of Messages (2017)

blog.discord.com

Post reply on HN
Print view View original

231–240 of 377 posts

Re: How Discord Stores Billions of Messages (2017)

#231

Post by emptysea »

aeyes wrote: post #202
andrewstuart wrote: post #114

I'd first reach for Postgres to do this. Anyone have any idea how Postgres would stack up in a similar challenge?

I see several issues: - No out of the box horizontal sharding, according to the post they had 4TB (compressed) data in the cluster in 2017. Looking at their growth I think it is safe to assume that today they would have >50TB which can't be done on a single node. You could use Citus but this is not exactly vanilla Postgres anymore. For such a simple data model wasting time implementing your own sharding solution and…

Why would the text data be stored in TOAST?

My understanding was PG only uses TOAST when the data is too large to fit in the row, and since PG compresses data before inserting wouldn't user messages be fine?

Re: How Discord Stores Billions of Messages (2017)

#232

Post by lmilcin »

kevinsundar wrote: post #229
lmilcin wrote: post #228

Well... we have 3 node MongoDB cluster and are processing up to a million trades... per second. And a trade is way more complex than a chat message. Has tens to hundreds of fields, may require enriching with data from multiple external services and then requires to be stored, be searchable with unknown, arbitrary bitemporal queries and may need multiple downstream systems to be notified depending on a lot of factors…

Thats not really what this article is about. Their problem wasn't throughput. What's the size of all the data in your MongoDB instance? And what's the latency in your reads? In the big data world the "complexity" of the data doesn't really mean much. It's just bytes.

> What's the size of all the data in your MongoDB instance?

3x12TB

> In the big data world the "complexity" of the data doesn't really mean much.

Oh how wrong you are.

It is much easier to deal with data when the only thing you need to do is to just move it from A to B. Like "find who should see this message, make sure they see it".

It is much different when you have large, rich domain model that runs tens of thousands of business rules on incoming data and each entity can have very different processing depending on its state and the event that came.

I am writing whole applications just to data-mine our processing flow just to be able to understand a little bit of what is happening there.

At that traffic you can't even log anything for each of the transactions. You have to work indirectly through various metrics, etc.

Re: How Discord Stores Billions of Messages (2017)

#233

Post by lmilcin »

dpedu wrote: post #230
lmilcin wrote: post #228

Well... we have 3 node MongoDB cluster and are processing up to a million trades... per second. And a trade is way more complex than a chat message. Has tens to hundreds of fields, may require enriching with data from multiple external services and then requires to be stored, be searchable with unknown, arbitrary bitemporal queries and may need multiple downstream systems to be notified depending on a lot of factors…

Sounds interesting. Does this "we" have any writings about this?

I have a plan to write a book on how to write reactive applications like that. Mostly collection of observations, tips, tricks, patterns for reactive composition, some very MongoDB specific solutions, etc.

Not sure how many people would be interested. Reactor has quite steep learning curve but also very little literature on how to use for anything non-trivial.

The aim is not just enable good throughput, but also achieve this without compromising on clarity of implementation. Which is where I think reactive, and specifically ReactiveX/Reactor, shines.

Re: How Discord Stores Billions of Messages (2017)

#234

Post by phailhaus »

lmilcin wrote: post #232
kevinsundar wrote: post #229

Earlier quoted context omitted.

Thats not really what this article is about. Their problem wasn't throughput. What's the size of all the data in your MongoDB instance? And what's the latency in your reads? In the big data world the "complexity" of the data doesn't really mean much. It's just bytes.

> What's the size of all the data in your MongoDB instance? 3x12TB > In the big data world the "complexity" of the data doesn't really mean much. Oh how wrong you are. It is much easier to deal with data when the only thing you need to do is to just move it from A to B. Like "find who should see this message, make sure they see it". It is much different when you have large, rich domain model that runs tens of thousan…

Again, the article is not about throughput. How fast can you search across all historical trades?

Re: How Discord Stores Billions of Messages (2017)

#235

Post by kevinsundar »

lmilcin wrote: post #232
kevinsundar wrote: post #229

Earlier quoted context omitted.

Thats not really what this article is about. Their problem wasn't throughput. What's the size of all the data in your MongoDB instance? And what's the latency in your reads? In the big data world the "complexity" of the data doesn't really mean much. It's just bytes.

> What's the size of all the data in your MongoDB instance? 3x12TB > In the big data world the "complexity" of the data doesn't really mean much. Oh how wrong you are. It is much easier to deal with data when the only thing you need to do is to just move it from A to B. Like "find who should see this message, make sure they see it". It is much different when you have large, rich domain model that runs tens of thousan…

Nice that's a pretty decent size, curious on the latency still. Thats the primary problem for a real time chat app.

Complexity of data and running business rules on it is not a data store problem though, that's a compute problem. It's highly parallelizable and compute is cheap.

For reference, my team runs transformations on about 1 PB of (uncompressed) data per day with 3 spark clusters, each with 50 nodes. We've got about 70ish PB of (compressed) data queryable. All our challenges come from storage, not compute.

Re: How Discord Stores Billions of Messages (2017)

#236

Post by sumnole »

lmilcin wrote: post #228

Well... we have 3 node MongoDB cluster and are processing up to a million trades... per second. And a trade is way more complex than a chat message. Has tens to hundreds of fields, may require enriching with data from multiple external services and then requires to be stored, be searchable with unknown, arbitrary bitemporal queries and may need multiple downstream systems to be notified depending on a lot of factors…

Just curious, what was the rationale for choosing MongoDB?

Re: How Discord Stores Billions of Messages (2017)

#237

Post by lmilcin »

phailhaus wrote: post #234
lmilcin wrote: post #232

Earlier quoted context omitted.

> What's the size of all the data in your MongoDB instance? 3x12TB > In the big data world the "complexity" of the data doesn't really mean much. Oh how wrong you are. It is much easier to deal with data when the only thing you need to do is to just move it from A to B. Like "find who should see this message, make sure they see it". It is much different when you have large, rich domain model that runs tens of thousan…

Again, the article is not about throughput. How fast can you search across all historical trades?

As an example, there are bitemporal queries like "for the given population of trades specified by following rules, find the set of trades that met the rules at a particular point in time, based on our knowledge at another given point in time". Also trades are versioned (are a stream of business events from trading system), then have amendments (each event may be amended in the future but the older version must be preserved). Our system can also amend the data (for example to add some additional data to the trade later). All this causes trades to be a tree of immutable versions you need to comb through. A trade can have anywhere from 1 to 30k versions.

This takes about 20 seconds. The process opens about 200 connections to the cluster and transfers data at about 2-4GB/s.

Re: How Discord Stores Billions of Messages (2017)

#238

Post by yawnxyz »

bonesinger wrote: post #58
SaltyBackendGuy wrote: post #49

Earlier quoted context omitted.

> It works nice and is free, but for how long? I wonder about this a lot. I wonder if they have some big 'whales' that help sustain their business OR they're just selling all of our data (is that enough to make money at discords scale??).

IMO its a good competitor to slack. They probably make money from businesses too. They have lots of options for permissions/roles and all kinds of API access to write bots for.

we're trying to use Discord for our multi-site grant-funded healthcare project... it's pretty messy to use. Would love to pay for some decent support... People are getting locked out of their accounts for some reason and working with their support team is very painful.

Re: How Discord Stores Billions of Messages (2017)

#239

Post by SilverRed »

Kaze404 wrote: post #134
porb121 wrote: post #86

Earlier quoted context omitted.

uh, how about the fact that vent/mumble/Skype were all awful?

I don't see how that's relevant? I don't even prefer those over Discord, but I don't think it's enough of an improvement to warrant the market share it has now.

So what do you think happened? That people were manipulated in to using discord? Or that they don't know what the alternatives are?

Everyone I have spoken to loves discord and thinks it is one of the best programs they have. It's only a select group of hacker news style users who complain about minute details the average person does not care about.

I know it's hard for most people on this site to understand but the average user has very different priorities. Being able to create a "server" with the click of a button is worth more than every other issue listed in this thread. Having to pay or self host to create a group is a total non starter in 2021.

Re: How Discord Stores Billions of Messages (2017)

#240

Post by SilverRed »

Kaze404 wrote: post #158
crocodiletears wrote: post #129

Earlier quoted context omitted.

Their moderation and community management tools are fantastic.

It's definitely improved over the years, but every remotely populated server I'm in uses bots for basic moderation features like ban words, proper bans/kicks (for example, temporary bans), warns, etc. There's still a long way to go in my opinion.

I'm not aware of anything that does a better job than discord. So they can be doing a fantastic job relative to the competition while still leaving stuff to be desired. Although bots are not really a bad solution and they leave the tools in the hands of the users who can now do just about anything.
Post reply on HN