Live data from Hacker News

How Discord Stores Billions of Messages (2017)

blog.discord.com

241–250 of 377 posts

Re: How Discord Stores Billions of Messages (2017)

#241

Earlier quoted context omitted.

Honestly, I find discord super frustrating. Can't have multiple chats open at the same time, can't close the right rail, etc. It's UX is subpar in almost every way that matters to me. I use it because _everyone_ uses it, not because I want to.

Ditto, feels fast and has a lot of features but to me the UX is very confusing.

Discord is somewhat of a power user UI. They cater to people who are in 10+ large communities and use lots of the features provided.

It's not the simplest tool if all you want to do is PM a friend or two.

Re: How Discord Stores Billions of Messages (2017)

#242

Well... we have 3 node MongoDB cluster and are processing up to a million trades... per second. And a trade is way more complex than a chat message. Has tens to hundreds of fields, may require enriching with data from multiple external services and then requires to be stored, be searchable with unknown, arbitrary bitemporal queries and may need multiple downstream systems to be notified depending on a lot of factors…

Just curious, what was the rationale for choosing MongoDB?

Idk. I wasn't the one making the choice. I am just the one trying to make the best with what I have inherited.

Re: How Discord Stores Billions of Messages (2017)

#243
post #230

Well... we have 3 node MongoDB cluster and are processing up to a million trades... per second. And a trade is way more complex than a chat message. Has tens to hundreds of fields, may require enriching with data from multiple external services and then requires to be stored, be searchable with unknown, arbitrary bitemporal queries and may need multiple downstream systems to be notified depending on a lot of factors…

Sounds interesting. Does this "we" have any writings about this?

I'm not "we" but I have some experience in this area.

Computers are fast, basically. ACID transactions can be slow (if they write to "the" disk before returning success), but just processing data is alarmingly speedy.

If you break down things into small operations and you aggregate by day, you can always have big numbers. The monitoring system that I wrote for Google Fiber ran on one machine and processed 40 billion log lines per day, with only a few seconds of latency from upload start -> dashboard/alert status updated. (We even wrote to Spanner once-per-upload to store state between uploads, and this didn't even register as an increase in load to them. Multiple hundred thousand globally-consistent transactional writes per minute without breaking a sweat. Good database!)

apenwarr wrote a pretty detailed look into the system here: https://apenwarr.ca/log/20190216 And like him, I miss having it every day.

Re: How Discord Stores Billions of Messages (2017)

#244
post #47

Earlier quoted context omitted.

I don't get it, you can run mumble on any random Linux box in your house, you don't need to pay to have it hosted somewhere. Works find running on any box on your desk. Discord makes you the product. It's gratis in exchange for letting them spy on you. If you don't know why that's bad...

> you can run mumble on any random Linux box in your house That seems easy to you. That would be easy for me too and most likely 90% of the people on HackerNews. But the average person doesn't have a "random Linux box" in their house. Most people don't even know what Linux is. Most people would be overwhelmed just looking for the terminal emulator on their computer, before they even typed a command into it. Most peop…

I find it astounding that people here can not even grasp the concept of why Discord is popular. I am perfectly capable of hosting my own server and doing everything manually. But it is clear as day why discord wiped out the competition while most of the comments here seem dazed by the fact and are left wondering why people don't just use IRC.

It's no wonder so many projects and FOSS tools fail to gain large userbases when it seems that most developers seem to be living on another planet entirely.

Re: How Discord Stores Billions of Messages (2017)

#245

Earlier quoted context omitted.

> What's the size of all the data in your MongoDB instance? 3x12TB > In the big data world the "complexity" of the data doesn't really mean much. Oh how wrong you are. It is much easier to deal with data when the only thing you need to do is to just move it from A to B. Like "find who should see this message, make sure they see it". It is much different when you have large, rich domain model that runs tens of thousan…

Nice that's a pretty decent size, curious on the latency still. Thats the primary problem for a real time chat app. Complexity of data and running business rules on it is not a data store problem though, that's a compute problem. It's highly parallelizable and compute is cheap. For reference, my team runs transformations on about 1 PB of (uncompressed) data per day with 3 spark clusters, each with 50 nodes. We've got…

The latency is a complex topic.

In order to be able to run so much stuff on MongoDB, we almost never run single queries to the database. If I fetch or insert trade data, I probably run a query for 10 thousand trades at the same time.

So what happens is, as data comes from multiple directions it is being batched (for example 1-10 thousand at a time), split into groups that can be processed together in a roughly similar process, and then travels the pipeline as a single batch which is super important as it allows amortizing some of the costs.

Also the processing pipeline has many, many steps in it. A lot of them have buffers inbetween so that steps don't get starved for data.

All this causes latency. I try to keep it subsecond but it is a tradeoff between throughput and latency.

It could have been implemented better, but the implementation would be complex and inflexible. I think having clear, readable, flexible implementation is worth a little bit of tradeoff in latency.

As to storage being source of most woes, I fully agree. In our case it is trying to deal with bloat of data caused by business wanting to add this or that. All this data causes database caches to be less effective, requires more network throughput, more CPU for parsing/serializing, needs to be replicated, etc. So half the effort is constantly trying to figure out why they want to add this or that and is it really necessary or can be avoided somehow.

Re: How Discord Stores Billions of Messages (2017)

#246
post #92
post #67

Can anyone share experiences with using Discord as a communications tool in a workplace? We're currently on Google Chat because it comes with the package that we pay for anyway, but it's pretty lame. So from time to time we consider jumping to Slack. But then, why not Discord?

Discord would be awesome for work ... I want to use it so bad. But the terms of service are completely unpalatable. Discord basically gets a perpetual license to anything you post.

Isn't this the same for literally every service you use since they need a license to use it just to display it in app?

Re: How Discord Stores Billions of Messages (2017)

#247
post #8

I'm curious of the 2021 measure of total disk space that Discord consumes. Servers that I'm in share images every few minutes, which must add up pretty quick.

I'd imagine that the images are not part of the main database and that they are in some kind of s3 like file storage system.

Re: How Discord Stores Billions of Messages (2017)

#248

Earlier quoted context omitted.

Again, the article is not about throughput. How fast can you search across all historical trades?

As an example, there are bitemporal queries like "for the given population of trades specified by following rules, find the set of trades that met the rules at a particular point in time, based on our knowledge at another given point in time". Also trades are versioned (are a stream of business events from trading system), then have amendments (each event may be amended in the future but the older version must be pre…

How many simultaneous queries of that nature can the system handle?

Re: How Discord Stores Billions of Messages (2017)

#249

Earlier quoted context omitted.

yes. whataboutism.

So I shouldn't be able to delete account information about me on HN or Discord? Care to explain this?

no, sorry if unclear. i meant the opposite, you should be able to.

Re: How Discord Stores Billions of Messages (2017)

#250

Well... we have 3 node MongoDB cluster and are processing up to a million trades... per second. And a trade is way more complex than a chat message. Has tens to hundreds of fields, may require enriching with data from multiple external services and then requires to be stored, be searchable with unknown, arbitrary bitemporal queries and may need multiple downstream systems to be notified depending on a lot of factors…

Just curious, what was the rationale for choosing MongoDB?

Arctic [0] uses mongodb so perhaps that was inspiration?

[0] https://github.com/man-group/arctic

Post reply on HN