Live data from Hacker News

How Discord Stores Billions of Messages (2017)

blog.discord.com

271–280 of 377 posts

Re: How Discord Stores Billions of Messages (2017)

#271
post #67

Can anyone share experiences with using Discord as a communications tool in a workplace? We're currently on Google Chat because it comes with the package that we pay for anyway, but it's pretty lame. So from time to time we consider jumping to Slack. But then, why not Discord?

We used Discord for a while for our team at work for over a year. Stopped using because company policy changed (we had no centralized chat program, some teams were on Skype, some on WhatsApp, then Slack was instituted). Context: 10-person, mostly technical, project-oriented, game development team.

---

It was a joy to use, we created channels left right and center and knew everyone needed would be in them thanks to the centralized "role-based" permission system. (We would create project-specific channels and an accompanying role, or client-specific roles for the few high-throughput clients that had lots of small projects)

At the time it did not have threading, which was one of the biggest pain points on the text-chat front.

---

The voice chat is very good, and having dedicated voice channels means you can emulate meeting rooms or desks and have people join as desired/needed. You could be working and idle at "kroltan's desk" voice channel, but even if you weren't, joining one is trivial (a single click, can be done independently by many people) compared to Slack (find the call button somewhere different each time because they redesign the UI every week, then wait for your peer to join the call).

Screen sharing is 720p on the free plan, so for meetings, it was hard to read documents, requiring zooming and whatnot. At the time there was also no setting to optimize for framerate or definition, so even 720p felt closer to 480p. Nowadays you can lower the framerate and also select the desired optimization, so you can ask Discord to optimize the stream for quality which is much better for documents, even in 720p.

---

The client is also much more responsive than other Electron-based chat programs, especially with big workspaces with close to a hundred channels (yes, for a 10-person team, we sure type a lot), search is basically instant and has very useful filters, mentioning roles is great and the notification settings are fine-grained enough to please everyone.

Re: How Discord Stores Billions of Messages (2017)

#272

Well... we have 3 node MongoDB cluster and are processing up to a million trades... per second. And a trade is way more complex than a chat message. Has tens to hundreds of fields, may require enriching with data from multiple external services and then requires to be stored, be searchable with unknown, arbitrary bitemporal queries and may need multiple downstream systems to be notified depending on a lot of factors…

A million sounds impressive, but this is clearly not serialized throughput based on other comments here. Getting a million of anything to fast NVMe is trivial if there is no contention and you are a little clever with IO.

I have written experimental datastores that can hit in excess of 2 million writes per second on a samsung 980 pro. 1k object size, fully serialized throughput (~2 gigabytes/second, saturates the disk). I still struggle to find problem domains this kind of perf can't deal with.

If you just care about going fast, use 1 computer and batch everything before you try to put it to disk. Doesn't matter what fancy branding is on it. Just need to play by some basic rules.

Primary advantage with 1 computer is that you can much more easily enforce a total global ordering of events (serialization) without resorting to round trip or PTP error bound delays.

Re: How Discord Stores Billions of Messages (2017)

#273

Earlier quoted context omitted.

The latency is a complex topic. In order to be able to run so much stuff on MongoDB, we almost never run single queries to the database. If I fetch or insert trade data, I probably run a query for 10 thousand trades at the same time. So what happens is, as data comes from multiple directions it is being batched (for example 1-10 thousand at a time), split into groups that can be processed together in a roughly simila…

I thought you are doing millions qps with a 3 nodes mongodb cluster, from the top level comment. That would be impressive. By batching 1-10 thousands records at a time, your use case is very different from discord, which needs to deliver individual messages as fast as possible.

[deleted]

Re: How Discord Stores Billions of Messages (2017)

#274
post #203

Earlier quoted context omitted.

I don't know much about image de-deuplication, but maybe they can get some sort of fingerprint/hash for an image, see if they already have it, and then serve that already existing image. I'd imagine a hash like SHA256 would be tricky because if that image was compressed an additional time at all throughout it's internet journey, then we'd get a different resulting hash, but maybe there is an effective way to fingerpr…

I'd imagine Discord uses deduplication, but I bet it doesn't save them 5% storage space.

The plural of anecdote isn't data, but about 20% of the images I post on Discord come from Discord in the first place, cross-posting among different servers.

Re: How Discord Stores Billions of Messages (2017)

#275

Could somebody help me understand the reasoning behind `timestamp = snowflake_id >> 22` Thanks :)

IIRC a snowflake ID's first few bits (or characters?) are by definition a unix timestamp.

Ah it gets rid of the non-time related bits. Below is a description of the formatting

* id is composed of: * time - 41 bits (millisecond precision w/ a custom epoch gives us 69 years) * configured machine id - 10 bits - gives us up to 1024 machines * sequence number - 12 bits - rolls over every 4096 per machine (with protection to avoid rollover in the same ms)

thanks :)

Re: How Discord Stores Billions of Messages (2017)

#276

Earlier quoted context omitted.

I thought you are doing millions qps with a 3 nodes mongodb cluster, from the top level comment. That would be impressive. By batching 1-10 thousands records at a time, your use case is very different from discord, which needs to deliver individual messages as fast as possible.

Data doesn't come or leave batched. This is just internal mechanism. Think in term of Discord, their database probably already queues and batches writes. Or maybe they could decide to fetch details of multiple users with a single query by noticing there are 10k concurrent asks for user details. So why have 10k queries when you could have 10 queries for 1k user objects? If you complain that my process is different bec…

Of course, cassandra/mongodb/etc can perform their own batching when writing to the commit log, and can also benefit from write combining by not flushing out the dirty data immediately. That's besides the point.

Your use case allows you to perform batching for writes at the *application layer*, while discord's use case doesn't.

Re: How Discord Stores Billions of Messages (2017)

#277

Earlier quoted context omitted.

Why would the text data be stored in TOAST? My understanding was PG only uses TOAST when the data is too large to fit in the row, and since PG compresses data before inserting wouldn't user messages be fine?

I think Discord messages can be unbounded in size

Interesting, trying it out on Discord the default max msg size is 2000 chars and with Discord Nitro the max is raised to 4000 chars.

Testing with Postgres, a 2000 char random sequence doesn't result in TOASTing, but a 4000 random sequence does get TOASTed

And for kicks, 4000 chars that aren't random compress well enough that they don't end up in TOAST.

Re: How Discord Stores Billions of Messages (2017)

#280

Earlier quoted context omitted.

I like Scylla. I've been working with it for a while now, and it's a good alternative for transactional loads. It's a hell of a lot faster than Cassandra, and much much much much cheaper than DynamoDB. Cassandra has always felt like improvements came in fits and starts. I work at a Fortune 50 company, and Amazon quoted us ~$2-3 million a year to run our load on Dynamo (we were paying $350k/year for Aurora). With Scyl…

I'd love to hear how Dynamo would end up being $2-3 million a year They sure do a great job of convincing people that it's cheap so I'm curious where the cost seems to blow up?

If you are doing north of a million ops on DynamoDB you can quickly run into the $2-3 million a year range.

In this 2018 benchmark, we were able to calculate that a sustained, provisioned of only 160k write ops / 80k read ops for DynamoDB would cost >$500k per year:

https://www.scylladb.com/2018/12/13/scylla-vs-amazon-dynamod...

That was a few years ago. These days, according to our most current pricing you could do DynamoDB provisioned, 1 year reserved for $38,658/month, which is "only" $463,896 annually (pop up the "Details" button and choose "vs. DynamoDB"):

https://www.scylladb.com/pricing/?writes=160000&reads=80000&...

The same workload on Scylla Cloud would be only $7,442/month, or $89,304 annually.

If you wanted, say, 1m ops — 500k write / 500k read ops — on DynamoDB, that'll run you $131,078/month, or $1,572,936 per year.

https://www.scylladb.com/pricing/?writes=500000&reads=500000...

The same workload on Scylla Cloud would run $29,768 reserved/month, or $357,216 per annum — 77% cheaper.

Of course, all of this is just pure list price. Depending on volume you might be able to negotiate better pricing. However, you'd need a really steep discount for DynamoDB just to get back to Scylla Cloud's list price.

Let me know if you spot any math errors or omissions on my part.

Post reply on HN