Live data from Hacker News

How Discord Stores Billions of Messages (2017)

blog.discord.com

171–180 of 377 posts

Re: How Discord Stores Billions of Messages (2017)

#171
post #94

Earlier quoted context omitted.

The reason for Discord over Slack, BTW is voice & video. Otherwise Slack is working great.

Slack added voice now: https://slack.com/help/articles/4402059015315-Start-a-huddle...

"Huddles" pick up so much noice from the background I stopped using it. Whenever I had to take a call without headphones on it was basically unusable. I'm a single data point however, your milage may vary.

Re: How Discord Stores Billions of Messages (2017)

#172

Maybe I don't know enough about databases, but could you really not have done this with postgres?

Afaik you could up until the data exceeds your capacity to fit in one machine, at which point you have to figure out how to split your data up in a way which lets you preserve all the strengths of sql (strong consistency). At that point you run into a lot of complexity with managing your shards.

Re: How Discord Stores Billions of Messages (2017)

#173

Earlier quoted context omitted.

Reading the article I was right now visiting Cassandra site to figure out what the catch is. Surely there should be a catch. Well, here it is. The partitioning in manual upto the SQL level.

The bigger catch is that when your partition grows too big and your nodes are hit by the OOMKiller, you have very few options other than create a new table and replay data, or use a cli tool to manually partition your data while the node is offline. Using Cassandra tends to mean pushing costs to your developers instead of spending more money on storage resources, and your devs will almost certainly spend a ton of tim…

what would you recommend now instead ?

Re: How Discord Stores Billions of Messages (2017)

#174
post #37

Earlier quoted context omitted.

It works nice and is free, but for how long? How long can they keep paying for that bandwidth and message data storage while keeping the thing essentially free?

Nitro pulls $10/mo and has enough benefits that a lot of people pay. Plus server boosts. I bet that they have good cash flow.

we use discord for work and to "boost" your server to a level where you get reasonable streaming you need to pay ~$60 and a higher upload limit, or ~$110 for the max. Which is pretty good in that it applies for all users

It's a bit of an odd model for paying for businesses, but works well in the gaming world where multiple people can essentially help pay for a server (if you want the extra toys)

Re: How Discord Stores Billions of Messages (2017)

#175

Discord had like $300M invested and they created unparalleled piece of software that ate whole market, damn. One of the most impressive softwares that I've seen and use after years of using ventrilo/mumble/teamspeak.

Elixir ftw

Want to build a real time chat app? Best build it on the BEAM.

Re: How Discord Stores Billions of Messages (2017)

#176

Earlier quoted context omitted.

At the time MongoDB's sharding story wasn't great. They've gotten better since, but still have a primary-replica set model that has a single point of failure/failover. Cassandra (and Scylla) are leaderless, peer-to-peer clustering. Any node can go offline and the cluster keeps humming. Cassandra shards per node. Scylla goes beyond that and shards per core. Cassandra and Scylla also use hinted handoffs so if a node is…

MongoDB has the equivalent of hinted handoffs. Changes are streamed to secondary nodes via the oplog, and the secondary just resumes where it was once it is back online. There is a limit to how long it can be offline (based on the size of the oplog), but that is the same limitation as hinted handoffs.

Thanks! Good to know.

Re: How Discord Stores Billions of Messages (2017)

#177
post #163

I would have used CockroachDB, it has all the requirements listed and you don't need to know in advance the queries you will perform when deciding the database schema.

On the day that this blog post was written, CockroachDB was only at beta-20170112 and didn't even had a production release yet.

v1.0 was released on May 10, 2017 [1], so I doubt it was even on their mind when they started working on the project.

[1] https://www.cockroachlabs.com/docs/releases/index.html

Re: How Discord Stores Billions of Messages (2017)

#178
post #163

I would have used CockroachDB, it has all the requirements listed and you don't need to know in advance the queries you will perform when deciding the database schema.

This would not work at scale for a company like Discord, with its volume of traffic. Cockroach, being consistency-oriented would quickly become transaction-bound. You want a database like a Cassandra or Scylla that is more performance/availability oriented. Otherwise you are going to see a lot of lag and latency in the Discord chat.

Cockroach is very, very good for a distributed SQL database. But it's still performance-limited in its very nature.

More here on the difference between NoSQL/NewSQL performance, using Scylla (a CQL-workalike) as a point of comparison:

https://www.scylladb.com/2021/01/21/cockroachdb-vs-scylla-be...

Re: How Discord Stores Billions of Messages (2017)

#179
post #163

I would have used CockroachDB, it has all the requirements listed and you don't need to know in advance the queries you will perform when deciding the database schema.

On the day that this blog post was written, CockroachDB was only at beta-20170112 and didn't even had a production release yet. v1.0 was released on May 10, 2017 [1], so I doubt it was even on their mind when they started working on the project. [1] https://www.cockroachlabs.com/docs/releases/index.html

Amazing. The whole industry has come quite a ways since 2017!

Re: How Discord Stores Billions of Messages (2017)

#180
post #82

Earlier quoted context omitted.

Honestly, I find discord super frustrating. Can't have multiple chats open at the same time, can't close the right rail, etc. It's UX is subpar in almost every way that matters to me. I use it because _everyone_ uses it, not because I want to.

I can't think of many reasons Discord "ate the whole market" besides smart marketing, honestly. It does audio rooms incredibly well, but everything else (even their developer support team) is just terrible.

Because most people used ventrilo, teamspeak or mumble, which required running or paying for a server, and had limited chat/ social network functionality, and skype which was just terrible. It came at the right time as gaming hit its stride in the mainstream when people needed a place for 'local' communities that was basically frictionless and discord was there to capture that audience because there was basically nothing else.

All the features afterwards are mostly just them throwing stuff at the wall and seeing what sticks.

Post reply on HN