Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Stores Billions of Messages (2017)

blog.discord.com

Post reply on HN
Print view View original

291–300 of 377 posts

Re: How Discord Stores Billions of Messages (2017)

#291

Post by polishdude20 »

lmilcin wrote: post #245
kevinsundar wrote: post #235

Earlier quoted context omitted.

Nice that's a pretty decent size, curious on the latency still. Thats the primary problem for a real time chat app. Complexity of data and running business rules on it is not a data store problem though, that's a compute problem. It's highly parallelizable and compute is cheap. For reference, my team runs transformations on about 1 PB of (uncompressed) data per day with 3 spark clusters, each with 50 nodes. We've got…

The latency is a complex topic. In order to be able to run so much stuff on MongoDB, we almost never run single queries to the database. If I fetch or insert trade data, I probably run a query for 10 thousand trades at the same time. So what happens is, as data comes from multiple directions it is being batched (for example 1-10 thousand at a time), split into groups that can be processed together in a roughly simila…

This just gives me Factorio flashbacks.

Re: How Discord Stores Billions of Messages (2017)

#292

Post by anaganisk »

welder wrote: post #226

Two horrible choices of databases... first they used MongoDB then they migrated to Cassandra. I've used tons of databases [1] in production and those two are the worst. [1] I've used RethinkDB, Postgres, MongoDB, MySQL, Cassandra, CockroachDB, TimescaleDB, SSDB, and others

Im not questioning your intellect, but each database has its own use case for usage. You might be expecting wrong things from Mongo or cassandra. Using gazillion databases for wrong use cases doesnt mean nothing. From your child comment, you cooked up an in house solution, it may be best suited for you. But for others it would be horrible too. If Cassandra is suited for Facebook, I think the problem is with you making a choice to use it for something its not suited for rather than the database itself.

Re: How Discord Stores Billions of Messages (2017)

#293

Post by welder »

anaganisk wrote: post #292
welder wrote: post #226

Two horrible choices of databases... first they used MongoDB then they migrated to Cassandra. I've used tons of databases [1] in production and those two are the worst. [1] I've used RethinkDB, Postgres, MongoDB, MySQL, Cassandra, CockroachDB, TimescaleDB, SSDB, and others

Im not questioning your intellect, but each database has its own use case for usage. You might be expecting wrong things from Mongo or cassandra. Using gazillion databases for wrong use cases doesnt mean nothing. From your child comment, you cooked up an in house solution, it may be best suited for you. But for others it would be horrible too. If Cassandra is suited for Facebook, I think the problem is with you makin…

Yes, you're right.

Re: How Discord Stores Billions of Messages (2017)

#294

Post by tsjq »

RHSeeger wrote: post #36
tester756 wrote: post #2

Discord had like $300M invested and they created unparalleled piece of software that ate whole market, damn. One of the most impressive softwares that I've seen and use after years of using ventrilo/mumble/teamspeak.

Honestly, I find discord super frustrating. Can't have multiple chats open at the same time, can't close the right rail, etc. It's UX is subpar in almost every way that matters to me. I use it because _everyone_ uses it, not because I want to.

discord is maha complicated UX. close to nightmare

Re: How Discord Stores Billions of Messages (2017)

#296

Post by saagarjha »

gunapologist99 wrote: post #195
sa1 wrote: post #105

Earlier quoted context omitted.

https://twitter.com/discord/status/1229357198918197248?lang=...

For reference, this tweet from @discord says: "All 3rd party apps or client modifiers are against our ToS, and the use of them can result in your account being disabled. I don't recommend using them."

It's not in the ToS, as the people in Twitter point out.

Re: How Discord Stores Billions of Messages (2017)

#297

Post by the_arun »

lmilcin wrote: post #233
dpedu wrote: post #230

Earlier quoted context omitted.

Sounds interesting. Does this "we" have any writings about this?

I have a plan to write a book on how to write reactive applications like that. Mostly collection of observations, tips, tricks, patterns for reactive composition, some very MongoDB specific solutions, etc. Not sure how many people would be interested. Reactor has quite steep learning curve but also very little literature on how to use for anything non-trivial. The aim is not just enable good throughput, but also achi…

Do you have a blog? So that we can learn those tips soon? :)

Re: How Discord Stores Billions of Messages (2017)

#298

Post by Aeolun »

fao_ wrote: post #253
lmilcin wrote: post #245

Earlier quoted context omitted.

The latency is a complex topic. In order to be able to run so much stuff on MongoDB, we almost never run single queries to the database. If I fetch or insert trade data, I probably run a query for 10 thousand trades at the same time. So what happens is, as data comes from multiple directions it is being batched (for example 1-10 thousand at a time), split into groups that can be processed together in a roughly simila…

I'm not sure why you're being downvoted when you're a domain expert talking about your craft? People have weird hangups on hacker news, it seems

Something about the tone of the messages rubs me entirely the wrong way.

Re: How Discord Stores Billions of Messages (2017)

#299

Post by Aeolun »

tester756 wrote: post #2

Discord had like $300M invested and they created unparalleled piece of software that ate whole market, damn. One of the most impressive softwares that I've seen and use after years of using ventrilo/mumble/teamspeak.

I don’t know about your definitions, but 300M is a fuckton of money to me.

Re: How Discord Stores Billions of Messages (2017)

#300

Post by cityzen »

PeterCorless wrote: post #280
cityzen wrote: post #254

Earlier quoted context omitted.

I'd love to hear how Dynamo would end up being $2-3 million a year They sure do a great job of convincing people that it's cheap so I'm curious where the cost seems to blow up?

If you are doing north of a million ops on DynamoDB you can quickly run into the $2-3 million a year range. In this 2018 benchmark, we were able to calculate that a sustained, provisioned of only 160k write ops / 80k read ops for DynamoDB would cost >$500k per year: https://www.scylladb.com/2018/12/13/scylla-vs-amazon-dynamod... That was a few years ago. These days, according to our most current pricing you could do…

Thanks for the response, Peter. I may shoot you an email with some additional questions.
Post reply on HN