We use scylla for our IoT stream, bucket per day, with a date index for second resolution data. The current day is a hot spot of, but we throw that in redis. It's running one of the largest re insurance providers IoT deployments.
How Discord Stores Billions of Messages (2017)
261–270 of 377 posts
Re: How Discord Stores Billions of Messages (2017)
#262Earlier quoted context omitted.
The latency is a complex topic. In order to be able to run so much stuff on MongoDB, we almost never run single queries to the database. If I fetch or insert trade data, I probably run a query for 10 thousand trades at the same time. So what happens is, as data comes from multiple directions it is being batched (for example 1-10 thousand at a time), split into groups that can be processed together in a roughly simila…
I thought you are doing millions qps with a 3 nodes mongodb cluster, from the top level comment. That would be impressive. By batching 1-10 thousands records at a time, your use case is very different from discord, which needs to deliver individual messages as fast as possible.
Data doesn't come or leave batched. This is just internal mechanism.
Think in term of Discord, their database probably already queues and batches writes. Or maybe they could decide to fetch details of multiple users with a single query by noticing there are 10k concurrent asks for user details. So why have 10k queries when you could have 10 queries for 1k user objects?
If you complain that my process is different because I refuse to run it inefficiently when I can spot an occasion to optimize then yes, it is different.
Re: How Discord Stores Billions of Messages (2017)
#263Two horrible choices of databases... first they used MongoDB then they migrated to Cassandra. I've used tons of databases [1] in production and those two are the worst. [1] I've used RethinkDB, Postgres, MongoDB, MySQL, Cassandra, CockroachDB, TimescaleDB, SSDB, and others
What would you have chosen?
Re: How Discord Stores Billions of Messages (2017)
#264Two horrible choices of databases... first they used MongoDB then they migrated to Cassandra. I've used tons of databases [1] in production and those two are the worst. [1] I've used RethinkDB, Postgres, MongoDB, MySQL, Cassandra, CockroachDB, TimescaleDB, SSDB, and others
Cassandra has been successfully deployed at many companies. Would you care to provide some insight into your experience and why you consider it one of the worst?
Here's a bug I ran into a few years ago:
https://datastax-oss.atlassian.net/browse/PYTHON-891
With all the issues I encountered using in prod, it gave the impression of an overly complicated key/value store.
Re: How Discord Stores Billions of Messages (2017)
#265Could somebody help me understand the reasoning behind `timestamp = snowflake_id >> 22` Thanks :)
IIRC a snowflake ID's first few bits (or characters?) are by definition a unix timestamp.
Re: How Discord Stores Billions of Messages (2017)
#266Two horrible choices of databases... first they used MongoDB then they migrated to Cassandra. I've used tons of databases [1] in production and those two are the worst. [1] I've used RethinkDB, Postgres, MongoDB, MySQL, Cassandra, CockroachDB, TimescaleDB, SSDB, and others
What would you have chosen?
An in-house data service built with RocksDB/LevelDB. [1]
Re: How Discord Stores Billions of Messages (2017)
#267Earlier quoted context omitted.
I don't see how that's relevant? I don't even prefer those over Discord, but I don't think it's enough of an improvement to warrant the market share it has now.
So what do you think happened? That people were manipulated in to using discord? Or that they don't know what the alternatives are? Everyone I have spoken to loves discord and thinks it is one of the best programs they have. It's only a select group of hacker news style users who complain about minute details the average person does not care about. I know it's hard for most people on this site to understand but the a…
What I think happened, as someone who's been using Discord daily since 2015, is that they came up with a slightly better product than the alternatives, spent enough in marketing (to gamers specifically) to convince investors that it was a platform worth investing in, and only then slowly started improving their faulty software.
To say people were manipulated into using Discord is obviously not true, but it's also disingenuous to deny the massive amount of marketing Discord pushed back when it first started, not only in advertisement but just branding in general.
I'm not going to address the latter part of your comment because I don't understand what you're trying to say. I'm of the belief that I'm allowed to voice the legitimate issues I have with the software that impact not only myself and other developers but users in general.
Re: How Discord Stores Billions of Messages (2017)
#268Earlier quoted context omitted.
I see several issues: - No out of the box horizontal sharding, according to the post they had 4TB (compressed) data in the cluster in 2017. Looking at their growth I think it is safe to assume that today they would have >50TB which can't be done on a single node. You could use Citus but this is not exactly vanilla Postgres anymore. For such a simple data model wasting time implementing your own sharding solution and…
Why would the text data be stored in TOAST? My understanding was PG only uses TOAST when the data is too large to fit in the row, and since PG compresses data before inserting wouldn't user messages be fine?
I think Discord messages can be unbounded in size
Re: How Discord Stores Billions of Messages (2017)
#269Earlier quoted context omitted.
> you can run mumble on any random Linux box in your house That seems easy to you. That would be easy for me too and most likely 90% of the people on HackerNews. But the average person doesn't have a "random Linux box" in their house. Most people don't even know what Linux is. Most people would be overwhelmed just looking for the terminal emulator on their computer, before they even typed a command into it. Most peop…
I find it astounding that people here can not even grasp the concept of why Discord is popular. I am perfectly capable of hosting my own server and doing everything manually. But it is clear as day why discord wiped out the competition while most of the comments here seem dazed by the fact and are left wondering why people don't just use IRC. It's no wonder so many projects and FOSS tools fail to gain large userbases…
This is an important point. I write that with no disrespect to all those FOSS developers who have devoted themselves to the work of creating new, interesting and useful things. But the fact is that usability, like intuition, is usually a very subjective matter. That's why QA and UAT were such an integral part of traditional software development, and why community engagement needs to be a two-way street.
Re: How Discord Stores Billions of Messages (2017)
#270Earlier quoted context omitted.
How many simultaneous queries of that nature can the system handle?
Are you not sure that financial data "with hundreds of fields" is more complex than chat data which has a relatively linear threading and only a handful of fields?
Actually, our threading is quite simple.
There is exactly as many threads (that do anything) as CPU cores.