Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Stores Billions of Messages (2017)

blog.discord.com

Post reply on HN
Print view View original

351–360 of 377 posts

Re: How Discord Stores Billions of Messages (2017)

#351

Post by aeyes »

emptysea wrote: post #277
daveidol wrote: post #268

Earlier quoted context omitted.

I think Discord messages can be unbounded in size

Interesting, trying it out on Discord the default max msg size is 2000 chars and with Discord Nitro the max is raised to 4000 chars. Testing with Postgres, a 2000 char random sequence doesn't result in TOASTing, but a 4000 random sequence does get TOASTed And for kicks, 4000 chars that aren't random compress well enough that they don't end up in TOAST.

Default for TOAST is 2KB and how many chars you can fit in this space depends on encoding, could be as low as 1000 uncompressed.

I think this limit can be adjusted, if you know that your message limit is 4000 chars its definitely a good idea.

Re: How Discord Stores Billions of Messages (2017)

#352

Post by fao_ »

lmilcin wrote: post #270
fao_ wrote: post #255

Earlier quoted context omitted.

Are you not sure that financial data "with hundreds of fields" is more complex than chat data which has a relatively linear threading and only a handful of fields?

Actually, our threading is quite simple. There is exactly as many threads (that do anything) as CPU cores.

I meant threading in as much as the connections or links between message data

All discord needs to store is:

{ channel_id, message_id, user_id, content, reply_to, datetime }

these days they added an extra thread_id field, sure. But the data itself is blisteringly uncomplex and there is only a single way to display it (ordered by time, i.e. the 'thread')

Re: How Discord Stores Billions of Messages (2017)

#353

Post by fao_ »

Aeolun wrote: post #338
bottled_poe wrote: post #310

Earlier quoted context omitted.

It reads like justified opinions from experience. Not seeing much emotional tone in there.

I dunno, sounds very ‘I am very smart’ to me. They may be right or they may not, but both solutions sound workable to me. Don’t let perfect be the enemy of good, and all that. There’s enough utter garbage around to shit on.

I think you're reading into it. They are stating that the solution in the post was overengineered, and describing an alternate solution that doesn't require as much abstraction or resources, but is manageable for data with a much higher dimensional structure

The fact that you read that as "I am very smart" and that that was a reason to downvote the post, tells more about you than it does the person you're supposedly describing.

Re: How Discord Stores Billions of Messages (2017)

#354

Post by lmilcin »

bob1029 wrote: post #345
lmilcin wrote: post #341

Earlier quoted context omitted.

Yes, that is my experience. Another super important thing to remember is that main goal of this is to have super simple code and very simple but rock solid guarantees. The main benefit is writing application code that is simple, easy to understand and simple to prove it works correctly, enabled by reliable infrastructure. When you are not focusing on various ridiculous technologies that each require PhDs to understan…

> When you are not focusing on various ridiculous technologies that each require PhDs to understand well, you can focus on your application stack, domain modeling, etc. to make it even more reliable. This is 100% our philosophy. I honestly don't understand why all high-stakes software isn't developed in the same way that we build these trading/data systems. I think this is the boundary between "engineering" and "art"…

Every time I write something like "Yes, you really can write reliable application. No, if it breaks you can't blame everybody and the universe around you. You made a mistake and you need to figure out how this happened and how to prevent it from happening in the future." I just get downvoted to hell.

I suspect in large part it is because when people fail at something they feel a need to find some external explanation of it. And it is all too easy when "business" actually is part of the problem.

The best people I worked with, let's just say I never heard them blaming business for their bugs. They own it, they solve it and they learn from it.

What I am not seeing is people actually have a hard look on what they have done and how they could have avoided the problems.

For example, the single most cause of failed projects I have seen, by far, is unnecessary complication stemming from easily avoidable technical debt.

Easily avoidable technical debt is something that could have reasonably be predicted at the early stage and solved by just making better decisions. Maybe not split your application to 30 services and then run it on Kubernetes? Maybe rather than separate services, pay attention to have proper modules and APIs within your application and your application will just fit couple of servers? Maybe having function calls rather thancascade of internal network hops is cheap way to get good performance rather than (ignore Amdahl's law and) try to incorporate some exotic database that nobody knows and will have to start learning from scratch?

Then people rewrite these projects and rather than understanding what caused the previous version to fail -- just repeat the same process only with new application stack.

Re: How Discord Stores Billions of Messages (2017)

#355

Post by adamnemecek »

lmilcin wrote: post #343
adamnemecek wrote: post #285

Earlier quoted context omitted.

By 16 billion things you mean 16 billion bytes? If you are talking about physical memory, then no, you can't occupy the entire memory. If you are talking about virtual memory, then you can store more data.

Actually, CPU processes things in words, not bytes. On 64-bit architecture the word is 64 bit or 8 bytes. But there is a lot of things that CPU can do even faster than that, because this limitation only relates to actual instruction execution (and even then there are instructions that can process multiple words at a time).

I know however the only interpretation of the statement above was that he meant bytes since few laptops have 64GB.

Re: How Discord Stores Billions of Messages (2017)

#356

Post by theptip »

aeyes wrote: post #202
andrewstuart wrote: post #114

I'd first reach for Postgres to do this. Anyone have any idea how Postgres would stack up in a similar challenge?

I see several issues: - No out of the box horizontal sharding, according to the post they had 4TB (compressed) data in the cluster in 2017. Looking at their growth I think it is safe to assume that today they would have >50TB which can't be done on a single node. You could use Citus but this is not exactly vanilla Postgres anymore. For such a simple data model wasting time implementing your own sharding solution and…

Do you have any case studies for ElasticSearch that you can recommend, in projects similar to this? Would be very interested in seeing what that option would look like.

Re: How Discord Stores Billions of Messages (2017)

#357

Post by [deleted] »

jfoutz wrote: post #321
enlyth wrote: post #314

Earlier quoted context omitted.

What about cryptocurrency trading which goes on continuously for 24 hours a day?

you can handle the load or not, right? A built in maintenance window is super nice, but servers crash all the time. So, that's a problem, or you've got a system in place. if you can handle failover, you've got free maintenance windows anyway, so it seems not any more difficult?

[deleted]

Re: How Discord Stores Billions of Messages (2017)

#358

Post by ricardobeat »

lmilcin wrote: post #325
throwdbaaway wrote: post #276

Earlier quoted context omitted.

Of course, cassandra/mongodb/etc can perform their own batching when writing to the commit log, and can also benefit from write combining by not flushing out the dirty data immediately. That's besides the point. Your use case allows you to perform batching for writes at the *application layer*, while discord's use case doesn't.

I don't see why discord's case can't use same tricks. If they have a lot of stuff happening at the same time and their application is relatively simple (from the point of view of number of different types of operation it performs) at any point in time it is bound to have many cases of the same operation being performed. Then it is just a case of structuring your application properly. Most applications are immediately…

What you're describing sounds like vanilla async concurrency. I seriously doubt 'most applications' use the one-thread-per-request model at this point in time, most major frameworks are async now. And it's not a silver bullet either, plenty of articles on how single-thread is sometimes a better fit for extremely high-performance apps.

After reading all of you responses, I still don't see how you think your learnings apply to Discord. They would not be able to fit the indexes in memory on MongoDB. They can't batch reads or writes at the application server level (the latency cost for messaging is not acceptable). Millions of queries happen every second, not one-off analytical workloads. It seems these two systems are far enough apart that really there is no meaningful comparison to be made here.

Re: How Discord Stores Billions of Messages (2017)

#359

Post by phailhaus »

fao_ wrote: post #255
phailhaus wrote: post #248

Earlier quoted context omitted.

How many simultaneous queries of that nature can the system handle?

Are you not sure that financial data "with hundreds of fields" is more complex than chat data which has a relatively linear threading and only a handful of fields?

I'm asking about how your system scales to the number of queries, but you seem to be taking every question personally. You seem to really want to make sure everyone knows that you think Discord's problems are easy to solve. I'm not saying Discord is more complicated, but you're not really giving enough information to prove that Discord's problems are a subset of yours.

Do you support more simultaneous queries than Discord?

Re: How Discord Stores Billions of Messages (2017)

#360

Post by imadeamistake »

legerdemain wrote: post #305
imadeamistake wrote: post #208

Earlier quoted context omitted.

Data is big $$$. Slap a couple of NoSQL databases and Spark on your resume and watch the money roll in. DBAs are disappearing with managed services, though.

Yeah, I don't know about "slap." We want you to have deep production experience with these systems. Designing them, deploying them at significant scale, predicting their pitfalls and avoiding them proactively. Diagnosing systemic problems and finding reasonable solutions. If you can't magically put out production fires, on huge high-throughput systems, potentially in the dead of night, we are unlikely to pay you $300…

The intent was to be a little hyperbolic and self-effacing. In terms of competent and capable developers, I think it’s hard to get a better return on your skill set than adding “data” stuff. And honestly I think it’s one of the most critical skill sets that is lacking across the board. So many bit companies have great data engineering teams, but generally other dev teams are left to design their own databases, which is a shit show. And even then, it’s amazing to me how difficult it is for data engineering teams to move from framework to framework without just mapping old solutions onto new technology.

My career has been primarily focused on something like “bringing modern data-driven solutions” to big companies. The one thing that is a constant challenge is that most teams (and leadership) aren’t prepared to handle responsibilities of data engineering and stewardship in transactional, operational systems. I feel like critical responsibility when I come on as a consultant is to impart knowledge about managing their data.

Post reply on HN