Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Stores Billions of Messages (2017)

blog.discord.com

Post reply on HN
Print view View original

321–330 of 377 posts

Re: How Discord Stores Billions of Messages (2017)

#321

Post by jfoutz »

enlyth wrote: post #314
t0mas88 wrote: post #312

Earlier quoted context omitted.

Trading is for multiple reasons ideal for this, one is that total global ordering is a key feature (and requirement) of the domain so this "1 fast big server" thing is good. It is also quite widely known that several of the big exchange operate this model, a single sequencer application and then using multicast to transmit the outcomes of what it sees. The other thing that is helping a lot here compared to Discord: T…

What about cryptocurrency trading which goes on continuously for 24 hours a day?

you can handle the load or not, right? A built in maintenance window is super nice, but servers crash all the time. So, that's a problem, or you've got a system in place. if you can handle failover, you've got free maintenance windows anyway, so it seems not any more difficult?

Re: How Discord Stores Billions of Messages (2017)

#322

Post by throwaddzuzxd »

dancemethis wrote: post #279
umvi wrote: post #23

Discord is so good. I just can't imagine it can stay this good forever. My fear is that eventually it will be bought out and aggressively monetized.

It's not good. It's user hostile software.

As someone who loves Discord, I'm curious why you'd think that, can you elaborate how it's user hostile?

Re: How Discord Stores Billions of Messages (2017)

#323

Post by throwdbaaway »

legerdemain wrote: post #131

We took a big bet on Cassandra, and then on an opinionated wrapper around Cassandra at $PASTJOB. The use case was a text search engine for syslog-type stuff. The product we built using Cassandra was widely known as our buggiest and least maintainable, and it died a merciful death after several years of being inflicted on customers. We didn't have a good handle on the exact perf implications of different values of rea…

> Maybe the situation has changed since 2016. In my experience with several employers since then, it seems like every enterprise architect fell in love with Cassandra around 2014-2015 and then had a long, painful, protracted breakup.

I think 2012-2014 was peak marketing from DataStax. There would be some new major feature with every new blog post, and it would mostly never work as expected. Between 2017 and now, things have settled down.

Re: How Discord Stores Billions of Messages (2017)

#324

Post by lmilcin »

jfoutz wrote: post #321
enlyth wrote: post #314

Earlier quoted context omitted.

What about cryptocurrency trading which goes on continuously for 24 hours a day?

you can handle the load or not, right? A built in maintenance window is super nice, but servers crash all the time. So, that's a problem, or you've got a system in place. if you can handle failover, you've got free maintenance windows anyway, so it seems not any more difficult?

> but servers crash all the time

This is both wise and stupid at the same time.

It is wise if you mean "be ready for servers to crash at any time by thinking they are going to crash at the worst possible moment".

But it is stupid, because people think they need massive parallel deployments just because servers will be constantly crashing and it is just not true. The cost they pay is in having couple of times more nodes than they really need to have if they got their focus right (making the application efficient first, scalable later)

The reality is, servers do not crash. At least not the kind of hardware I am working on.

I have been responsible for keeping communication with a stock exchange for like 3 years in one of my past jobs and during that time we haven't lost a single packet.

And aside from some massive parallel loads which used tens of thousands of nodes and aside from one time my server room boiled over due to failed AC (and no environmental monitoring) I never had a server crash on me for the past 20 years.

So you can reasonably assume that your servers will be functioning properly (if you bought quality) and it kinda helps a lot at design stage.

Re: How Discord Stores Billions of Messages (2017)

#325

Post by lmilcin »

throwdbaaway wrote: post #276
lmilcin wrote: post #262

Earlier quoted context omitted.

Data doesn't come or leave batched. This is just internal mechanism. Think in term of Discord, their database probably already queues and batches writes. Or maybe they could decide to fetch details of multiple users with a single query by noticing there are 10k concurrent asks for user details. So why have 10k queries when you could have 10 queries for 1k user objects? If you complain that my process is different bec…

Of course, cassandra/mongodb/etc can perform their own batching when writing to the commit log, and can also benefit from write combining by not flushing out the dirty data immediately. That's besides the point. Your use case allows you to perform batching for writes at the *application layer*, while discord's use case doesn't.

I don't see why discord's case can't use same tricks. If they have a lot of stuff happening at the same time and their application is relatively simple (from the point of view of number of different types of operation it performs) at any point in time it is bound to have many cases of the same operation being performed.

Then it is just a case of structuring your application properly.

Most applications are immediately broken, by design, by having a thread dedicated to the request/response pair. It then becomes difficult to have parts of that processing from different threads be selected and processed together to take benefit of amortizing costs.

The alternative I am using is funneling all requests into a single pipeline and having that pipeline split into stages distributed over CPU cores. So it comes in (by way of Kafka or REST call, etc.), it is queued, it goes to CPU core #1, gets some processing there, then moves to CPU core #2, gets some other processing there, gets published to CPU core #3 and so on.

Now, each of these components can work on huge number of tasks at the same time. For example when the step is to enrich the data, it might be necessary to shoot a message to another REST service and wait for response. During that time the component picks up other items to do the same.

As you see, this architecture practically begs to use batching and amortize costs.

Re: How Discord Stores Billions of Messages (2017)

#327

Post by gpderetta »

lmilcin wrote: post #270
fao_ wrote: post #255

Earlier quoted context omitted.

Are you not sure that financial data "with hundreds of fields" is more complex than chat data which has a relatively linear threading and only a handful of fields?

Actually, our threading is quite simple. There is exactly as many threads (that do anything) as CPU cores.

I think parent meant threading as in message threads.

Re: How Discord Stores Billions of Messages (2017)

#328

Post by anonymoushn »

enlyth wrote: post #314
t0mas88 wrote: post #312

Earlier quoted context omitted.

Trading is for multiple reasons ideal for this, one is that total global ordering is a key feature (and requirement) of the domain so this "1 fast big server" thing is good. It is also quite widely known that several of the big exchange operate this model, a single sequencer application and then using multicast to transmit the outcomes of what it sees. The other thing that is helping a lot here compared to Discord: T…

What about cryptocurrency trading which goes on continuously for 24 hours a day?

Cryptocurrency venues handle extremely low throughput and, with maybe one exception, regularly go down for hours.

Re: How Discord Stores Billions of Messages (2017)

#329

Post by jfoutz »

lmilcin wrote: post #324
jfoutz wrote: post #321

Earlier quoted context omitted.

you can handle the load or not, right? A built in maintenance window is super nice, but servers crash all the time. So, that's a problem, or you've got a system in place. if you can handle failover, you've got free maintenance windows anyway, so it seems not any more difficult?

> but servers crash all the time This is both wise and stupid at the same time. It is wise if you mean "be ready for servers to crash at any time by thinking they are going to crash at the worst possible moment". But it is stupid, because people think they need massive parallel deployments just because servers will be constantly crashing and it is just not true. The cost they pay is in having couple of times more nod…

> 3 years in one of my past jobs and during that time we haven't lost a single packet.

I. uh. I. That's amazing. Fuck, that's incredible. I'm trying to envision how I could track the success of each packet ever sent over the last 3 years. Just the metrics around that seems overwhelming. Jesus, that makes me question a lot about my personal skills.

Fucking 'A man! Not losing a single packet in 3 years is. Shit I don't have the words. Congratulations! That is SICK (old American slang for good, very good). I can't even figure out how you would track that.

Yeah, I'll delete my comments if you'd like. Wow. not a packet in 3 years. That's just. I don't know what that is. That's making me question my choice of profession. Damn. That's sooooo good. Holy shit that's crazy good.

Re: How Discord Stores Billions of Messages (2017)

#330

Post by anonymoushn »

unchar1 wrote: post #320
naikrovek wrote: post #303

Earlier quoted context omitted.

Go and Rust won't help there, but Zig might.

I can understand that Go has a GC so it (might) be slower. But wouldn't Rust and Zig have about the same performance?

Maybe if you use nostd. I evaluated a bunch of Rust libraries for some server software, but I could not use any of them because they pervasively assume that it is ok to make syscalls to allocate memory. If you'd like to write software that makes few syscalls in the steady state, you can do it in rust, but you can't use libraries. Or String or Vec, I guess.
Post reply on HN