Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Stores Billions of Messages (2017)

blog.discord.com

Post reply on HN
Print view View original

201–210 of 377 posts

Re: How Discord Stores Billions of Messages (2017)

#201

Post by ThatMedicIsASpy »

Kaze404 wrote: post #82
RHSeeger wrote: post #36

Earlier quoted context omitted.

Honestly, I find discord super frustrating. Can't have multiple chats open at the same time, can't close the right rail, etc. It's UX is subpar in almost every way that matters to me. I use it because _everyone_ uses it, not because I want to.

I can't think of many reasons Discord "ate the whole market" besides smart marketing, honestly. It does audio rooms incredibly well, but everything else (even their developer support team) is just terrible.

The audio quality of discord is incredibly bad when I compare it to teamspeak3 (opus music 10)

Re: How Discord Stores Billions of Messages (2017)

#202

Post by aeyes »

andrewstuart wrote: post #114

I'd first reach for Postgres to do this. Anyone have any idea how Postgres would stack up in a similar challenge?

I see several issues:

- No out of the box horizontal sharding, according to the post they had 4TB (compressed) data in the cluster in 2017. Looking at their growth I think it is safe to assume that today they would have >50TB which can't be done on a single node. You could use Citus but this is not exactly vanilla Postgres anymore. For such a simple data model wasting time implementing your own sharding solution and (more importantly) shard migration makes no sense.

- Discord is storing text data, in Postgres this will be stored in TOAST tables which has some drawbacks.

- Their workload is mostly inserts, almost no updates. Vacuum only operates on complete tables so you would wast I/O and CPU processing data which you don't even touch. You can partition tables but it's a manual process and you have to make compromises. In 2017, Postgres partitioning still had many performance drawbacks.

- No out of the box redundancy.

- Once your data doesn't fit in memory, Postgres performance becomes unpredictable.

Personally I would have chosen ElasticSearch for this project.

Re: How Discord Stores Billions of Messages (2017)

#203

Post by jjice »

objektif wrote: post #188
latchkey wrote: post #12

Earlier quoted context omitted.

Unique images or just copied from elsewhere?

How does that matter? Do they keep track of all images on the internet?

I don't know much about image de-deuplication, but maybe they can get some sort of fingerprint/hash for an image, see if they already have it, and then serve that already existing image.

I'd imagine a hash like SHA256 would be tricky because if that image was compressed an additional time at all throughout it's internet journey, then we'd get a different resulting hash, but maybe there is an effective way to fingerprint images. I have a utility on my machine (czkawka maybe?) that does really good image de-duplication with what seemed like a common algorithm (based on a quick look at the source).

No idea though, just spit balling.

Re: How Discord Stores Billions of Messages (2017)

#205

Post by rcxdude »

Scaless wrote: post #41
dvt wrote: post #10

Earlier quoted context omitted.

> ventrilo/mumble/teamspeak To be fair, Mumble is FOSS, and Ventrilo and Teamspeak have literally not iterated since 2005. Discord is pretty mediocre software (remember when they accidentally allowed iframe XSS RCE attacks? A very amateurish mistake), but the incumbents were an absolute dumpster fire.

For Mumble in particular, the devs had their heads in the clouds for so long that it is no surprise that it is no longer relevant. If you had a mic that had issues in any way (buzzing, volume, balance), "The Wizard" and "AGC" were supposed to fix it for you. Do not fret little one, for you do not need nor want to manually fiddle with settings, The Wizard will make everything right [1]! The pivotal feature that was th…

That issue is pretty amazing, especially the developer straight-up accusing those saying 'this is a killer feature stopping us from moving to mumble' of lying (considering the substantial overlap between the features of the various options competing in the space at the time this is really obviously the kind of thing which could be a deal-breaker: I remember the arguments over which option to use based on far less substantial differences than this).

Re: How Discord Stores Billions of Messages (2017)

#206

Post by imadeamistake »

PeterCorless wrote: post #181
legerdemain wrote: post #131

We took a big bet on Cassandra, and then on an opinionated wrapper around Cassandra at $PASTJOB. The use case was a text search engine for syslog-type stuff. The product we built using Cassandra was widely known as our buggiest and least maintainable, and it died a merciful death after several years of being inflicted on customers. We didn't have a good handle on the exact perf implications of different values of rea…

We just benchmarked Cassandra 4.0, which is brand-spanking new. The good news: C 4.0 is a far better performing database than C 3.11. The new GCs definitely get rid of the long tail latency nightmares: https://www.scylladb.com/2021/08/19/cassandra-4-0-vs-cassand... However, we also compared it to Scylla's latest release, and though C 4 is better*, you can still find other CQL-compatible databases that outperform it.…

I like Scylla. I've been working with it for a while now, and it's a good alternative for transactional loads. It's a hell of a lot faster than Cassandra, and much much much much cheaper than DynamoDB. Cassandra has always felt like improvements came in fits and starts. I work at a Fortune 50 company, and Amazon quoted us ~$2-3 million a year to run our load on Dynamo (we were paying $350k/year for Aurora). With Scylla, we're looking at > $100k to get an order of magnitude better performance and a nice stable system that doesn't wake me up in the middle of the fucking night.

I was following Scylla since the beginning (only because I like their mascot), and it's actually sort of interesting to see what's going on with the company. I've spent the past few years designing things where transactional systems backed by Cassandra. This is the first time I've been able to use Scylla on someone else's dime, though. The unpleasantly big company I'm at right now is looking to replace a bunch of infrastructure with ScyllaDB (Couchbase, Cassandra, Elasticsearch, DynamoDB). It's catching on for sure, but it still doesn't return any results when I search Dice. It looks like Discord is hiring, though...

Re: How Discord Stores Billions of Messages (2017)

#207

Post by rcxdude »

sascha_sl wrote: post #39
tester756 wrote: post #34

Earlier quoted context omitted.

>These days? Well, most server providers have some sort of basic flood mitigations in place now, and even more advanced protection has become affordable. Hmm I didn't meant your server being DDoSd, but you being DDoS (but probably that's what you meant with Skype P2P example?)

Well yeah. Though Vent, Teamspeak and Mumble never had these issues (if you could trust the server admin). Skype (at the time, no idea now) was a very shoddily written piece of software. It was trivial to query the IP of any online user, even if they were not on your contact list or appearing offline. You had to use a VPN or carefully conceal your Skype ID, I did work with a somewhat popular live streamer back then (…

Skype wasn't shoddy per se, but it was extremely aggressively peer-to-peer, which resulted in exactly this kind of exposure of connection information.

Re: How Discord Stores Billions of Messages (2017)

#208

Post by imadeamistake »

trashcan wrote: post #193
objektif wrote: post #183

Earlier quoted context omitted.

How much is very very high pay?

I don't think it would be appropriate for me to say very specifically, but I suspect about double what a software engineer with the same amount of experience would earn.

Data is big $$$. Slap a couple of NoSQL databases and Spark on your resume and watch the money roll in. DBAs are disappearing with managed services, though.

Re: How Discord Stores Billions of Messages (2017)

#210

Post by PeterCorless »

nonbirithm wrote: post #74
kccqzy wrote: post #30

Earlier quoted context omitted.

I've also noticed that in a lot of tech-related social circles people are increasingly choosing Discord over Slack. That's a trend I totally didn't expect: at least until a few years ago it was clear that Discord was for gamers and Slack was for work and everyone else. That changed quickly. Impressive indeed!

They also pivoted their marketing message away from being "for gamers" and towards anyone who wanted "a place to hang out," like developer groups or high schoolers.

It's the TikTok effect.
Post reply on HN