How Discord Stores Billions of Messages (2017) • Hacker News Board

Re: How Discord Stores Billions of Messages (2017)

#211

Post by jamesdwilson » Tue, Aug 24, 2021, 11:45 PM UTC

losteric wrote: post #9

jamesdwilson wrote: post #4

With privacy concerns, companies should be shamed for storing billions of messages.

That is a fair concern. However, as a customer - searching my message history is a desirable feature. I would rather see meaningful individual and corporate accountability for privacy breaches. The threat of jail and/or 100MM's in fines should motivate better data handling.

We've seen this happen over and over again:

- a company amasses a large trove of sensitive information

- it is exposed to adversaries or political enemies

- the information is used against the people

Re: How Discord Stores Billions of Messages (2017)

#212

Post by jamesdwilson » Tue, Aug 24, 2021, 11:46 PM UTC

colesantiago wrote: post #21

jamesdwilson wrote: post #4

With privacy concerns, companies should be shamed for storing billions of messages.

> With privacy concerns, companies should be shamed for storing billions of messages. should we shame ycombinator for storing the messages, accounts and comments on hacker news then? I am still unable to delete my account here even though the CCPA and the GDPR exists. But here we are.

yes. whataboutism.

Re: How Discord Stores Billions of Messages (2017)

#213

Post by PHGamer » Tue, Aug 24, 2021, 11:47 PM UTC

tester756 wrote: post #2

Discord had like $300M invested and they created unparalleled piece of software that ate whole market, damn. One of the most impressive softwares that I've seen and use after years of using ventrilo/mumble/teamspeak.

its just slack for gaming. the ui is ripped off as suck. it is better than ventrilo but its not like they are that much better, just they realized a good concept and took it.

Re: How Discord Stores Billions of Messages (2017)

#214

Post by PeterCorless » Tue, Aug 24, 2021, 11:50 PM UTC

imadeamistake wrote: post #208

trashcan wrote: post #193

Earlier quoted context omitted.

I don't think it would be appropriate for me to say very specifically, but I suspect about double what a software engineer with the same amount of experience would earn.

Data is big $$$. Slap a couple of NoSQL databases and Spark on your resume and watch the money roll in. DBAs are disappearing with managed services, though.

If you have been branding yourself as a DBA, time to lift and shift to "DevOps."

Re: How Discord Stores Billions of Messages (2017)

#215

Post by weikju » Tue, Aug 24, 2021, 11:52 PM UTC

ThatMedicIsASpy wrote: post #199

Dylan16807 wrote: post #194

Earlier quoted context omitted.

It would help if discord would stop threatening to ban people for using third party clients.

I've ditched the client for web only with custom css. Also allows me to block every kind of tracking (opened programs). I only use it as a chat client and still run a TS3 server because discords audio is just garbage

Would it be possible to share this custom CSS please? Also, what tracking/how are you stopping it?

Re: How Discord Stores Billions of Messages (2017)

#216

Post by PeterCorless » Tue, Aug 24, 2021, 11:55 PM UTC

jjice wrote: post #203

objektif wrote: post #188

Earlier quoted context omitted.

How does that matter? Do they keep track of all images on the internet?

I don't know much about image de-deuplication, but maybe they can get some sort of fingerprint/hash for an image, see if they already have it, and then serve that already existing image. I'd imagine a hash like SHA256 would be tricky because if that image was compressed an additional time at all throughout it's internet journey, then we'd get a different resulting hash, but maybe there is an effective way to fingerpr…

Yes. There are ways to group images that seem to be the same. TinEye and Google image search do that. So you'd have a collection of related hashes that equal "Bob's prom photo where he looks like a goofer."

Re: How Discord Stores Billions of Messages (2017)

#217

Post by dragonfax » Tue, Aug 24, 2021, 11:56 PM UTC

KKV databases (Cassandra and DynamoDB are good examples) have a common problem with hotspots or "hot partitions". The most common mistake is to use a timestamp of any kind in the range (cluster) column. Then, whatever partition represents "today" or "this hour" ends up being the hot partition.

The article mentions hot partitions becomming a problem with max partition size, but they're also a problem with scalability. Say, if your writing a very high throughput of logs into the table (contrived example), then your bottlenecked by the rate at which you can write to one partition.

Adding the bucket id (say, the current day or hour), is a common solution, and solves the max partition size issue, but not the scalability issue of hot partitions.

Re: How Discord Stores Billions of Messages (2017)

#218

Post by PeterCorless » Tue, Aug 24, 2021, 11:58 PM UTC

Ansil849 wrote: post #71

Nothing about any sort of encryption.

Scylla, Discord's replacement for Cassandra, supports both encryption in transit (server-to-server within the cluster; client-to-server) and encyption at rest for stored data.

Re: How Discord Stores Billions of Messages (2017)

#219

Post by PeterCorless » Wed, Aug 25, 2021, 12:07 AM UTC

Quarrelsome wrote: post #108

do they ever explain what their "anti-entropy" processes are for and what they do?

There are a few:

1. Hinted Handoffs - if a node has a transient failure, the other nodes store up messages, like your buddy might take notes in class if you had to go to the bathroom. They'd pass you those notes when you got back. "Here's what you missed." When the node comes back online it processes all new operations and works through its backlog of hinted handoffs to get caught up. Because of the backlog it creates, hinted handoffs are only stacked up for a few hours. If the node never comes back up, or comes back after that window...

2. Repairs - in an eventually-consistent database you might miss an update or two over time. Or maybe you're a replacement node that has to fill in for a failed node. The replacement will get streamed data from the other replicas to get it started, or you might restore sstables from a backup, but then you should run a repair job to make sure all your replicas are properly in sync.

(That's my understanding. Let me know if that sounds correct from the hands-on experts.)

Re: How Discord Stores Billions of Messages (2017)

#220

Post by objektif » Wed, Aug 25, 2021, 12:08 AM UTC

PeterCorless wrote: post #216

jjice wrote: post #203

Earlier quoted context omitted.

I don't know much about image de-deuplication, but maybe they can get some sort of fingerprint/hash for an image, see if they already have it, and then serve that already existing image. I'd imagine a hash like SHA256 would be tricky because if that image was compressed an additional time at all throughout it's internet journey, then we'd get a different resulting hash, but maybe there is an effective way to fingerpr…

Yes. There are ways to group images that seem to be the same. TinEye and Google image search do that. So you'd have a collection of related hashes that equal "Bob's prom photo where he looks like a goofer."

Yes definitely I have seen it work in action but you cant just tell a user "here use this smaller and more pixelated version of your image that we think is kind of similar".