How Discord Stores Billions of Messages (2017) • Hacker News Board

Re: How Discord Stores Billions of Messages (2017)

#131

Post by legerdemain » Tue, Aug 24, 2021, 8:28 PM UTC

We took a big bet on Cassandra, and then on an opinionated wrapper around Cassandra at $PASTJOB. The use case was a text search engine for syslog-type stuff.

The product we built using Cassandra was widely known as our buggiest and least maintainable, and it died a merciful death after several years of being inflicted on customers.

We didn't have a good handle on the exact perf implications of different values of read/write replication. Writing product code to handle a range of eventual consistency scenarios is challenging. The memory consumption and duration of compactions and column/node repair jobs is hard to model and accommodate. It's hard to tell what the cluster is doing at any given moment. Our experience with support plans from Datastax was also pretty dismal.

Maybe the situation has changed since 2016. In my experience with several employers since then, it seems like every enterprise architect fell in love with Cassandra around 2014-2015 and then had a long, painful, protracted breakup.

Re: How Discord Stores Billions of Messages (2017)

#132

Post by judge2020 » Tue, Aug 24, 2021, 8:29 PM UTC

mvanaltvorst wrote: post #112

jonshariat wrote: post #57

Earlier quoted context omitted.

Ditto, feels fast and has a lot of features but to me the UX is very confusing.

Personally, I find that Discord feels extremely bloated compared to almost any other program on my computer. Probably has to do with the fact that Discord has still not released a native ARM version of its client for the Apple M1. Nearly a year after its release there really is no excuse for any Electron application to stick to x86_64+Rosetta 2. Discord is literally the only x86 application that is still installed on…

Discord recently downgraded from 64-bit to 32-bit on Windows as well, I imagine there were issues with the interconnects with other, native parts of Discord like screen-sharing that were easily 'solved' by only distributing 32 bit binaries.

Re: How Discord Stores Billions of Messages (2017)

#133

Post by yukinon » Tue, Aug 24, 2021, 8:30 PM UTC

legerdemain wrote: post #131

We took a big bet on Cassandra, and then on an opinionated wrapper around Cassandra at $PASTJOB. The use case was a text search engine for syslog-type stuff. The product we built using Cassandra was widely known as our buggiest and least maintainable, and it died a merciful death after several years of being inflicted on customers. We didn't have a good handle on the exact perf implications of different values of rea…

If you had to do it again, what would you choose instead?

Re: How Discord Stores Billions of Messages (2017)

#134

Post by Kaze404 » Tue, Aug 24, 2021, 8:33 PM UTC

porb121 wrote: post #86

Kaze404 wrote: post #82

Earlier quoted context omitted.

I can't think of many reasons Discord "ate the whole market" besides smart marketing, honestly. It does audio rooms incredibly well, but everything else (even their developer support team) is just terrible.

uh, how about the fact that vent/mumble/Skype were all awful?

I don't see how that's relevant? I don't even prefer those over Discord, but I don't think it's enough of an improvement to warrant the market share it has now.

Re: How Discord Stores Billions of Messages (2017)

#135

Post by oarsinsync » Tue, Aug 24, 2021, 8:33 PM UTC

porb121 wrote: post #86

Kaze404 wrote: post #82

Earlier quoted context omitted.

I can't think of many reasons Discord "ate the whole market" besides smart marketing, honestly. It does audio rooms incredibly well, but everything else (even their developer support team) is just terrible.

uh, how about the fact that vent/mumble/Skype were all awful?

Mumble client is awful. Mumble audio quality and latency is fantastic.

Mumble is also self hosted, and nobody wants to host anything anymore when there's a free alternative that's good enough and hosted by someone else.

Skype is just universally terrible.

EDIT: since I'm part of that tiny minority exception that proves the rule: s/nobody wants/the vast majority do not want/

Re: How Discord Stores Billions of Messages (2017)

#136

Post by judge2020 » Tue, Aug 24, 2021, 8:34 PM UTC

humaniania wrote: post #107

m4rtink wrote: post #37

Earlier quoted context omitted.

It works nice and is free, but for how long? How long can they keep paying for that bandwidth and message data storage while keeping the thing essentially free?

Nitro pulls $10/mo and has enough benefits that a lot of people pay. Plus server boosts. I bet that they have good cash flow.

Our only source is this WSJ article (excerpt from qz[0]):

Discord declined to share how many Nitro subscribers it has, but the Wall Street Journal reported that Discord generated $130 million in revenue last year, up from $45 million in 2019. In the same time period, its monthly user base doubled.

0: https://qz.com/2034087/chat-app-discord-is-shedding-its-game....

Re: How Discord Stores Billions of Messages (2017)

#137

Post by mjevans » Tue, Aug 24, 2021, 8:37 PM UTC

porb121 wrote: post #86

Kaze404 wrote: post #82

Earlier quoted context omitted.

I can't think of many reasons Discord "ate the whole market" besides smart marketing, honestly. It does audio rooms incredibly well, but everything else (even their developer support team) is just terrible.

uh, how about the fact that vent/mumble/Skype were all awful?

The only bad part of mumble calls for voice come from:

  * Where the server is hosted / quality of server
  * Poor client UI

The client UI issue is how easy it is to work-around bad audio from other users. It's possible to do, the UI just completely sucks.

User interface and end user fulfillment just aren't great generally for OSS. I think it would take a commons improvement project with either government grants (infrastructure) paying for results AND/OR a university spearheading the development project.

Re: How Discord Stores Billions of Messages (2017)

#138

Post by YetAnotherNick » Tue, Aug 24, 2021, 8:39 PM UTC

didibus wrote: post #118

bhouser wrote: post #94

Earlier quoted context omitted.

The reason for Discord over Slack, BTW is voice & video. Otherwise Slack is working great.

Slack added voice now: https://slack.com/help/articles/4402059015315-Start-a-huddle...

They had voice from beginning. It just sucked so bad that they rebranded it to launch halfway decent option.

Re: How Discord Stores Billions of Messages (2017)

#139

Post by rcxdude » Tue, Aug 24, 2021, 8:42 PM UTC

Kaze404 wrote: post #82

RHSeeger wrote: post #36

Earlier quoted context omitted.

Honestly, I find discord super frustrating. Can't have multiple chats open at the same time, can't close the right rail, etc. It's UX is subpar in almost every way that matters to me. I use it because _everyone_ uses it, not because I want to.

I can't think of many reasons Discord "ate the whole market" besides smart marketing, honestly. It does audio rooms incredibly well, but everything else (even their developer support team) is just terrible.

I don't think they did much marketing. They provided, for free, a replacement for a chat room, forum, and voice chat, which was in total easier to set up than any of the previous options any gaming community had for those, with similar levels of functionality.

Re: How Discord Stores Billions of Messages (2017)

#140

Post by NotAnOtter » Tue, Aug 24, 2021, 8:50 PM UTC

bcrosby95 wrote: post #120

NotAnOtter wrote: post #7

This is like a masterclass in how to answer system design questions. Maybe a bit verbose. They cover requirements, how to answer those requirements, relevant tech for the problem, implementation, and techniques for maintenance

Are you saying that one person, without consulting other engineers, or having to do any research, made this decision in less than a day? Because I have some bad news for you. The sentiment you express here is why interviewing is so shit these days.

No? But I am saying this is like an extremely articulate, over the top, excessively detailed answer for a system design question. I'm not saying this is what people should aim for, just that it's a good example of the types of things you should discuss during a system design interview. I mean that's what it IS. It's a system design.