Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Stores Billions of Messages (2017)

blog.discord.com

Post reply on HN
Print view View original

341–350 of 377 posts

Re: How Discord Stores Billions of Messages (2017)

#341

Post by lmilcin »

bob1029 wrote: post #331
lmilcin wrote: post #324

Earlier quoted context omitted.

> but servers crash all the time This is both wise and stupid at the same time. It is wise if you mean "be ready for servers to crash at any time by thinking they are going to crash at the worst possible moment". But it is stupid, because people think they need massive parallel deployments just because servers will be constantly crashing and it is just not true. The cost they pay is in having couple of times more nod…

> The reality is, servers do not crash This is the regime we operate in as well. For our business, a failure, while really bad, is not catastrophic (we still maintain non-repudiation). We look at it like any other risk model in the market. For many in our industry, the cost of not engineering this way and eating the occasional super rare bag of shit is orders of magnitude higher than otherwise tolerable. One well-man…

Yes, that is my experience.

Another super important thing to remember is that main goal of this is to have super simple code and very simple but rock solid guarantees.

The main benefit is writing application code that is simple, easy to understand and simple to prove it works correctly, enabled by reliable infrastructure.

When you are not focusing on various ridiculous technologies that each require PhDs to understand well, you can focus on your application stack, domain modeling, etc. to make it even more reliable.

Re: How Discord Stores Billions of Messages (2017)

#342

Post by lmilcin »

onei wrote: post #336
anonymoushn wrote: post #330

Earlier quoted context omitted.

Maybe if you use nostd. I evaluated a bunch of Rust libraries for some server software, but I could not use any of them because they pervasively assume that it is ok to make syscalls to allocate memory. If you'd like to write software that makes few syscalls in the steady state, you can do it in rust, but you can't use libraries. Or String or Vec, I guess.

Why is memory allocation via syscalls bad? I get it for embedded (which was mentioned above so perhaps that's what this targets), but I kind of assumed malloc was a syscall underneath on an actual OS and that was fine.

Actually, you don't need to run any functions to cause a context switch. For example, at the very lowest level, even trying to access a memory page that is presently in physical memory but does not have entry in TLB causes CPU to interrupt to OS to ask for mapping.

Re: How Discord Stores Billions of Messages (2017)

#343

Post by lmilcin »

adamnemecek wrote: post #285
tonymet wrote: post #284

Earlier quoted context omitted.

Considering that a cpu can do 3 billion things a second , and a typical laptop can store 16 billion things in memory , it shouldn’t take more than 5 of these to handle “billions of messages” . I agree with you that modern frameworks are inefficient

By 16 billion things you mean 16 billion bytes? If you are talking about physical memory, then no, you can't occupy the entire memory. If you are talking about virtual memory, then you can store more data.

Actually, CPU processes things in words, not bytes. On 64-bit architecture the word is 64 bit or 8 bytes.

But there is a lot of things that CPU can do even faster than that, because this limitation only relates to actual instruction execution (and even then there are instructions that can process multiple words at a time).

Re: How Discord Stores Billions of Messages (2017)

#344

Post by Kaze404 »

xmprt wrote: post #309
Kaze404 wrote: post #267

Earlier quoted context omitted.

What I think happened, as someone who's been using Discord daily since 2015, is that they came up with a slightly better product than the alternatives, spent enough in marketing (to gamers specifically) to convince investors that it was a platform worth investing in, and only then slowly started improving their faulty software. To say people were manipulated into using Discord is obviously not true, but it's also dis…

The last part of the comment was probably a reply to this: "I don't see how that's relevant? I don't even prefer those over Discord, but I don't think it's enough of an improvement to warrant the market share it has now." You don't see how those valuable those improvements are but the average person does and that's why it has a massive market share.

Ah, I see. Thanks for the clarification.

Re: How Discord Stores Billions of Messages (2017)

#345

Post by bob1029 »

lmilcin wrote: post #341
bob1029 wrote: post #331

Earlier quoted context omitted.

> The reality is, servers do not crash This is the regime we operate in as well. For our business, a failure, while really bad, is not catastrophic (we still maintain non-repudiation). We look at it like any other risk model in the market. For many in our industry, the cost of not engineering this way and eating the occasional super rare bag of shit is orders of magnitude higher than otherwise tolerable. One well-man…

Yes, that is my experience. Another super important thing to remember is that main goal of this is to have super simple code and very simple but rock solid guarantees. The main benefit is writing application code that is simple, easy to understand and simple to prove it works correctly, enabled by reliable infrastructure. When you are not focusing on various ridiculous technologies that each require PhDs to understan…

> When you are not focusing on various ridiculous technologies that each require PhDs to understand well, you can focus on your application stack, domain modeling, etc. to make it even more reliable.

This is 100% our philosophy. I honestly don't understand why all high-stakes software isn't developed in the same way that we build these trading/data systems.

I think this is the boundary between "engineering" and "art". In my experience, there are a lot of developers who feel like what they do is not engineering because they believe it to be so subjective and open to interpretation. Perhaps there is a mentality that it cant ever be perfect or 100% correct, so why even try upholding such a standard as realistic? It is certainly more entertaining to consume new shiny technology than sitting down with business owners in boring meetings for hours every week...

In reality, you can build software like you build nuclear reactors. It is all a game of complexity management and discipline. Surprisingly, it usually costs less when you do it this way, especially after accounting for the total lifecycle of the product/service. If you can actually build a "perfect" piece of software, you can eliminate entire parts of the org chart. How many developer hours are spent every day at your average SV firm fighting bugs and other regressions? What if you could take this to a number approximating zero?

The classical retort I hear from developers when I pose comments like these is "Well the business still isnt sure exactly what the app should do or look like". My response to that is "Then why are you spinning up Kubernetes clusters when you should be drawing wireframes and schema designs for the customer to review?"

Re: How Discord Stores Billions of Messages (2017)

#346

Post by easton »

keithnz wrote: post #174
humaniania wrote: post #107

Earlier quoted context omitted.

Nitro pulls $10/mo and has enough benefits that a lot of people pay. Plus server boosts. I bet that they have good cash flow.

we use discord for work and to "boost" your server to a level where you get reasonable streaming you need to pay ~$60 and a higher upload limit, or ~$110 for the max. Which is pretty good in that it applies for all users It's a bit of an odd model for paying for businesses, but works well in the gaming world where multiple people can essentially help pay for a server (if you want the extra toys)

Why did you pick Discord over Slack or Teams? I'd be driven crazy if there wasn't any SSO or fancy admin features, but then again I'm a nerd who cares about things like that (and also why I really want Discord for Enterprise to happen). Is it just because it's easier to use?

Re: How Discord Stores Billions of Messages (2017)

#347

Post by [deleted] »

BigJono wrote: post #317
combyn8tor wrote: post #313

Earlier quoted context omitted.

Well on one hand you've got engineers at a billion dollar company explaining how they've solved a problem. On the other hand you've got some random commentor on HN over-simplifying a complex engineering solution.

Sounds to me like it's some "random commentor" who has solved a similar problem at a similar scale with a solution that's much simpler.

[deleted]

Re: How Discord Stores Billions of Messages (2017)

#348

Post by lmilcin »

michael_j_ward wrote: post #339
lmilcin wrote: post #237

Earlier quoted context omitted.

As an example, there are bitemporal queries like "for the given population of trades specified by following rules, find the set of trades that met the rules at a particular point in time, based on our knowledge at another given point in time". Also trades are versioned (are a stream of business events from trading system), then have amendments (each event may be amended in the future but the older version must be pre…

Thanks for all the great insight. Did you consider an explicitly bitemporal database like crux[0]? [0] https://github.com/juxt/crux ?

Our application predates crux. As to crux, I can't comment because I did not know about it when I was actively searching for a product to do this.

Re: How Discord Stores Billions of Messages (2017)

#349

Post by lmilcin »

gpderetta wrote: post #327
lmilcin wrote: post #270

Earlier quoted context omitted.

Actually, our threading is quite simple. There is exactly as many threads (that do anything) as CPU cores.

I think parent meant threading as in message threads.

Just recently Symphony switched order in which I have received two messages from a colleague. This completely changed the meaning of the message and we got in an argument that was only resolved after I have posted screenshot of what he wrote.

It seems, the threading might not be that simple after all.

Re: How Discord Stores Billions of Messages (2017)

#350

Post by lmilcin »

Cullinet wrote: post #257
lmilcin wrote: post #233

Earlier quoted context omitted.

I have a plan to write a book on how to write reactive applications like that. Mostly collection of observations, tips, tricks, patterns for reactive composition, some very MongoDB specific solutions, etc. Not sure how many people would be interested. Reactor has quite steep learning curve but also very little literature on how to use for anything non-trivial. The aim is not just enable good throughput, but also achi…

I'm interested in getting your book published. Career in publishing and specialist media but a lot of it spent on related problems to your subject. Semi retired have risk capital to get to the right distribution maintaining well above industry standard terms. Email in profile.

Thanks. I will try to self publish. I want to keep freedom over content and target and I am not looking for acclaim for having my name on a book from a well known publisher. I am just hoping to help people solve their problems.
Post reply on HN