Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Discord is switching from Go to Rust

blog.discordapp.com

Post reply on HN
Print view View original

471–480 of 670 posts

Re: Why Discord is switching from Go to Rust

#471

Post by tatersolid »

zozbot234 wrote: post #79
oconnor663 wrote: post #6

Earlier quoted context omitted.

There are two things you'd have to do at the same time that make this complicated: - You'd have to ensure that your large data structure gets allocated entirely within the special region. That's simple enough if all you have is a big array, but it gets more complicated if you've got something like a map of strings. Each map cell and each string would need to get allocated in the special region, and all of the types i…

> Since the whole point of the region is that the GC doesn't scan it, nothing in the region will be able to keep anything outside the region alive. You can treat external references as GC roots.

> You can treat external references as GC roots.

Which brings you back to doing an expensive scan of the large off-heap data structure you were trying to avoid.

Re: Why Discord is switching from Go to Rust

#472

Post by haimez »

andoriyu wrote: post #445
haimez wrote: post #413

Earlier quoted context omitted.

I’m not sure I follow. GC implementations that don’t copy (relocate) are inherently subject to the performance cost of “fragmentation” (in the sense of scattering memory accesses over non-adjacent regions). This is a very high price to pay when you’re dealing with modern hardware.

Allocator underneath is keeping track of freed memory, so next allocation has high chance of being squeezed into memory region that has been used before. It's obviously not as good as say GC that relocates after sweep, but at least it doesn't leave gaping holes.

Indeed, but it also doesn’t maintain locality of access nearly as well for young objects (the most commonly manipulated ones) and even older ones that survive.

Re: Why Discord is switching from Go to Rust

#473

Post by jerf »

hopia wrote: post #463
jerf wrote: post #257

Earlier quoted context omitted.

"Would it have been better if they went with Elixir?" No. It would have been unshippably bad. BEAM is generally fairly slow. It was fast at multitasking for a while, but that advantage has been claimed by several other runtimes in 2020. As a language, it is much slower than Rust. Plus, if you tried to implement a gigantic shared cache map in Erlang/Elixir, you'd have two major problems: One is that you'd need huge ch…

Not to disagree with your analysis of the performance implications, but I don't think having all that data under a single or a few processes would be the right architectural pattern to handle this in Elixir. The article says that the data is basically "per-user", indicating that the active client connection process could be used to store the data. It already hosts other data related to the client (connection) anyway.…

"The article says that the data is basically "per-user","

Given that this is a table of who is "online", I don't think that's per-user in the sense that you are inferring. I infer that it's not a whole bunch of little local data that doesn't interact, it's a big global table of who is online and not online, constantly being heavily read from and written to in real time. Consider from the perspective of Bob's Erlang process that he wants to go offline and notify all of his currently-online friends that is is going offline. Bob's Erlang process doesn't have that data. Bob's Erlang process is going to get it from the Big Table of Who's Online. That table is the problem; it can't be stored in Bob's Erlang process.

I was at least imagining that the table could be partitioned into pieces pretty trivially (first X bits of the hash), but with Erlang's design, that implies an IPC just to ask some server process to give me the PID of the chunk I need to talk to, which itself is going to bottleneck. (In practice we'd probably cheat and use a NIF to do that, but that amounts to an admission that Erlang can't do this, so....)

At smaller scales you could try to live update Bob's local information as it changes, but this breaks down in all sorts of ways at scales far smaller than Discord, scales much closer to "a single mid-sized company".

"Another could be storing the data in mnesia, BEAM's internal mutable in-memory DB."

I have used mnesia for loads literally a ten-thousandth as small as this, if that (I could probably tack two more zeros on there), and it breaks down. It is an absolutely ludicrous idea that mnesia could handle what Discord is doing here. Last I knew the official Erlang community consensus was basically that mnesia really shouldn't be used for anything serious; my experience backed that up.

I think a non-trivial part of the reason why Erlang hasn't taken off is that its community still seems to exist in 2003, where it's a really incredible unique language that solves huge problems that nobody else does. In 2003, it rather has a point. But a lot of things have learned from Erlang, and incorporated its lessons into newer designs, and moved on.

See my other comment for what other runtimes have Erlang's advantages, but I'd invite you just to consider what we seem to basically agree on here; Erlang would be wildly slower and require a lot more hardware than Rust, the Rust code probably wasn't that hard to write, ... and the Rust code is way more likely to be correct than the Erlang code, too. I mean, what more "catching up to Elixir's inherent concurrency advantages" in this context than "did a job Elixir couldn't possibly do" do you want?

Re: Why Discord is switching from Go to Rust

#474

Post by millstone »

xmprt wrote: post #466
millstone wrote: post #452

Earlier quoted context omitted.

Rust has its share of random-feeling nonsense, like requiring PhantomData to "work around" unused type params, or that you can't compare arrays longer than 32 elements.

Not comparing arrays longer than 32 elements by == sounds like a feature to me.

The 32-count cliff is not just for Eq - it also defeats niceties like Debug, Hash, Default. This isn't a deliberate design decision, it's due to a (current) limitation of Rust's value-type generics.

The point is to illustrate that when you WTF using Rust, sometimes it's not you, it really is Rust.

Re: Why Discord is switching from Go to Rust

#475

Post by Techies4Trump »

wpietri wrote: post #446
the-alchemist wrote: post #96

Looks like the big challenge is managing a large, LRU cache, which tends to be a difficult problem for GC runtimes. I bet the JVM, with its myriad tunable GC algorithms, would perform better, especially Shenandoah and, of course, the Azul C4. The JVM world tends to solve this problem by using off-heap caches. See Apache Ignite [0] or Ehcache [1]. I can't speak for how their Rust cache manages memory, but the thing to…

Maybe I've missed this, but why do they need a particularly large LRU cache? Surely this isn't all one process, so presumably they could reduce spikes by splitting the same load across yet more processes?

Larger cache = faster performance and less load on the database.

I only glossed over the article but the problem they had with Go seems to be the GC incurred from having a large cache. Their cache eviction algorithm was efficient, but every 2 minutes there was a GC run which slowed things down. Re-implementing this algorithm in Rust gave them better performance because the memory was freed right after the cache eviction.

Splitting it across more processes will result in more cache misses and more DB calls.

Re: Why Discord is switching from Go to Rust

#476

Post by romaniitedomum »

hajile wrote: post #342
romaniitedomum wrote: post #332

You're switching to Rust because Go is too slow? Colour me sceptical, but this seems more like an excuse to adopt a trendy language than a considered technical decision. Rust is designed first and foremost for memory safety, and it sacrifices a lot of developer time to achieve this, so if memory safety isn't high in your list of concerns Rust is probably not going to bring many benefits.

Did you read the article? The naive Rust version was better than the tuned golang version in every metric. The most important one (latency) simply wasn't fixable due to golang's GC (something that is a bit of a general GC issue I might add).

Did you read my comment? I don't dispute that the Rust version is faster in every way. I am disputing that rewriting in Rust was a sensible technical decision, and in support of this I point you to where the author describes having to use a nightly build of the compiler to get async support. Given that they had to jump through a lot of hoops to make this work, I am saying they could have achieved the same speed increase with less effort using a stable C or C++ compiler. Hell, had they invested a fraction of the time spent rewriting in Rust in the Go version, I'll bet they could have improved it to the point where there was no need to rewrite it at all.

It's clear that Discord use Rust a lot, and that they are looking for any excuse to replace existing code with Rust code.

Re: Why Discord is switching from Go to Rust

#477

Post by kick »

correct_horse wrote: post #430
kick wrote: post #28

Earlier quoted context omitted.

Minor nitpick: That already happened, Limbo is what happened when Bell Labs wrote Java.

Huh. I managed to hear about Inferno, but not remember the Limbo part. In that case, Go is Bell Labs' second attempt at Java.

Third, there was also a language I can't remember the name of that happened at the same time as Alef.

Re: Why Discord is switching from Go to Rust

#478

Post by logicchains »

roca wrote: post #407
Polyisoprene wrote: post #387

Earlier quoted context omitted.

Sure, but generics are available in Java as well and compared to Haskell and similar language I rate Rust as closer to Java.

I think most people would rate Rust as being closer to Haskell than Java at the type level. Rust traits are very much like Haskell type classes. Java gives you classic OOP, but neither Rust nor Haskell do. Rust generics are much more like Haskell generics than Java generics. Rust and Haskell both have associated types, Java doesn't. Java generics are crippled due to type erasure; Rust and Haskell don't have those lim…

>Java generics are crippled due to type erasure; Rust and Haskell don't have those limitations.

Both Rusk and Haskell erase types at runtime. The difference is they don't rely on runtime reflection for anything, so it doesn't hurt them.

Re: Why Discord is switching from Go to Rust

#479

Post by romaniitedomum »

anon4242 wrote: post #366
romaniitedomum wrote: post #332

You're switching to Rust because Go is too slow? Colour me sceptical, but this seems more like an excuse to adopt a trendy language than a considered technical decision. Rust is designed first and foremost for memory safety, and it sacrifices a lot of developer time to achieve this, so if memory safety isn't high in your list of concerns Rust is probably not going to bring many benefits.

Turns out that that boring stuff - type theory - they throw at you in uni, can be quite useful. Not only can it help with things like memory safety but also with speed. This is why for instance C++ std::sort() is faster than C qsort(), better type information available to the compiler allows it to make better optimizations. In rust the type system is king.

No, in Ada the type system is king. If type theory is the solution, then Ada and SPARK, Ada's restricted subset for extra safety, leave Rust in the dust.

Re: Why Discord is switching from Go to Rust

#480

Post by romaniitedomum »

smabie wrote: post #344
romaniitedomum wrote: post #332

You're switching to Rust because Go is too slow? Colour me sceptical, but this seems more like an excuse to adopt a trendy language than a considered technical decision. Rust is designed first and foremost for memory safety, and it sacrifices a lot of developer time to achieve this, so if memory safety isn't high in your list of concerns Rust is probably not going to bring many benefits.

What would you recommend that doesn’t have a GC? Zig? C? Rust is a fine choice. Besides if you really don’t care, just make the entire program unsafe and you’ll still reap benefits over C or C++.

Whatever language and toolset gets the job done with the least amount of effort. Given the hoops that Discourse had to jump through to get Rust working, that wasn't a good technical decision. They'd have got the same result with less pain with C++.
Post reply on HN