Live data from Hacker News

Why Discord is switching from Go to Rust

blog.discordapp.com

461–470 of 670 posts

Re: Why Discord is switching from Go to Rust

#461
post #414

Earlier quoted context omitted.

From /u/DiscordJesse on reddit: > We tried upgrading a few times. 1.8, 1.9, and 1.10. None of it helped. We made this change in May 2019. Just getting around to the blog post now since we've been busy. https://www.reddit.com/r/programming/comments/eyuebc/why_dis...

Another interesting comment in the same reddit thread, from /u/brian-discord ( https://old.reddit.com/r/programming/comments/eyuebc/why_dis... ): > Another Discord engineer chiming in here. I worked on trying to fix these spikes on the Go service for a couple weeks. We did indeed try moving up the latest Go at the time (1.10) but this had no effect. > For a more detailed explanation, it helps to understand what is go…

Ugh, linked lists fuck everything up. It’s never the right data structure. Use a vector!

Re: Why Discord is switching from Go to Rust

#462
post #416

Earlier quoted context omitted.

Check out this: https://github.com/frol/completely-unscientific-benchmarks

D, Nim and Crystal all do very well on all metrics. Author finds Rust pretty close but not as maintainable. Interesting that the top 3 (performance close to C++, but more maintainable) all are niche languages that haven't really broken into the mainstream.

I strongly believe, that people should consider the ecosystem part of maintainability risk and claim

Rust have by far the better ecosystem compared to those 3

Re: Why Discord is switching from Go to Rust

#463
post #257

Would it have been better if they went with Elixir? Write their code in a functional style. Get the benefits of the Erlang BEAM platform. Their system runs over the web, so time sensitivity isn’t as important, in comparison to video games, VR, or AR. Anyone ever done a performance comparison breakdown between something like Elixir vs. Rust?

"Would it have been better if they went with Elixir?" No. It would have been unshippably bad. BEAM is generally fairly slow. It was fast at multitasking for a while, but that advantage has been claimed by several other runtimes in 2020. As a language, it is much slower than Rust. Plus, if you tried to implement a gigantic shared cache map in Erlang/Elixir, you'd have two major problems: One is that you'd need huge ch…

Not to disagree with your analysis of the performance implications, but I don't think having all that data under a single or a few processes would be the right architectural pattern to handle this in Elixir.

The article says that the data is basically "per-user", indicating that the active client connection process could be used to store the data. It already hosts other data related to the client (connection) anyway. I think updating and querying it globally would be the trouble in that case.

Another could be storing the data in mnesia, BEAM's internal mutable in-memory DB. Probably better, but still not ideal to solve this.

Anyway, you're right in that no matter how you'd try to solve this problem on pure Elixir you'd still be seeing some bottlenecks because BEAM just isn't very well suitable for this kind of problems, hence Rust.

But can you elaborate on what you mean by other platforms catching up with Elixir's inherent concurrency advantages? Which modern platforms give similar features?

Re: Why Discord is switching from Go to Rust

#464
post #85
post #70

I feel like from the definition of the service, the entire thing could easily be replaced with a Redis cluster.

We originally cached this data with a Redis cluster but we hit scaling issues. The Read States service only exists because Redis had issues.

We use Elixir too and ironically, Redis became our bottleneck as well. What a useless dependency it is when running on BEAM.

Re: Why Discord is switching from Go to Rust

#465
post #434

Earlier quoted context omitted.

It's still a huge whoosh. You're starting at 1.9 and you're testing 4 micro versions to 1.10.. what is the point of that? None of those non-major versions are going to significantly change how the GC works. They could have tried 1 other version (not 4) and picked either the latest (1.13) or the version that contains the GC improvements (1.12) to test. Usually when you are looking to upgrade something you skim the rel…

> 1.12 seems to specifically address their performance concern Maybe it does, maybe it doesn't. Maybe 1.14 has a performance regression with their specific load and they get screwed. We'll never know. You know why? They permanently solved their GC problems by eliminating GC. That's the smart play.

Actually it turns out that they DID test the latest version which at the time was 1.10 (see my edit). I guess you should be surprised now? :D

Re: Why Discord is switching from Go to Rust

#466

Earlier quoted context omitted.

I write Java during the day but wish I could write Rust instead. Rust has a much nicer standard library, and the semantics and abstractions are so incredibly beautiful. It's a language that has learned from the millions of human-years that went into other languages and ecosystems. Every corner and seam in the language design speaks to this. Traits, union type enums, pattern matching, option and result types, derive()…

Rust has its share of random-feeling nonsense, like requiring PhantomData to "work around" unused type params, or that you can't compare arrays longer than 32 elements.

Not comparing arrays longer than 32 elements by == sounds like a feature to me.

Re: Why Discord is switching from Go to Rust

#467
post #257

Earlier quoted context omitted.

"Would it have been better if they went with Elixir?" No. It would have been unshippably bad. BEAM is generally fairly slow. It was fast at multitasking for a while, but that advantage has been claimed by several other runtimes in 2020. As a language, it is much slower than Rust. Plus, if you tried to implement a gigantic shared cache map in Erlang/Elixir, you'd have two major problems: One is that you'd need huge ch…

>It was fast at multitasking for a while, but that advantage has been claimed by several other runtimes in 2020. Such as?

Go, Rust is apparently getting there since it just smoked Go on a pretty core multitasking-heavy task here, and a lot of the old "stodgy" languages like Java while none of the cool kids were looking have gotten pretty good at large numbers of threads too.

I'm also expecting some sort of sensible solution to this sort of concurrency challenge to simply be baseline expected functionality for the next generation of languages. Anyone sitting down in 2020 to write The Next Big Language who ignores the fact that by the time they're done, their CPUs are going to be 64-core and the GPUs will be several-hundred-thousand core is really going to be missing the boat.

I actually encourage Erlang partisans to consider this a win in the general sense. Quite serious. If you consider languages as a multi-decade conversation, almost everything Erlang "said" in the late 1990s and early 2000s is in fact proving out to be true. However, while Erlang was a trailblazer, it isn't going to be that Next Big Language, nor anything like it. It got a lot of things right, but it's got too much wrong to be the NBL, and even if you did the minimal fixes, the result wouldn't be backwards compatible with Erlang anymore so it'd be a new language.

A lot of Erlang partisans are making the error of thinking the next language needs to be just like Erlang, except perhaps more so. But that's not how progress gets made. The good ideas are ripped out at a much more granular level and recombined with all sorts of other ideas and the end result may be quite different than what you expect. Go, for instance, can very much be seen as a direct sequel to Erlang among other things, even though it may not seem to have "OTP" or "built-in multinode concurrency" or whatever other apparent bullet-list features Erlang has, because a lot of those bullet-list features are really just epiphenomena of the real underlying features. Rust has its own nods to Erlang too; the whole "ownership" thing comes from the experiences with both mutation and immutability in a threading environment. But rather than making "Erlang, but a bit moreso", you get something that takes the lessons, ponders on them for another 15 years, and produces something different. It isn't Erlang, because at the time that Erlang was being written (Joe Armstrong RIP), nobody had the experience to think of Rust and imagine it could make a practical language. (Nor am I entirely sure the computers of the time could have compiled it; even if we sat through the clock time I'm not sure Rust could be stuffed into a 1997 desktop computer's RAM.)

Re: Why Discord is switching from Go to Rust

#468
post #456
post #86

Earlier quoted context omitted.

Right; Go is purpose-built for writing web services, and web services tend to be pretty tolerant of (small) latency spikes because virtually anyone who's calling one is already expecting at least some latency

> Go is purpose-built for writing web services Is this true? Go was built specifically for C++ developers, which, even when Go was first release, was a pretty unpopular language for writing web services (though maybe not at Google?). That a non-trivial number of Ruby/Python/Node developers switched was unexpected. (1) https://commandcenter.blogspot.com/2012/06/less-is-exponenti...

your quote is just corroborating what the reply is saying. go was written for web services which were written in c++ at google.

Re: Why Discord is switching from Go to Rust

#469
post #423

Earlier quoted context omitted.

You don't need microservices for that, though. One might as well have moved that piece into a library.

And then deal with cross-language FFI boundaries and cross-language builds.

This is what clicked for me on microservices years back. That the language wasn’t important and if I couldn’t do it in python or C, someone else could in Go or Java or etc.

Compared to if I wrote something in house entirely in C... lolno

Re: Why Discord is switching from Go to Rust

#470

I've heard lots of hot takes on "what Go really is". Here's mine. Go is what would have happened if Bell Labs wrote Java.

>Go is what would have happened if Bell Labs wrote Java

And Unix is what happened when Bell Labs wrote an operating system -- something that was born outdated from the start.

Just like Golang.

Post reply on HN