Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Discord is switching from Go to Rust

blog.discordapp.com

Post reply on HN
Print view View original

481–490 of 670 posts

Re: Why Discord is switching from Go to Rust

#481

Post by jamra »

jhgg wrote: post #67
mangatmodi wrote: post #52

Why would they switch to rust, rather than upgrading from 3 years old version?

This blog post perhaps is a bit "after the fact" we had made the switch over mid 2019, and wanted to try out rust as well for services like this, due to adoption elsewhere in the company. Also, after upgrading between 4 golang versions on this service and noticing it didn't materially change performance, we decided to just spend our time on the rewrite (for fun, and latency) and to get a head start into the asynchron…

This comment doesn’t make sense. Didn’t rust not have async back then?

The timelines don’t appear to fit

Re: Why Discord is switching from Go to Rust

#482

Post by hopia »

jerf wrote: post #473
hopia wrote: post #463

Earlier quoted context omitted.

Not to disagree with your analysis of the performance implications, but I don't think having all that data under a single or a few processes would be the right architectural pattern to handle this in Elixir. The article says that the data is basically "per-user", indicating that the active client connection process could be used to store the data. It already hosts other data related to the client (connection) anyway.…

"The article says that the data is basically "per-user"," Given that this is a table of who is "online", I don't think that's per-user in the sense that you are inferring. I infer that it's not a whole bunch of little local data that doesn't interact, it's a big global table of who is online and not online, constantly being heavily read from and written to in real time. Consider from the perspective of Bob's Erlang p…

Yeah the scale is what makes this problem a problem here. I've done exactly that "online" stuff per user process and it works fine on a small scale, even when it needs to be globally inferred. But I suspect it'd quickly become the bottleneck when scaling.

I had no idea mnesia was that fragile though, what gives? What kind of issues did you encounter with it? What do you use now to solve those issues with Erlang/Elixir?

Sure, we all know Erlang doesn't shine in computationally intensive workloads. Obviously, Rust was the right call here. But stateful distributed soft real-time concurrency, can you really say with a straight face that Rust comes with all the same features as BEAM out-of-the-box? Or any other modern platform for that matter. I've yet to see Erlang/Elixir beaten in that particular niche.

Re: Why Discord is switching from Go to Rust

#483

Post by gmjosack »

jamra wrote: post #481
jhgg wrote: post #67

Earlier quoted context omitted.

This blog post perhaps is a bit "after the fact" we had made the switch over mid 2019, and wanted to try out rust as well for services like this, due to adoption elsewhere in the company. Also, after upgrading between 4 golang versions on this service and noticing it didn't materially change performance, we decided to just spend our time on the rewrite (for fun, and latency) and to get a head start into the asynchron…

This comment doesn’t make sense. Didn’t rust not have async back then? The timelines don’t appear to fit

Tokio and Futures have existed since 2016. I worked on the initial loqui implementation that powers rpc at Discord in raw Futures/Tokio in late 2018. async/await was also on nightly back then. Jesse finished it and migrated it to async/await and later used that as the basis for Read States. The timelines make perfect sense.

Re: Why Discord is switching from Go to Rust

#484

Post by lumost »

millstone wrote: post #474
xmprt wrote: post #466

Earlier quoted context omitted.

Not comparing arrays longer than 32 elements by == sounds like a feature to me.

The 32-count cliff is not just for Eq - it also defeats niceties like Debug, Hash, Default. This isn't a deliberate design decision, it's due to a (current) limitation of Rust's value-type generics. The point is to illustrate that when you WTF using Rust, sometimes it's not you, it really is Rust.

A great example of this can be seen writing any recursive data-structure such as a doubly linked list.

Re: Why Discord is switching from Go to Rust

#485

Post by jfim »

chrisseaton wrote: post #316
stingraycharles wrote: post #275

Earlier quoted context omitted.

> The JVM world tends to solve this problem by using off-heap caches. See Apache Ignite [0] or Ehcache [1]. For those who care, I was interested how off-heap caching works in Java and I did some quick searching around the Apache Ignite code. The meat is here: - GridUnsafeMemory, an implementation of access to entries allocated off-heap. This appears to implement some common Ignite interface, and invokes calls to a “G…

> context switches between JVM and native code are typically fairly expensive Aren't these Unsafe memory read and write methods intrinsified by any serious compiler? I don't believe they're using JNI or doing any kind of managed/native transition, except in the interpreter. They turn into the same memory read and write operations in the compiler's intermediate representation as Java field read and writes do.

They are optimized, yes, but from what I recall from reading the JVM code a few years ago, some optimizations don't get applied to those reads/writes. For example, summing two arrays together will be vectorized to use SSE instructions while doing so through Unsafe won't [0].

[0] https://cr.openjdk.java.net/~vlivanov/talks/2017_Vectorizati...

Re: Why Discord is switching from Go to Rust

#486

Post by anarazel »

earthboundkid wrote: post #461
cesarb wrote: post #414

Earlier quoted context omitted.

Another interesting comment in the same reddit thread, from /u/brian-discord ( https://old.reddit.com/r/programming/comments/eyuebc/why_dis... ): > Another Discord engineer chiming in here. I worked on trying to fix these spikes on the Go service for a couple weeks. We did indeed try moving up the latest Go at the time (1.10) but this had no effect. > For a more detailed explanation, it helps to understand what is go…

Ugh, linked lists fuck everything up. It’s never the right data structure. Use a vector!

That's only true if you actually need to access more than a few elements at once. And if you never need to insert/delete to/from anywhere but the end.

Re: Why Discord is switching from Go to Rust

#488

Post by jhgg »

falcolas wrote: post #450
jhgg wrote: post #432

Earlier quoted context omitted.

Assuming only one thread at a time needs to access the LRU cache (not hard with the shared-nothing message passing architecture which we employ here), the lifetime of the object being checked out from the cache is able to be understood at compile time, and we can just use the borrow checker to ensure that it remains that way (we've got a mutable reference to the LRU, and we can use that to get a mutable reference to…

Yes, I'm quite familiar with rust's borrow checking model. I've programmed some in rust, and the rest has been beaten into my head quite thoroughly by Rustaceans. I don't care for Rust, but I understand it. Locking on one thread at a time seems like a pretty obvious performance flaw. It just doesn't seem like an appropriate design for the given workload (lots of requests, lots of stored items, largely write-only (exc…

In practice, for our service, most of our CPU time is not spent in data mutation, but rather networking and serialization (this is btw, the same conclusion Redis came to when they added "multi-threading".)

You can scale-out by running multiple instances of the service (shared-nothing, N many depending on how cores you want to run on.) Or, you can do message-passing between cores.

In this case, we have 2 modes of scale-up/out (add more nodes to the cluster, or add more shared-nothing LRU caches that are partitioned internally that the process runs, allowing for more concurrency).

We however only run one LRU per node, as it turns out that the expensive part is not the bottleneck here, nor will it probably ever be.

Re: Why Discord is switching from Go to Rust

#489

Post by jamra »

gmjosack wrote: post #483
jamra wrote: post #481

Earlier quoted context omitted.

This comment doesn’t make sense. Didn’t rust not have async back then? The timelines don’t appear to fit

Tokio and Futures have existed since 2016. I worked on the initial loqui implementation that powers rpc at Discord in raw Futures/Tokio in late 2018. async/await was also on nightly back then. Jesse finished it and migrated it to async/await and later used that as the basis for Read States. The timelines make perfect sense.

Alright. I stand corrected. I used Go back when it was beta, but it never stuck with me. I still like it for small script like tasks. I also happen to think Rust is amazing. The learning curve kept me away for a while.

It would still be interesting to see them post how go > 1.12 would do since it no longer has stop the world garbage collection.

Re: Why Discord is switching from Go to Rust

#490

Post by Tomis02 »

SlowRobotAhead wrote: post #469
kccqzy wrote: post #423

Earlier quoted context omitted.

And then deal with cross-language FFI boundaries and cross-language builds.

This is what clicked for me on microservices years back. That the language wasn’t important and if I couldn’t do it in python or C, someone else could in Go or Java or etc. Compared to if I wrote something in house entirely in C... lolno

Landing in a shop that uses N programming languages for N microservices would be a pretty miserable experience.
Post reply on HN