Why not C++, if performance was an issue?
Why C++? Why would you want the same performance as Rust with less safety?
Why Discord is switching from Go to Rust
131–140 of 670 posts
Re: Why Discord is switching from Go to Rust
#132> After digging through the Go source code, we learned that Go will force a garbage collection run every 2 minutes at minimum. In other words, if garbage collection has not run for 2 minutes, regardless of heap growth, go will still force a garbage collection. > We figured we could tune the garbage collector to happen more often in order to prevent large spikes, so we implemented an endpoint on the service to change…
It does sound like Discord's case was fairly extraordinary in terms of the degree of the spike: > We kept digging and learned the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire LRU cache in order to determine if the memory was truly free from references. So maybe this is one of those things that just doesn't come up in most cases?…
Yes, this is a maximally pessimal case for most forms of garbage collection. They don't say, but I would imagine these are very RAM-heavy systems. You can get up to 768GB right now on EC2. Fill that entire thing up with little tiny objects the size of usernames or IDs for users, or even merely 128GB systems or something, and the phase where you crawl the RAM to check references by necessity is going to be slow.
This is something important to know before choosing a GC-based language for a task like this. I don't think "generating more garbage" would help, the problem is the scan is slow.
If Discord was forced to do this in pure Go, there is a solution, which is basically to allocate a []byte or a set of []bytes, and then treat it as expanse of memory yourself, managing hashing, etc., basically, doing manual arena allocation yourself. GC would drop to basically zero in that case because the GC would only see the []byte slices, not all the contents as individual objects. You'll see this technique used in GC'd languages, including Java.
But it's tricky code. At that point you've shucked off all the conveniences and features of modern languages and in terms of memory safety within the context of the byte expanses, you're writing in assembler. (You can't escape those arrays, which is still nice, but hardly the only possible issue.)
Which is, of course, where Rust comes in. The tricky code you'd be writing in Go/Java/other GC'd language with tons of tricky bugs, you end up writing with compiler support and built-in static checking in Rust.
I would imagine the Discord team evaluated the option of just grabbing some byte arrays and going to town, but it's fairly scary code to write. There are just too many ways to even describe for such code to end up having a 0.00001% bug that will result in something like the entire data structure getting intermittently trashed every six days on average or something, virtually impossible to pick up in testing and possibly even escaping canary deploys.
Probably some other languages have libraries that could support this use case. I know Go doesn't ship with one and at first guess, I wouldn't expect to find one for Go, or one I would expect to stand up at this scale. Besides, honestly, at feature-set maturity limit for such a library, you just end up with "a non-GC'd inner platform" for your GC'd language, and may well be better off getting a real non-GC'd platform that isn't an inner platform [1]. I've learned to really hate inner platforms.
By contrast... I'd bet this is fairly "boring" Rust code, and way, way less scary to deploy.
Re: Why Discord is switching from Go to Rust
#133This was an extremely interesting read.
I'm quiet disappointed though they did not update their Go Version to 1.13[0][1] which would normally have remove the spike issue and thus he latency before they move to Rust...
Rust seems more performant with proper usage ( tokio + async ) but I'm more worried about the ecosystem that doesn't seem has mature has Go.
We could quote the recent[2] Drama with Actix...
[0]https://golang.org/doc/go1.13#runtime [1]https://golang.org/doc/go1.12#runtime [2]https://github.com/fafhrd91/actix-web-postmortem
Re: Why Discord is switching from Go to Rust
#134Earlier quoted context omitted.
GC isn't "slow" insofar as it's non deterministic. Modern garbage collectors are extremely fast, in fact.
Running every two minutes sounds pretty deterministic.
It was perhaps too deterministic. What's not mentioned in the blog is that after running for long enough, the cluster would line up it's GCs, and each node would do the 2 minute GC at exactly the same time causing bigger spikes as the entire cluster would degrade. I'm guessing all it takes is a few day night cycles combined with a spike in traffic to make all the nodes reset their forced GC timers to the same time.
Re: Why Discord is switching from Go to Rust
#135Earlier quoted context omitted.
> everyone can read the code with the indentation they prefer, while the file stays the same. Have you ever worked in a code base with many contributors that changed over the course of years? In my experience it always ends up a jumble where indentation is screwed up and no particular tab setting makes things right. I've worked on files where different lines in the same file might assume tab spacing of 2, 3, 4, or 8.…
On the good side, the problem of mixing tabs and spaces does normally not appear in Go sources, as gofmt always converts spaces to tabs, so there is no inconsistant indentation. Normally I prefer spaces to tabs because I dislike the mixing, but gofmt solves this nicely for me.
Please explain to me how this works for the case I outlineed, eg:
some_function(arg1, arg2, arg3, arg4,
arg5, arg6);
Re: Why Discord is switching from Go to Rust
#136Replatforming to solve this problem was a bit silly in my opinion. The solution to the problem was "do fewer allocations" which can be done in any language.
They addressed this in the article:
> These latency spikes definitely smelled like garbage collection performance impact, but we had written the Go code very efficiently and had very few allocations. We were not creating a lot of garbage.
The problem was due to the GC scanning all of their allocated memory and taking a long time to do so, regardless of it all being necessary and valid memory usage.
Re: Why Discord is switching from Go to Rust
#137Earlier quoted context omitted.
I don't see why that's hilarious. Lots of programs take a second or two to load and it only happens once on boot for me. "Read states" is just discord telling you which channels and servers you have unread messages in
Discord takes longer to start up than Microsoft Word. Desktop development is a total wasteland these days -- there isn't nearly as much effort put into optimization as server side. They're not paying for your local compute, so they can waste as much of it as they want.
I feel that it's not really fair to expect them to natively implement their app on every platform and put tons of resources into it's client performance - anecdotally discord is a very responsive app - see [0].
But think of it this way, all the effort they put into their desktop app works on all major OSes without a problem. They even get to reuse most of the code for access from the browser, with no installation required.
Now imagine approaching your PM and saying "Look I know we put X effort into making our application work on all the platforms, but it would be even faster if we instead did 4x effort for native implementations + the browser".
[0] From what I've seen in the "gamer community" is that most gamers don't care that much about that kind of extra performance. Discord itself doesn't feel slow once it's started. Joining a voice channel is instant, quickly switching to a different server and then to a chat channel to throw in a some text is fast and seamless (Looking at you MS Teams!!!).
Sure Mumble/Teamspeak are native and faster, but where are their first party mobile apps and web clients? One of the incredible things Discord did to really aid in adoption was allow for web clients, so when you found some random person on the internet, you didn't have to tell them to download some chat client, they could try it through their browser first.
tl;dr
Yes electron apps can be slow, but discord IMO has fine client side performance, and they clearly do put resources into optimizing it. Yes it "could be faster" with native desktop apps, but their target community seems perfectly content as is.
Re: Why Discord is switching from Go to Rust
#138Why would they switch to rust, rather than upgrading from 3 years old version?
This blog post perhaps is a bit "after the fact" we had made the switch over mid 2019, and wanted to try out rust as well for services like this, due to adoption elsewhere in the company. Also, after upgrading between 4 golang versions on this service and noticing it didn't materially change performance, we decided to just spend our time on the rewrite (for fun, and latency) and to get a head start into the asynchron…
Shiny toy syndrome, basically.
Re: Why Discord is switching from Go to Rust
#139When I see this kind of GC performance, I wonder why you wouldn't change the implementation to use some sort of pool allocator. I am guessing each Read State object is identical to one another (e.g. some kind of struct) so why not pre-allocate your memory budget of objects and just keep an unused list outside of your HasMap? In a way this is even closer to a ring where upon ejection you could write the object to disk (or Cassandra), re-initialise the memory and then reuse the object for the new entry.
I suppose that won't stop the GC from scanning the memory though ... so maybe they had something akin to that. I assume that a company associated with games and with some former games programmers would have thought to use pool allocators. Honestly, if that strategy didn't work then I would be a bit frustrated with Go.
I have to say, out of all of the non-stop spamming of Rust I see on this site - this is definitely the first time I've thought to myself that this is a very appropriate use of the language. This kind of simple yet high-throughput workhorse of a system is a great match for Rust.
Re: Why Discord is switching from Go to Rust
#140This was an extremely interesting read. I'm quiet disappointed though they did not update their Go Version to 1.13[0][1] which would normally have remove the spike issue and thus he latency before they move to Rust... Rust seems more performant with proper usage ( tokio + async ) but I'm more worried about the ecosystem that doesn't seem has mature has Go. We could quote the recent[2] Drama with Actix... [0] https://…
Why would you want to bring up the Actix author's drama? That doesn't seem like something that should reflect on a language one way or the other.