Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Discord is switching from Go to Rust

blog.discordapp.com

Post reply on HN
Print view View original

221–230 of 670 posts

Re: Why Discord is switching from Go to Rust

#221

Post by ithkuil »

steveklabnik wrote: post #148
pshc wrote: post #146

Earlier quoted context omitted.

Great comment and thanks for the reading material. Now I'm wondering if there's a Rust library for a generational copying arena--one that compacts strings/blobs over time.

Generational arenas yes, but copying, I'm not aware of one. It's very hard to get the semantics correct, since you can't auto-re-write pointers/indices.

Perhaps such a library could help you record the location of the variables that contain pointers to the strings and keep that pointer up to data as the ownership of the string moves from variable to variable?

I'm other words, doing some of the work a moving compacting collector would do during compaction but continuously during normal program execution.

Re: Why Discord is switching from Go to Rust

#222

Post by erk__ »

wongarsu wrote: post #77
_bxg1 wrote: post #32

Earlier quoted context omitted.

It does sound like Discord's case was fairly extraordinary in terms of the degree of the spike: > We kept digging and learned the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire LRU cache in order to determine if the memory was truly free from references. So maybe this is one of those things that just doesn't come up in most cases?…

Games written in the Unity engine are (predominately) written in C#, a garbage collected language. Keeping large amounts of data around isn't that unusual since reading from disk is often prohibitively slow, and it's normal to minimize memory allocation/garbage generation (using object pools, caches etc), and manually trigger the GC in loading screens and in other opportune places (as easy as calling System.GC.Collec…

C# uses a generational gc iirc so it may be better suited for a system where you have a relativly stable collection that does not need to be fully garbage collected all the time and have a smaller and more volitile set of objects that will be gc'ed more often. I don't think the current garbage collector in go does anything similar to that.

Re: Why Discord is switching from Go to Rust

#223

Post by gwbas1c »

rvcdbn wrote: post #157

Seems like you were hitting: runtime: Large maps cause significant GC pauses #9477 [0] Looks like this issue was resolved for maps that don't contain pointers by [1]. From the article, sounds like the map keys were strings (which do contain pointers, so the map would need to be scanned by the GC). If pointers in the map keys and values could be avoided, it would have (if my understanding is correct) removed the need…

Everything I've read indicates that RAM caches work poorly in a GC environment.

The problem is that garbage collectors are optimized for applications that mostly have short-lived objects, and a small amount of long-lived objects.

Things like large in-RAM LRU are basically the slowest thing for a garbage collector to do, because the mark-and-sweep phase always has to go through the entire cache, and because you're constantly generating garbage that needs to be cleaned.

Re: Why Discord is switching from Go to Rust

#224

Post by hinkley »

jasondclinton wrote: post #214
rvcdbn wrote: post #157

Seems like you were hitting: runtime: Large maps cause significant GC pauses #9477 [0] Looks like this issue was resolved for maps that don't contain pointers by [1]. From the article, sounds like the map keys were strings (which do contain pointers, so the map would need to be scanned by the GC). If pointers in the map keys and values could be avoided, it would have (if my understanding is correct) removed the need…

Finding out if that does resolve the author's issue would be interesting but I'm not sure that that would be particularly supportive data in favor of Go. If anything it would reinforce the downsides of Go's GC implementation: prone sudden pitfalls only avoidable with obtuse, error-prone fiddling that makes the code more complex. After spending weeks fighting with Java's GC tuning for a similar production service tail…

The common factor in most of my decisions to look for a new job has been realizing that I feel like a very highly compensated janitor instead of a developer.

Once I spend even the plurality of my time cleaning up messes instead of doing something new (and there are ways to do both), then all the life is sucked out of me and I just have to escape.

Telling me that I have to keep using a tool with known issues that we have to process or patches to fix would be super frustrating. And the more times we stumble over that problem the worse my confirmation bias will be.

Even if the new solution has a bunch of other problems, the set that is making someone unhappy is the one that will cause them to switch teams or quit. This is one area where management is in a tough spot with respect to rewrites.

Rewrites don't often fix many things, but if you suspect they're the only thing between you and massive employee turnover, you're between a rock and a hard place. The product is going to change dramatically, regardless of what decision you make.

Re: Why Discord is switching from Go to Rust

#225

Post by firethief »

tick_tock_tick wrote: post #218
firethief wrote: post #191

Earlier quoted context omitted.

Wow, that puts Discord's "absurd hack" into perspective! I feel like the moral here is a corollary to that law where people will depend on any observable behavior of the implementation: people will use any available means to tune important performance parameters; so you might as well expose an API directly, because doing so actually results in less dependence on your implementation details than if people resort to ce…

I mean if you read Twitch's hack they intentionally did it in code so they didn't need to tune the GC parameter. They wanted to avoid all environment config.

I missed that part. I thought they would use a parameter if it were available, because they said this:

> For those interested, there is a proposal to add a target heap size flag to the GC which will hopefully make its way into the Go runtime soon.

What's wrong with the existing parameter?

I'm sure they aren't going this far to avoid all environment config without a good reason, but any good reason would be a flaw in some part of their stack.

Re: Why Discord is switching from Go to Rust

#226

Post by asimpletune »

rvcdbn wrote: post #157

Seems like you were hitting: runtime: Large maps cause significant GC pauses #9477 [0] Looks like this issue was resolved for maps that don't contain pointers by [1]. From the article, sounds like the map keys were strings (which do contain pointers, so the map would need to be scanned by the GC). If pointers in the map keys and values could be avoided, it would have (if my understanding is correct) removed the need…

Ok but in rust those pointers can just be borrowed obviating the need for gc at all.

Re: Why Discord is switching from Go to Rust

#227

Post by azakai »

monocasa wrote: post #144
reggieband wrote: post #139

When I see this kind of GC performance, I wonder why you wouldn't change the implementation to use some sort of pool allocator. I am guessing each Read State object is identical to one another (e.g. some kind of struct) so why not pre-allocate your memory budget of objects and just keep an unused list outside of your HasMap? In a way this is even closer to a ring where upon ejection you could write the object to disk…

Yeah, they already weren't allocating, it was a GC pause that just scanned and would come up with essentially no extra garbage every two minutes.

A pool allocator could have reduced the number of existing allocations (1 big one instead of many small ones), making those spikes less significant. (But that depends on how Go handles interior pointers and GC, so I'm not sure.)

Re: Why Discord is switching from Go to Rust

#229

Post by kerkeslager »

kikimora wrote: post #216
kerkeslager wrote: post #158

Earlier quoted context omitted.

> It's always good to see a case-study/anecdote, but nothing in here is surprising. It also doesn't really invalidate Go in any way. Well, sure, because categorizing languages as "valid/invalid" doesn't make any sense. But it does show yet another example of how designing a language to solve Google's fairly-unique problems doesn't result in a general-purpose language suitable for solving most people's problems.

Long GC pauses caused by large collections/caches are decade long problem with no real wide spread solution so far. With Java and .NET you can resort to off-heap data. Not sure if this is possible with Go.

> Long GC pauses caused by large collections/caches are decade long problem with no real wide spread solution so far.

This is arguable, but the fact is that "large collections/caches" isn't Discord's situation.

Re: Why Discord is switching from Go to Rust

#230

Post by steveklabnik »

ithkuil wrote: post #221
steveklabnik wrote: post #148

Earlier quoted context omitted.

Generational arenas yes, but copying, I'm not aware of one. It's very hard to get the semantics correct, since you can't auto-re-write pointers/indices.

Perhaps such a library could help you record the location of the variables that contain pointers to the strings and keep that pointer up to data as the ownership of the string moves from variable to variable? I'm other words, doing some of the work a moving compacting collector would do during compaction but continuously during normal program execution.

There's no way to hook into the move, so I don't see how it would be possible, or at least, not with techniques similar to compacting GCs.
Post reply on HN