Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Discord is switching from Go to Rust

blog.discordapp.com

Post reply on HN
Print view View original

121–130 of 670 posts

Re: Why Discord is switching from Go to Rust

#121

Post by monocasa »

nickserv wrote: post #76
andai wrote: post #66

Earlier quoted context omitted.

Offtopic but what are you missing when you have to use tabs instead of spaces? I can understand different indentation preferences but I can change the indentation width per tab in my editor. And then everyone can read the code with the indentation they prefer, while the file stays the same.

It's just an example of something that the Go team took a decision on, and won't allow you to change. I mean, even Python lets you choose. I don't really have a problem with it however, even if I do prefer spaces.

There's a difference between making decisions that are really open to bikeshedding, and making sweeping decisions in contexts that legitimately need per app tuning like immature GCs.

The Azul guys get to claim that you don't need to tune their gc, golang doesn't.

Re: Why Discord is switching from Go to Rust

#122

Post by abraxas »

_ph_ wrote: post #104
abraxas wrote: post #65

Earlier quoted context omitted.

Seems like Go is more suitable for the “spin up, spin down, never let the GC run” kind of scenario that is being pushed by products like AWS Lambda and other function as a service frameworks.

Why do you think it is? Go has a really great gc which mostly runs in parallel to your program with gc stops only in the doman of less than milliseconds. Discord ran into a corner case where they did not create enough garbage to trigger gc cycles, but had a performance impact due to scheduled gc cycles for returning memory to the OS (which they wouldn't need to do either).

Because many services eventually become performance bottlenecked either via accumulation of users or accumulation of features. In either case eventually performance becomes very critical.

Re: Why Discord is switching from Go to Rust

#123

Post by Will_Parker »

tasty_freeze wrote: post #106
andai wrote: post #66

Earlier quoted context omitted.

Offtopic but what are you missing when you have to use tabs instead of spaces? I can understand different indentation preferences but I can change the indentation width per tab in my editor. And then everyone can read the code with the indentation they prefer, while the file stays the same.

> everyone can read the code with the indentation they prefer, while the file stays the same. Have you ever worked in a code base with many contributors that changed over the course of years? In my experience it always ends up a jumble where indentation is screwed up and no particular tab setting makes things right. I've worked on files where different lines in the same file might assume tab spacing of 2, 3, 4, or 8.…

> In my experience it always ends up a jumble where indentation is screwed up and no particular tab setting makes things right.

Consider linting tools in your build.

Re: Why Discord is switching from Go to Rust

#124

Post by jhgg »

arnsholt wrote: post #63
unlinked_dll wrote: post #26

It'd be cool to look at more signal statistics from the CPU plot. It appears that Go has a lower CPU floor, but it's killed by the GC spikes, presumably due to the large cache mentioned by the author. This is interesting to me. It suggests that Rust is better at scale than Go, and I would have thought with Go's mature concurrency model and implementation would have been optimized for such cases while Rust would shine…

My first guess for the slightly higher CPU floor of the Rust version is that the Rust code has to do slightly more work per request, since it will free memory as it gets dropped, whereas the Go code doesn't do any freeing per request, but then gets hit with the periodic spike every two minutes where the entire heap has to be traversed for GC.

tokio 0.1 was definitely less efficient, when we compare go to 0.2, tokio uses less cpu consistently, even when compared to a cluster of the same size almost a year later with our growth over the time since we switched over.

Re: Why Discord is switching from Go to Rust

#126

Post by _ph_ »

tasty_freeze wrote: post #106
andai wrote: post #66

Earlier quoted context omitted.

Offtopic but what are you missing when you have to use tabs instead of spaces? I can understand different indentation preferences but I can change the indentation width per tab in my editor. And then everyone can read the code with the indentation they prefer, while the file stays the same.

> everyone can read the code with the indentation they prefer, while the file stays the same. Have you ever worked in a code base with many contributors that changed over the course of years? In my experience it always ends up a jumble where indentation is screwed up and no particular tab setting makes things right. I've worked on files where different lines in the same file might assume tab spacing of 2, 3, 4, or 8.…

On the good side, the problem of mixing tabs and spaces does normally not appear in Go sources, as gofmt always converts spaces to tabs, so there is no inconsistant indentation. Normally I prefer spaces to tabs because I dislike the mixing, but gofmt solves this nicely for me.

Re: Why Discord is switching from Go to Rust

#127

Post by therockhead »

The_rationalist wrote: post #95
jhgg wrote: post #67

Earlier quoted context omitted.

This blog post perhaps is a bit "after the fact" we had made the switch over mid 2019, and wanted to try out rust as well for services like this, due to adoption elsewhere in the company. Also, after upgrading between 4 golang versions on this service and noticing it didn't materially change performance, we decided to just spend our time on the rewrite (for fun, and latency) and to get a head start into the asynchron…

Out of curiosity, why didn't you choose Kotlin? It can reuse the Java ecosystem which allow you to save tons of money, and give you advanced features and scalability. It is a sexier and more ergonomic language too. And with e.g ZGC, you can have a GC that is fine tunable, and that has very low latency. By choosing rust you will suffer a great deal of the limitations of it's poor, not production ready, ecosystem. I'm…

> By choosing rust you will suffer a great deal of the limitations of it's poor, not production ready, ecosystem.

Why do you think that? Seems like Rust is a great choice for this type of high performance work.

Re: Why Discord is switching from Go to Rust

#128

Post by The_rationalist »

Borrowed from a comment:

Garbage collection has gotten a lot of updates in the last 3 years. Why would you not take the exceedingly trivial step of just upgrading to the latest Go stable in order to at least try for the free win? From the go 1.12 release notes: “Go 1.12 significantly improves the performance of sweeping when a large fraction of the heap remains live. This reduces allocation latency immediately following a garbage collection.” ¯\_(ツ)_/¯ This sounds like “we just wanted to try Rust, ok?” Which is fine. But like, just say that.

Re: Why Discord is switching from Go to Rust

#129

Post by jrockway »

This seems like a nice microservices success story. It's so easy to replace a low-performing piece of infrastructure when it is just a component with a well-defined API. Spin up the new version, mirror some requests to see how it performs, and turn off the old one. No drama, no year-long rewrites. Just a simple fix for the component that needed it the most.

Re: Why Discord is switching from Go to Rust

#130

Post by loeg »

_bxg1 wrote: post #32
flafla2 wrote: post #16

> After digging through the Go source code, we learned that Go will force a garbage collection run every 2 minutes at minimum. In other words, if garbage collection has not run for 2 minutes, regardless of heap growth, go will still force a garbage collection. > We figured we could tune the garbage collector to happen more often in order to prevent large spikes, so we implemented an endpoint on the service to change…

It does sound like Discord's case was fairly extraordinary in terms of the degree of the spike: > We kept digging and learned the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire LRU cache in order to determine if the memory was truly free from references. So maybe this is one of those things that just doesn't come up in most cases?…

A GC scan of a large LRU (or any large object graph) is expensive in CPU terms because the many of the pointers traversed will not be in any CPU cache. Memory access latency is extremely high relative to how fast CPUs can process cached data.

You could maybe hack around the GC performance without destroying the aims of LRU eviction by batching additions to your LRU data structure to reduce the number of pointers by a factor of N. It's also possible that a Go BTree indexed by timestamp, with embedded data, would provide acceptable LRU performance and would be much friendlier on the cache. But it might also not have acceptable performance. And Go's lack of generic datastructures makes this trickier to implement vs Rust's BtreeMap provided out of the box.

Post reply on HN