Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Discord is switching from Go to Rust

blog.discordapp.com

Post reply on HN
Print view View original

231–240 of 670 posts

Re: Why Discord is switching from Go to Rust

#231

Post by masklinn »

asimpletune wrote: post #226
rvcdbn wrote: post #157

Seems like you were hitting: runtime: Large maps cause significant GC pauses #9477 [0] Looks like this issue was resolved for maps that don't contain pointers by [1]. From the article, sounds like the map keys were strings (which do contain pointers, so the map would need to be scanned by the GC). If pointers in the map keys and values could be avoided, it would have (if my understanding is correct) removed the need…

Ok but in rust those pointers can just be borrowed obviating the need for gc at all.

Given it's a cache the entries would not have an existing natural owner… except for the cache itself.

There would be no need for a GC to traverse the entire map, but that's because rust doesn't use a GC.

Re: Why Discord is switching from Go to Rust

#232

Post by nemothekid »

cies wrote: post #210
deweller wrote: post #179

Earlier quoted context omitted.

As an outsider to both the Go and Rust cultures, I read the Actix news and walked away with the impression that the Rust ecosystem is less mature.

Go's is more pragmatic. Rust's is more purist, and that reflects on the language features (more functional, more free in allowing you to use it for any purpose where Go is network-app specific, more strict in typing), the licensing and the attitude towards collaboration. That collaboration thing is why Actix exploded I think. While mostly an isolated incident it does show some clash between the author's values (and p…

The Go community has a very similar story, where someone released a web framework, with an unorthodox set of features, and was flamed to the point where he abandoned the project and quit OSS.

https://github.com/go-martini/martini

Re: Why Discord is switching from Go to Rust

#233

Post by nemothekid »

dennisgorelik wrote: post #228

> Changing to a BTreeMap instead of a HashMap in the LRU cache to optimize memory usage. Why would BTreeMap be faster than HashMap? HashMap performance is O(1), while BTreeMap performance is O(log N).

1. They never said it was faster, only that memory usage was better. Regardless, it could be the case that log N 2. Memory usage on a hash map would be worse especially if the fill ratio is relatively low.

Re: Why Discord is switching from Go to Rust

#234

Post by PeterCorless »

masklinn wrote: post #217
jasondclinton wrote: post #214

Earlier quoted context omitted.

Finding out if that does resolve the author's issue would be interesting but I'm not sure that that would be particularly supportive data in favor of Go. If anything it would reinforce the downsides of Go's GC implementation: prone sudden pitfalls only avoidable with obtuse, error-prone fiddling that makes the code more complex. After spending weeks fighting with Java's GC tuning for a similar production service tail…

The good news are that Go's GC has basically no tunables, so you wouldn't have spent weeks on that. The bad news is that it has basically no tunables so if it's a tuning issue you're either fucked or have to put "tuning" hacks right into the code if you find any that works (e.g. twitch's "memory ballast" to avoid overly aggressive GC runs: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i... )

There are tradeoffs with all languages. C++ avoids the GC, but you then have to make sure you know how to avoid the common pitfalls of that language.

We use C++ at Scylla (saw that we got a shout-out in the blog! Woot!) but it's not like there isn't a whole industry about writing blogs avoiding C++ pitfalls.

C++ pitfalls through the years... • https://www.horstmann.com/cpp/pitfalls.html (1997) • https://stackoverflow.com/questions/30373/what-c-pitfalls-sh... (2008) • http://blog.davidecoppola.com/2013/09/cpp-pitfalls/ (2013) • https://www.typemock.com/pitfalls-c/ (2018)

I am not saying any of these (Go, Rust, C++, or even Java) are "right" or "wrong" per se, because that determination is situational. Are you trying to optimize for performance, for code safety, for taking advantage of specific OS hooks, or oppositely, to be generically deployable across OSes, or for ease of development? For the devs at Scylla, the core DB code is C++. Some of our drivers and utilities are Golang (like our shard aware driver). There's also a Cassandra Rust driver — it'd be sweet if someone wants to make it shard-aware for Scylla!

Re: Why Discord is switching from Go to Rust

#235

Post by rvcdbn »

jasondclinton wrote: post #214
rvcdbn wrote: post #157

Seems like you were hitting: runtime: Large maps cause significant GC pauses #9477 [0] Looks like this issue was resolved for maps that don't contain pointers by [1]. From the article, sounds like the map keys were strings (which do contain pointers, so the map would need to be scanned by the GC). If pointers in the map keys and values could be avoided, it would have (if my understanding is correct) removed the need…

Finding out if that does resolve the author's issue would be interesting but I'm not sure that that would be particularly supportive data in favor of Go. If anything it would reinforce the downsides of Go's GC implementation: prone sudden pitfalls only avoidable with obtuse, error-prone fiddling that makes the code more complex. After spending weeks fighting with Java's GC tuning for a similar production service tail…

For any tracing GC, costs are going to be proportional to the number of pointers that need to be traced. So I would not call reducing the use of pointers to ameliorate a GC issue "obtuse, error-prone fiddling". On the contrary, it seems like one of the first approaches to look at when faced with the problem of too much GC work.

Really all languages with tracing GC are at a disadvantage when you have a huge number of long-lived objects in the heap. The situation is improved with generational GC (which Go doesn't have) but the widespread use of off-heap data structures to solve the problem even in languages like Java with generational GC suggests this alone isn't a good enough solution.

In Go's defense, I don't know another GC'ed language in which this optimization is present in the native map data structure.

Re: Why Discord is switching from Go to Rust

#236

Post by masklinn »

steveklabnik wrote: post #230
ithkuil wrote: post #221

Earlier quoted context omitted.

Perhaps such a library could help you record the location of the variables that contain pointers to the strings and keep that pointer up to data as the ownership of the string moves from variable to variable? I'm other words, doing some of the work a moving compacting collector would do during compaction but continuously during normal program execution.

There's no way to hook into the move, so I don't see how it would be possible, or at least, not with techniques similar to compacting GCs.

Maybe by reifying the indirection? The compacting arena would hand out smart pointers which would either always bounce through something (to get from an indentity to the actual memory location, at a cost) or it'd keep track and patch the pointers it handed out somehow.

Possibly half and half, I don't remember what language it was (possibly obj-c?) which would hand out pointers, and on needing to move the allocations it'd transform the existing site into a "redirection table". Accessing pointers would check if they were being redirected, and update themselves to the new location if necessary.

edit: might have been the global refcount table? Not sure.

Re: Why Discord is switching from Go to Rust

#237

Post by robocat »

Nyra wrote: post #50
flafla2 wrote: post #16

> After digging through the Go source code, we learned that Go will force a garbage collection run every 2 minutes at minimum. In other words, if garbage collection has not run for 2 minutes, regardless of heap growth, go will still force a garbage collection. > We figured we could tune the garbage collector to happen more often in order to prevent large spikes, so we implemented an endpoint on the service to change…

Funnily enough, something similar happened at Twitch regarding their API front end written in Go: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...

summary: Go 1.5, memory usage (heap) of 500MB on a VM with 64GiB of physical memory, with 30% of CPU cycles spent in function calls related to GC, and unacceptable problems during traffic spikes. Optimisation hack to somewhat fix problem was to allocate 10GiB, but not using the allocation at all, which caused a beneficial change in the GC behaviour!

Re: Why Discord is switching from Go to Rust

#238

Post by outworlder »

hinkley wrote: post #224
jasondclinton wrote: post #214

Earlier quoted context omitted.

Finding out if that does resolve the author's issue would be interesting but I'm not sure that that would be particularly supportive data in favor of Go. If anything it would reinforce the downsides of Go's GC implementation: prone sudden pitfalls only avoidable with obtuse, error-prone fiddling that makes the code more complex. After spending weeks fighting with Java's GC tuning for a similar production service tail…

The common factor in most of my decisions to look for a new job has been realizing that I feel like a very highly compensated janitor instead of a developer. Once I spend even the plurality of my time cleaning up messes instead of doing something new (and there are ways to do both), then all the life is sucked out of me and I just have to escape. Telling me that I have to keep using a tool with known issues that we h…

While I completely agree with the "janitor" sentiment... and for Newton's sake I feel like Wall-E daily...

> Telling me that I have to keep using a tool with known issues that we have to process or patches to fix would be super frustrating.

All tools have known issues. It's just that some have way more issues than others. And some may hurt more than others.

Go has reached an interesting compromise. It has some elegant constructs and interesting design choices (like static compilation which also happens to be fast). The language is simple, so much so that you can learn the basics and start writing useful stuff in a weekend. But it is even more limiting than Java. A Lisp, this thing is not. You can't get very creative – which is an outstanding property for 'enterprises'. Boring, verbose code that makes you want to pull your teeth out is the name of the game.

And I'm saying this as someone who dragged a team kicking and screaming from Python to Go. That's on them – no-one has written a single line of unit tests in years, so now they at least get a whiny compiler which will do basic sanity checks before things blow up in prod. Things still 'panic', but less frequently.

Re: Why Discord is switching from Go to Rust

#239

Post by typical182 »

kerkeslager wrote: post #163
biomcgary wrote: post #54

Earlier quoted context omitted.

I know latency for GC with large heaps improved in Go1.12. See: https://golang.org/doc/go1.12#runtime

The article states specifically that part of the problem was heaps were never large. EDIT: Actually, no it didn't, I misunderstood it.

Where does it say that?

It says things like:

“We were not creating a lot of garbage.”

... but that statement there doesn’t say anything about the heap size, including the size and count of live objects (i.e., not garbage).

It also says:

“There are millions of Users in each cache. There are tens of millions of Read States in each cache.”

Large is often in the eye of the beholder, but I missed it if it said anything specifically about not having a large heap size.

Re: Why Discord is switching from Go to Rust

#240

Post by bearcherian »

rvcdbn wrote: post #157

Seems like you were hitting: runtime: Large maps cause significant GC pauses #9477 [0] Looks like this issue was resolved for maps that don't contain pointers by [1]. From the article, sounds like the map keys were strings (which do contain pointers, so the map would need to be scanned by the GC). If pointers in the map keys and values could be avoided, it would have (if my understanding is correct) removed the need…

The article also mentions the service was on go 1.9.2, which was released 10/2017. I'd be curious to see if the same issues exist on a build based on a more recent version of Go.
Post reply on HN