Live data from Hacker News

Why Discord is switching from Go to Rust

blog.discordapp.com

611–620 of 670 posts

Re: Why Discord is switching from Go to Rust

#611
post #235

Earlier quoted context omitted.

Finding out if that does resolve the author's issue would be interesting but I'm not sure that that would be particularly supportive data in favor of Go. If anything it would reinforce the downsides of Go's GC implementation: prone sudden pitfalls only avoidable with obtuse, error-prone fiddling that makes the code more complex. After spending weeks fighting with Java's GC tuning for a similar production service tail…

For any tracing GC, costs are going to be proportional to the number of pointers that need to be traced. So I would not call reducing the use of pointers to ameliorate a GC issue "obtuse, error-prone fiddling". On the contrary, it seems like one of the first approaches to look at when faced with the problem of too much GC work. Really all languages with tracing GC are at a disadvantage when you have a huge number of…

Except that plenty of languages with tracing GC have also off GC memory allocation.

Since you mention not knowing such languages, have a look at Eiffel, D, Modula-3, Active Oberon, Nim, C#/F# (specially after the latest improvements).

Also Java will eventually follow the same idea as Eiffel (where inline classes are similar to expanded classes in Eiffel), and ByteBuffers can be off-GC heap.

Re: Why Discord is switching from Go to Rust

#612
post #157

Seems like you were hitting: runtime: Large maps cause significant GC pauses #9477 [0] Looks like this issue was resolved for maps that don't contain pointers by [1]. From the article, sounds like the map keys were strings (which do contain pointers, so the map would need to be scanned by the GC). If pointers in the map keys and values could be avoided, it would have (if my understanding is correct) removed the need…

Everything I've read indicates that RAM caches work poorly in a GC environment. The problem is that garbage collectors are optimized for applications that mostly have short-lived objects, and a small amount of long-lived objects. Things like large in-RAM LRU are basically the slowest thing for a garbage collector to do, because the mark-and-sweep phase always has to go through the entire cache, and because you're con…

Which is why on GC languages that also support value types and off GC-heap allocations, one makes use of them, instead of throwing out the baby with the water.

Re: Why Discord is switching from Go to Rust

#613

Earlier quoted context omitted.

There's no way to hook into the move, so I don't see how it would be possible, or at least, not with techniques similar to compacting GCs.

Maybe by reifying the indirection? The compacting arena would hand out smart pointers which would either always bounce through something (to get from an indentity to the actual memory location, at a cost) or it'd keep track and patch the pointers it handed out somehow . Possibly half and half, I don't remember what language it was (possibly obj-c?) which would hand out pointers, and on needing to move the allocations…

I would give out handles and have a Guard Object, that allows you to get smart pointers from handles as long as Guard Object is in scope. Then when Guard Object is out of scope, the smart pointers would get invalidated.

Re: Why Discord is switching from Go to Rust

#614
post #482
post #473

Earlier quoted context omitted.

"The article says that the data is basically "per-user"," Given that this is a table of who is "online", I don't think that's per-user in the sense that you are inferring. I infer that it's not a whole bunch of little local data that doesn't interact, it's a big global table of who is online and not online, constantly being heavily read from and written to in real time. Consider from the perspective of Bob's Erlang p…

Yeah the scale is what makes this problem a problem here. I've done exactly that "online" stuff per user process and it works fine on a small scale, even when it needs to be globally inferred. But I suspect it'd quickly become the bottleneck when scaling. I had no idea mnesia was that fragile though, what gives? What kind of issues did you encounter with it? What do you use now to solve those issues with Erlang/Elixi…

"I had no idea mnesia was that fragile though, what gives? What kind of issues did you encounter with it? What do you use now to solve those issues with Erlang/Elixir?"

I had ~10,000 devices in the field with unique identifiers creating long-term, persistent connections to a central cluster. An mnesia table stored basically $PERSISTENT_ID -> PID they are connected to. It needed to be updated when they connected and disconnected, which let me emphasize was a relatively rare occurrence; the ideal system would be connected for days at a time, not connecting & disconnecting dozens of times a minute. At most, reconnection flurries might occasionally occur where they'd all try to connect over the course of a few minutes (they had backoff code built in) if the cluster was down for some reason.

Mnesia fell over. A lot. All I could find online as an explanation was basically "yeah, don't do that with mnesia". Bizarrely, it wasn't the connection flurries that did it, either... it was the normal "maybe a few dozen events per second" that tended to do it. Erlang itself was usually fine. (Although for machines right next to each other in a rack, I did lose the clustering more often than I'd like, and have to hit the REPL to re-associated nodes together. Much less often than mnesia corrupted itself, though.)

"can you really say with a straight face that Rust comes with all the same features as BEAM out-of-the-box?"

Well, that's another way of looking at what I was trying to say. That's the wrong question. Rust doesn't need "all the same features as BEAM". Rust needs "the features necessary to do the work". While the Erlang community is looking for a language that has "all the same features as BEAM" and smugly congratulating themselves that no other language seems to have cracked that yet, a number of languages are passing them by by implementing different features. Many of those languages, as I said, are informed by Erlang. Many of these new languages are choosing their "not exactly like Erlang" features in knowledge, not ignorance, as I think the Erlang community thinks.

Besides, Erlang builds in a lot of things that can be libraries in other languages. I built the replacement in Go. Mostly because it was hard to get people who wanted to work in Erlang but despite the rage on HN anytime Go comes up, getting people who are willing to work in Go was trivial even 5 years ago. (Hiring someone who knows Go already is still a bit of a challenge, but crosstraining someone into it is easy.) For the port, I wrote https://github.com/thejerf/reign . You will look at it and go "But Erlang has this and that and the other thing with its clustering, and your thing doesn't have those things!" And my response is twofold: First, that some of those things are supported in Go code in other ways than what you are expecting, and that was not intended to be "Erlang in Go" but "a library for helping port Erlang programs into Go without rearchitecting", and second... the resulting cluster has been more reliable and more performant (we actually cut the cluster from 4 to 2, because now even a single machine can handle the entire load), and all the "features" reign is missing, well, maybe they aren't so important out of the context of Erlang. I suppose in my own way this is another sort of story like Discord's; on the metrics I care about, my home-grown clustering library worked better for me than Erlang's clustering code.

(In fact, Go's even got the edge on Erlang for GC for my use case, which is one of the ways in which the new system is more performant. Now, it happens that my system is architected on sending around messages that may frequently be several megabytes in size, and Erlang was really designed for sending around lots of messages in the kilobyte range. Even as I was using it, Erlang got a lot better with handling that, but it still was never as good or fast as Go, and Go's only gotten better since then, too. I was able to do things in Go for performance to re-use my buffers that are impossible in Erlang.)

So, I mean, while I do deeply respect Erlang for its pioneering position, and I am particularly grateful for the many years I spent with it back when it was the only option of its sort (if I had to write the project in question in C++ or something, I just wouldn't have; do not think I "hate" Erlang or something, I am very grateful for it), if I am a bit less starry-eyed about it than some it's because I see it as... just code. It's just code. Erlang gets no special access to CPU instructions or special Erlang-only hardware that allows it to do things no other language can. It's just code. Code that can be and has been written in other languages, in other environments.

I like Erlang in a lot of ways, and respect its place in history. But it's community is insular, maybe even a bit sick, and I don't really expect that to change, because once an individual realizes it, they tend to just leave, leaving behind only the True Believers, who still believe that Erlang is the unique and special snowflake... that it was... 15 years ago.

Re: Why Discord is switching from Go to Rust

#615

Earlier quoted context omitted.

Maybe I've missed this, but why do they need a particularly large LRU cache? Surely this isn't all one process, so presumably they could reduce spikes by splitting the same load across yet more processes?

Larger cache = faster performance and less load on the database. I only glossed over the article but the problem they had with Go seems to be the GC incurred from having a large cache. Their cache eviction algorithm was efficient, but every 2 minutes there was a GC run which slowed things down. Re-implementing this algorithm in Rust gave them better performance because the memory was freed right after the cache evict…

I am of course talking about the same amount of total cache RAM, just split among more processes. Depending on distribution of the calls, you might get more cache misses, but I don't think it's guaranteed, and if it is, I don't think we can assume it's significant. Heck, you could even use more cache RAM; the cost of a total rewrite plus ongoing maintenance in a new language covers a fair bit of hardware these days.

Re: Why Discord is switching from Go to Rust

#616
Usually this kind of article is about "migrating from massively popular language to more niche language that we like better".

This is more from niche to niche. Thought that was interesting, but yet the discussion here wasn't all that different to the usual. Guess it's flamewars always, regardless of popularity.

Re: Why Discord is switching from Go to Rust

#617

Earlier quoted context omitted.

Finding out if that does resolve the author's issue would be interesting but I'm not sure that that would be particularly supportive data in favor of Go. If anything it would reinforce the downsides of Go's GC implementation: prone sudden pitfalls only avoidable with obtuse, error-prone fiddling that makes the code more complex. After spending weeks fighting with Java's GC tuning for a similar production service tail…

The common factor in most of my decisions to look for a new job has been realizing that I feel like a very highly compensated janitor instead of a developer. Once I spend even the plurality of my time cleaning up messes instead of doing something new (and there are ways to do both), then all the life is sucked out of me and I just have to escape. Telling me that I have to keep using a tool with known issues that we h…

It helps if your think gardener instead of janitor.

Re: Why Discord is switching from Go to Rust

#618
post #597

Earlier quoted context omitted.

That's not correct. The implementation of "make", for example, looks like Go but isn't - it relies on internal details of the gc compiler that isn't part of the spec [1]. That's why a Go user can't implement "make" in Go. [1] https://golang.org/ref/spec

In which language do you think is "make" implemented?

If I may interject: I believe you are both trying to make orthogonal points. calcifer's is trying to say that some features of Go are compiler intrinsics, and cannot be implemented as a library. You are making a different point, which is that those intrinsics are implemented in Go, the host language. Both statement can be true at the same time, but I agree that the terms were not used entirely accurately, causing confusion.

Re: Why Discord is switching from Go to Rust

#619

Earlier quoted context omitted.

I write Java during the day but wish I could write Rust instead. Rust has a much nicer standard library, and the semantics and abstractions are so incredibly beautiful. It's a language that has learned from the millions of human-years that went into other languages and ecosystems. Every corner and seam in the language design speaks to this. Traits, union type enums, pattern matching, option and result types, derive()…

Rust has its share of random-feeling nonsense, like requiring PhantomData to "work around" unused type params, or that you can't compare arrays longer than 32 elements.

Which language doesn't.

Re: Why Discord is switching from Go to Rust

#620
post #227

Earlier quoted context omitted.

Yeah, they already weren't allocating, it was a GC pause that just scanned and would come up with essentially no extra garbage every two minutes.

A pool allocator could have reduced the number of existing allocations (1 big one instead of many small ones), making those spikes less significant. (But that depends on how Go handles interior pointers and GC, so I'm not sure.)

It has to track interior pointers. The problem for them seems to be the mark phase, where the GC has to track down all the pointers either way.
Post reply on HN