Taming Go’s memory usage, or how we avoided rewriting our client in Rust • Hacker News Board

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#31

Post by slaymaker1907 » Tue, Sep 21, 2021, 7:40 PM UTC

miloignis wrote: post #20

angelzen wrote: post #16

Earlier quoted context omitted.

Tangentially, I did a bit of Rust work recently. I was sadly unable to find a concise credible answer to a rather elementary best-practices question: How does ownership interact with nested datastructures? Is it possible to build a heap tree without Boxing every node explicitly?

This question is a bit subtle, it depends on exactly what you mean. You could make a tree using only borrow checked references and the compiler would make sure that parent nodes go out of scope at the same time or before the child nodes they point to, but I don't think that's what you're talking about. In general, if it's a datastructure where you have to use pointers, you'll have them Box'ed, but you would try to av…

I never thought about using a Vec for these, but that is a great idea for keeping the memory management sane for tree/linked lists.

One thing I would add that you need to be wary of destructors with large pointer data structures in Rust since it can easily stack overflow. When using Option> you need to be careful to call Option::take on the pointers in a loop to avoid stack overflow.

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#32

Post by sreque » Tue, Sep 21, 2021, 7:41 PM UTC

olau wrote: post #23

wrs wrote: post #10

Buried in here are great examples of why rewrites don’t help: “The module that does this inference was recompiling those regular expressions each time it was asked to do the work.” “The reason for the allocation was a buffer holding decompressed data, before feeding it to a parser. …the output of the decompression could be fed directly into the parser, without any extra buffer.” The problem here isn’t that the langua…

> The problem here isn’t that the language has GC, it’s that memory usage was just not considered. While I agree with the gist of what you're saying, I do think runtimes based on the we'll-clean-it-up-some-day GC paradigm makes it more important to consider memory allocation than less laissez-faire paradigms (like RAII or reference counting), contrary to how it's presented in the glamorous brochures.

More importantly, GC'ed languages tend to use at least 2x the memory of un-GC'ed languages and have to deal with the consequences of GC-induced pauses and generally inferior native code interop. Whether that matters to you or not depends on your application. No one is going to use a GC'ed language in the Linux Kernel, but practically 100% of backend applications are written in GC'ed languages because the productivity benefits are of automatic memory management are massive.

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#33

Post by geodel » Tue, Sep 21, 2021, 7:41 PM UTC

Well good for author that they were able to fix the issue.

However I think writing efficient code in even in managed memory languages for large, heavily used service is kind of normal thing and not above and beyond normal work.

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#34

Post by tptacek » Tue, Sep 21, 2021, 7:42 PM UTC

jjoonathan wrote: post #28

wrs wrote: post #10

Buried in here are great examples of why rewrites don’t help: “The module that does this inference was recompiling those regular expressions each time it was asked to do the work.” “The reason for the allocation was a buffer holding decompressed data, before feeding it to a parser. …the output of the decompression could be fed directly into the parser, without any extra buffer.” The problem here isn’t that the langua…

Right, but GC encourages you to not think about memory at all until the program starts tipping over and fixing the underlying cause of the leak now requires an architecture change because the "we hold onto everything" assumption got baked into the structure in 2 places that you know about and 5 that you don't. I don't miss the rote parts of manual memory management, but it had the enormously beneficial side effect of…

Plenty of C programs do the equivalent of ioutil.ReadAll; it's not a GC thing.

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#35

Post by titzer » Tue, Sep 21, 2021, 7:43 PM UTC

jjoonathan wrote: post #28

wrs wrote: post #10

Buried in here are great examples of why rewrites don’t help: “The module that does this inference was recompiling those regular expressions each time it was asked to do the work.” “The reason for the allocation was a buffer holding decompressed data, before feeding it to a parser. …the output of the decompression could be fed directly into the parser, without any extra buffer.” The problem here isn’t that the langua…

Right, but GC encourages you to not think about memory at all until the program starts tipping over and fixing the underlying cause of the leak now requires an architecture change because the "we hold onto everything" assumption got baked into the structure in 2 places that you know about and 5 that you don't. I don't miss the rote parts of manual memory management, but it had the enormously beneficial side effect of…

GC will not fix trashy programming. The problem is that many GC'd languages have adopted a style guide that commits to a lot of unnecessary allocations. For example, in Java, you can't parse an integer out of the middle of a string without allocating in-between. Ditto with lots of other common operations. Java has oodles of trashy choices. With auto-boxing, allocations are hidden. Without reified (let's say, type-specialized) generics, all the collection classes carry extra overhead for boxing values.

I write almost all of my code in Virgil these days. It is fully garbage-collected but nothing forces you into a trashy style. E.g. I use (and reuse) StringBuilders, DataReaders, and TextReaders that don't create unnecessary intermediate garbage. It makes a big difference.

Sometimes avoiding allocation means reusing a data structure and "resetting" or clearing its internal state to be empty. This works if you are careful about it. It's a nightmare if you are not careful about it.

I'm not going back to manual memory management, and I don't want to think about ownership. So GC.

edit: Java also highly discourages reimplementing common JDK functionality, but I've found building a customized datastructure that fits exactly my needs (e.g. an intrusive doubly-linked list) can work wonders for performance.

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#36

Post by jerf » Tue, Sep 21, 2021, 7:43 PM UTC

olau wrote: post #23

wrs wrote: post #10

Buried in here are great examples of why rewrites don’t help: “The module that does this inference was recompiling those regular expressions each time it was asked to do the work.” “The reason for the allocation was a buffer holding decompressed data, before feeding it to a parser. …the output of the decompression could be fed directly into the parser, without any extra buffer.” The problem here isn’t that the langua…

> The problem here isn’t that the language has GC, it’s that memory usage was just not considered. While I agree with the gist of what you're saying, I do think runtimes based on the we'll-clean-it-up-some-day GC paradigm makes it more important to consider memory allocation than less laissez-faire paradigms (like RAII or reference counting), contrary to how it's presented in the glamorous brochures.

Put it this way: Each of the things mentioned in that post were errors that could just as easily have been made in Rust, and Rust would not necessarily have helped avoid. At best you can make a case for the errors being more explicit, but in my personal experience even that would be weak.

The last error in particular, using byte buffers instead of a streaming abstraction, is pervasive in programming. I don't know if Rust is necessarily any worse than Go's library environment for dealing with that problem but I doubt it's any better. By having io.Reader in the standard library from the beginning (and not because of any other particular virtue of the language, IMHO) it has had one of the best ecosystems for dealing with streams without having to manifest them as full bytes around [1].

It amounts to, the root problem is that they didn't have the problem they thought they have. Rust will blow the socks off the competition w.r.t. memory efficiency of lots of small objects, which is why it's so solid in the browser space. But that's not the problem they were having. Go's just fine where they seem to have ultimately ended up, stream processing things with transient per-object processing. Even if you do some allocation in the processing, the GC ends up not being a big deal because the runs end up scanning over not much memory not all that frequently. This is why Go is so popular in network servers. Could Rust do better? Yes. Absolutely, beyond a shadow of a doubt. But not enough to matter, in a lot of cases.

[1]: An expansion on that thought if you like: https://news.ycombinator.com/item?id=28368080

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#37

Post by tsimionescu » Tue, Sep 21, 2021, 7:45 PM UTC

arsome wrote: post #17

henning wrote: post #14

> Rust has manual memory management, which means that whenever we’re writing code we’ll have to take the time to manage memory ourselves. No.

Yeah, sounds like someone doesn't understand lifetimes and RAII. Even in modern C++ the number of times you have to actually think about memory management instead of lifetimes is basically zero unless you have to work with old libraries.

But thinking about lifetimes and RAII is 90% of memory management.

Basically whether you write C, C++, or Rust, you have to track ownership the same ways, the only thing that changes is how much the compiler helps you with that. However, if you write your program in Java, Lisp or Haskell, you simply do not care about ownership for memory-only objects, and can structure your program significantly differently.

This can have significant impact on certain types of workflows, especially when it comes to shared objects. A well-known example is when implementing lock-free data structures based on compare-and-swap, where you need to free the old copy of the structure after a successful compare-and-swap; but, you can't free it since you don't know who may still be reading from it. Here is an in-depth write-up from Andrei Alexandrescu on the topic [0].

Note: I am using "object" here in the sense from C - basically any piece of data that was allocated.

[0] http://erdani.org/publications/cuj-2004-10.pdf

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#38

Post by silisili » Tue, Sep 21, 2021, 7:48 PM UTC

xondono wrote: post #15

wrs wrote: post #10

Buried in here are great examples of why rewrites don’t help: “The module that does this inference was recompiling those regular expressions each time it was asked to do the work.” “The reason for the allocation was a buffer holding decompressed data, before feeding it to a parser. …the output of the decompression could be fed directly into the parser, without any extra buffer.” The problem here isn’t that the langua…

> Buried in here are great examples of why rewrites don’t help That has not been my experience. Rewrites do sometimes help, because in a lot of codebases there’s too many “pet” modules or badly designed frozen interfaces. Rewrites can help in those situations, because there’s no sacred cows anymore. The issue is that a lot of people do rewrites as translations, without touching structures.

Agreed with this 100%.

So many posts here over the years of examples of 'how we rewrote from x to y and saw 2000% gains', where x and y are languages. Such examples are 100% meaningless. Rewrites from the ground up -should- always be way faster, since it's all greenfield. If trying to make a language comparison, rewrite the entire thing in both languages!

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#39

Post by tptacek » Tue, Sep 21, 2021, 7:50 PM UTC

The big wins in this article, in what I believe was the order of impact:

* They do raw packet reassembly using gopacket, and gopacket keeps TCP reassembly buffers that can grow without bound when you miss a TCP segment. They capped the buffers, and the huge 5G spikes went away.

* They were reading whole buffers into memory before handing them off to YAML and JSON parsers. They passed readers instead.

* They were using a protobuf diffing library that used `reflect` under the hood, which allocates. They generated their own explicit object inspection thingies.

* They stopped compiling regexps on the fly and moved the regexps to package variables. (I actually don't know if this was a significant win; there might just be the three big wins.)

This is a great article. But none of these seem Go-specific†, or even GC-specific. They're doing something really ambitious (slurping packets up off the wire against busy API servers, reassembling them in userland into streams, and then parsing the contents of the streams). Memory usage was going to be fiddly no matter what they built with. The problems they ran up against seem pretty textbook.

Frankly I'm surprised Go acquitted itself as well as it did here.

† Maybe the perils of `reflect` count as a Go thing; it's worth noting that there's folk wisdom in Go-land to avoid `reflect` when possible.

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#40

Post by typical182 » Tue, Sep 21, 2021, 7:50 PM UTC

Very nice write up.

Go’s focus on simplicity means that there is only a single parameter, SetGCPercent, which controls how much larger the heap is than the live objects within it.

FWIW, there is a new proposal from a member of the core Go team to add a second GC knob in the form of a soft limit on total memory:

https://github.com/golang/proposal/blob/master/design/48409-...

It includes some provisions to make sure that the application can keep making progress and avoid death spirals (part of the reason why it is a "soft" limit), and also includes some new GC-related telemetry.

From the blog write up, a second GC knob with a soft limit might have only been a minor help here, with the bigger wins coming from the code changes they described in the blog.