> But our profile wasn’t ever showing us 500GB of live data, just a little bit more than 200MB in the worst cases. This suggested to me that we’d done all we could with live objects. Is this a typo? Weren't seeing 500 MB of live data, just a little more than 200MB in the worst case? EDIT: Btw, I read the entire article. It was fascinating, thank you!
Taming Go’s memory usage, or how we avoided rewriting our client in Rust
11–20 of 231 posts
Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust
#12Bit confused by this part of the article: > PRO-REWRITE: Rust has manual memory management, so we would avoid the problem of having to wrestle with a garbage collector because we would just deallocate unused memory ourselves, or more carefully be able to engineer the response to increased load. > ANTI-REWRITE: Rust has manual memory management, which means that whenever we’re writing code we’ll have to take the time…
Rust = when writing your code, you explicitly describe the ownership and lifetime of your objects and how your functions are allowed to consume/copy etc. them and get safety as a result
C = when writing your code, you explicitly allocate and free your objects and you get no assistance from the language about when it is safe to copy/dereference/free/etc. a pointer/allocation
Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust
#13Buried in here are great examples of why rewrites don’t help: “The module that does this inference was recompiling those regular expressions each time it was asked to do the work.” “The reason for the allocation was a buffer holding decompressed data, before feeding it to a parser. …the output of the decompression could be fed directly into the parser, without any extra buffer.” The problem here isn’t that the langua…
Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust
#14No.
Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust
#15Buried in here are great examples of why rewrites don’t help: “The module that does this inference was recompiling those regular expressions each time it was asked to do the work.” “The reason for the allocation was a buffer holding decompressed data, before feeding it to a parser. …the output of the decompression could be fed directly into the parser, without any extra buffer.” The problem here isn’t that the langua…
That has not been my experience. Rewrites do sometimes help, because in a lot of codebases there’s too many “pet” modules or badly designed frozen interfaces.
Rewrites can help in those situations, because there’s no sacred cows anymore. The issue is that a lot of people do rewrites as translations, without touching structures.
Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust
#16Bit confused by this part of the article: > PRO-REWRITE: Rust has manual memory management, so we would avoid the problem of having to wrestle with a garbage collector because we would just deallocate unused memory ourselves, or more carefully be able to engineer the response to increased load. > ANTI-REWRITE: Rust has manual memory management, which means that whenever we’re writing code we’ll have to take the time…
I feel that this is one of those common misconceptions about Rust. Rust's memory management is nothing like C or non-modern C++'s with malloc/free or new/delete. Rust uses modern-C++'s RAII model, typically, to allocate memory. The compiler is smart enough to know when to call drop() (which is essentially free/delete, but with the possibility of additional behavior). You can also call drop() yourself. What I think pe…
Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust
#17> Rust has manual memory management, which means that whenever we’re writing code we’ll have to take the time to manage memory ourselves. No.
Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust
#18Earlier quoted context omitted.
I feel that this is one of those common misconceptions about Rust. Rust's memory management is nothing like C or non-modern C++'s with malloc/free or new/delete. Rust uses modern-C++'s RAII model, typically, to allocate memory. The compiler is smart enough to know when to call drop() (which is essentially free/delete, but with the possibility of additional behavior). You can also call drop() yourself. What I think pe…
Tangentially, I did a bit of Rust work recently. I was sadly unable to find a concise credible answer to a rather elementary best-practices question: How does ownership interact with nested datastructures? Is it possible to build a heap tree without Boxing every node explicitly?
Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust
#19> Rust has manual memory management, which means that whenever we’re writing code we’ll have to take the time to manage memory ourselves. No.
Yeah, sounds like someone doesn't understand lifetimes and RAII. Even in modern C++ the number of times you have to actually think about memory management instead of lifetimes is basically zero unless you have to work with old libraries.
Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust
#20Earlier quoted context omitted.
I feel that this is one of those common misconceptions about Rust. Rust's memory management is nothing like C or non-modern C++'s with malloc/free or new/delete. Rust uses modern-C++'s RAII model, typically, to allocate memory. The compiler is smart enough to know when to call drop() (which is essentially free/delete, but with the possibility of additional behavior). You can also call drop() yourself. What I think pe…
Tangentially, I did a bit of Rust work recently. I was sadly unable to find a concise credible answer to a rather elementary best-practices question: How does ownership interact with nested datastructures? Is it possible to build a heap tree without Boxing every node explicitly?
In general, if it's a datastructure where you have to use pointers, you'll have them Box'ed, but you would try to avoid that if you can. In your example of a heap, you'd want to use an array-based implementation, probably backed by a growable Vec, and use indexes internally. A peek function would still return a normal Rust reference to the data, and the borrow checker would make sure that you don't mutate the heap's backing array while that reference was still in use, etc.