Over allocating maps and arrays in standard libs is not really a memory "leak" . Other langs do it.
[flagged]
Please go read Wikipedia: https://en.wikipedia.org/wiki/Memory_safety.
Identifying Rust's collect: >() memory leak footgun
11–20 of 129 posts
Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun
#12Over allocating maps and arrays in standard libs is not really a memory "leak" . Other langs do it.
After skimming the first few paragraphs, I almost commented something similar to yours. But after reading it until the end, the problem is more nuanced than that.
The issue is more about reusing memory that has been allocated when you "convert" (in Rust term: into_iter()) a Vec to another Vec. More surprisingly, the reuse still happens even when the new Vec has a different type than the original Vec's type. You should read the whole post—it's more interesting than I (and you) thought.
Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun
#13Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun
#14Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun
#15This is a good time to check how your code performs in beta vs stable. This particular case is new in beta and it would be interesting to catch regressions before they land.
Someone already filed this as a bug: https://github.com/rust-lang/rust/issues/120091
Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun
#16Over allocating maps and arrays in standard libs is not really a memory "leak" . Other langs do it.
You’re right, but it’s (I agree; incorrectly) been called a memory leak for a long time, to inappropriately over allocate/not free resources.
Coming from my java days, they called this a memory leak there too.
It’s definitely a more inappropriate term in rust, where memory leak has a definite meaning already.
Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun
#17Cross posting my comment from reddit[1] because I think it's interesting. ----- Nice post. I love calling attention to this. Just a few months ago, I ran into the ~same~ similar problem, although it wasn't caused by `collect()`. It was caused by "normal" `Vec` usage: https://github.com/BurntSushi/aho-corasick/commit/474393be8d... The issue appeared when building large Aho-Corasick automatons. Otherwise, it usually do…
> Just a few months ago, I ran into the same problem, although it wasn't caused by `collect()`. It was caused by "normal" `Vec` usage
If it wasn't caused by `collect()`, then I suppose it's a related problem, but not the same problem. Your problem was caused by, as your commit message says, "ignorance [original: ignorant] of the broader effects of this strategy [double-when-full] in aggregate"
The OP's problem, otoh, is more about reusing memory that has been allocated when you "convert" (in Rust term: into_iter()) a Vec to another Vec. What's more, the reuse also happens even when the new Vec has a different type from the original Vec's type. This behavior is more surprising than the double-when-full strategy, which is more widely known by programmers (even if sometimes they "forget" about the implications).
Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun
#18Cross posting my comment from reddit[1] because I think it's interesting. ----- Nice post. I love calling attention to this. Just a few months ago, I ran into the ~same~ similar problem, although it wasn't caused by `collect()`. It was caused by "normal" `Vec` usage: https://github.com/BurntSushi/aho-corasick/commit/474393be8d... The issue appeared when building large Aho-Corasick automatons. Otherwise, it usually do…
What we need is a page-based Vec that mmaps (anon) for the storage but leaves the unused portions zero-bytes and therefore not part of RSS until actually required. (And when clearing/shrinking sections, madvise DONTNEED the pages).
That is, the vec could expand to areas much larger than the actual used size, but this would have no effect on process RSS until those pages get dirtied with actual data.
Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun
#19Over allocating maps and arrays in standard libs is not really a memory "leak" . Other langs do it.
Conceptually, collect seems to create a new vector. In that case you would expect that the memory of the old vector is freed. The library tries to be smart and reuses the old vector as an (undocumented?) optimization. So the old memory that is no longer needed is not released.
Whether you call that a leak or not, Rust is known for its predictability. Allocating 200x the memory which a programmer would expect if he doesn't know the implementation details is bad.
Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun
#20Cross posting my comment from reddit[1] because I think it's interesting. ----- Nice post. I love calling attention to this. Just a few months ago, I ran into the ~same~ similar problem, although it wasn't caused by `collect()`. It was caused by "normal" `Vec` usage: https://github.com/BurntSushi/aho-corasick/commit/474393be8d... The issue appeared when building large Aho-Corasick automatons. Otherwise, it usually do…
What we need is a page-based Vec that mmaps (anon) for the storage but leaves the unused portions zero-bytes and therefore not part of RSS until actually required. (And when clearing/shrinking sections, madvise DONTNEED the pages). That is, the vec could expand to areas much larger than the actual used size, but this would have no effect on process RSS until those pages get dirtied with actual data.
Couldn't this obfuscate OOM type problems by not triggering actual memory issues until much later than the allocation?