Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Identifying Rust's collect: >() memory leak footgun

blog.polybdenum.com

Post reply on HN
Print view View original

61–70 of 129 posts

Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun

#61

Post by devmor »

hyperpape wrote: post #56
nickez wrote: post #53

Earlier quoted context omitted.

Please read the article. His produces 200x intermediary values. Clickbait title, since it wasn't a leak.

I agree that this is not a memory leak. However, the semantic distinction between "this uses much more memory than expected" and "this is a memory leak" is a little subtle, and it seems pretty rude to call it clickbait.

I disagree that it’s a small semantic difference.

I don’t think it’s clickbait though, I think the author was just misusing terminology.

Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun

#62

Post by cmrx64 »

sapiogram wrote: post #60
tsnl wrote: post #59

Earlier quoted context omitted.

Another downside of Box is that Vec::into_boxed_slice copies all elements from the source vector. Vec::shrink_to_fit does not appear to copy. This is based on the stable docs.

It doesn't copy the elements if the source vector is already at max capacity, which you can often arrange for beforehand.

For example, by explicitly calling shrink_to_fit. So you either have fragmentation, or an extra copy if you're not careful, but none of the solutions let you forget about the detail entirely.

Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun

#63

Post by hyperpape »

devmor wrote: post #61
hyperpape wrote: post #56

Earlier quoted context omitted.

I agree that this is not a memory leak. However, the semantic distinction between "this uses much more memory than expected" and "this is a memory leak" is a little subtle, and it seems pretty rude to call it clickbait.

I disagree that it’s a small semantic difference. I don’t think it’s clickbait though, I think the author was just misusing terminology.

I said it was subtle, not small. I agree it's a valuable distinction.

Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun

#64

Post by vlovich123 »

hedgehog wrote: post #58
vlovich123 wrote: post #51

I’ve read the article and I’m still lost as to how this optimization results in a 200x discrepancy between capacity and length. Was the mapped element size 200x smaller in this case? Or is there some bug where repeated mappings like this cause a 2x growth based on the previous element size instead of the target element size?

I think the behavior is that if you have a Vec of u128 (say 1000), filter that to fewer elements (say 10), and then collect it into a Vec of u32 you might expect the resulting value to be around 40 bytes but in beta Rust it is 16000 bytes. In current Rust the collect would cause a new 10 element Vec of u32 to be allocated, in beta it reuses the original larger allocation. The author's code is doing a bit more but ess…

Ok. I missed that there’s a filter step that compounds the problem. The more I read the less this sounds like a bug and more like application code is missing a shrink_to_fit and was relying on a pessimization.

That being said, it’s also not an unreasonable expectation on the user’s behalf that the size and capacity don’t get crazy different in code as innocuous and idiomatic as this.

I wonder how the open bug will end up training itself.

Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun

#65

Post by vlovich123 »

0x457 wrote: post #52
vlovich123 wrote: post #44

Earlier quoted context omitted.

It seems like you have a better grasp on the problem than I. I understand why capacity in terms of number of elements doubles. But why did he end up using 200x more memory? Is it that the transient memory used the 18GiB and then he reduced it to vector of elements that were 200x smaller? Or is it that there’s some other problem that when you collect, it’s reallocating 2x using the wrong type to do the computation?

Because it doubles in every node and OP has 300k of them[1]. It's not a leak, since memory is...well...accounted for and easily freed by either dropping Vec or calling shirk_to_fit. IMO it's expected behavior. Honestly, if I had 300k Vecs, I would call `shrink_to_fit` even without that optimization. 1: Since the edge list calculation logic is run for one node at a time in sequence, the fact that it temporarily uses 1…

Ah right. Vec of vecs and the interior Vec has wasted capacity due to the reduction of the element size. Thanks!

Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun

#67

Post by invpt »

cmrx64 wrote: post #62
sapiogram wrote: post #60

Earlier quoted context omitted.

It doesn't copy the elements if the source vector is already at max capacity, which you can often arrange for beforehand.

For example, by explicitly calling shrink_to_fit. So you either have fragmentation, or an extra copy if you're not careful, but none of the solutions let you forget about the detail entirely.

If you look at the source for into_boxed_slice, it calls shrink_to_fit at the beginning before doing anything else. Hence the documentation is slightly wrong, and no copies occur.

Edit: I submitted a PR to clear up the docs: https://github.com/rust-lang/rust/pull/120110

Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun

#68

Post by mratsim »

hyperpape wrote: post #63
devmor wrote: post #61

Earlier quoted context omitted.

I disagree that it’s a small semantic difference. I don’t think it’s clickbait though, I think the author was just misusing terminology.

I said it was subtle, not small. I agree it's a valuable distinction.

A memory leak means it leaks, it's not anymore under control. Here the memory is under control, it can be reclaimed by the program.

Re: Identifying Rust's collect:<Vec<_>>() memory leak footgun

#69

Post by sapiogram »

cmrx64 wrote: post #62
sapiogram wrote: post #60

Earlier quoted context omitted.

It doesn't copy the elements if the source vector is already at max capacity, which you can often arrange for beforehand.

For example, by explicitly calling shrink_to_fit. So you either have fragmentation, or an extra copy if you're not careful, but none of the solutions let you forget about the detail entirely.

Or alternatively, the vector was created with `Vec::with_capacity()` or `ExactSizeIterator::collect()`, and had the correct capacity all along.
Post reply on HN