what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?
Maps and Memory Leaks in Go
11–20 of 72 posts
Re: Maps and Memory Leaks in Go
#12i don't see any issue here. same behavior can be achieved with slices: foo := make([]int, 0, 1000) for { for k := range bar { foo = append(foo, k) } foo = foo[:0] } the slice will grow as much as the largest dataset. map will be the same. you need to let go of it to be GCd and create a new one.
The point is that you can remove the entries from the map, and the map won't ever shrink. If you're using large value type in the map[1], the map's dead storage will be large - by a functionally unbound amount. Most sane collection libraries shrink their backing store after some sufficiently large portion becomes dead.
[1] I would argue a general purpose hash table/map should really switch to using a hash code=>index mapping automatically if the value type size is sufficiently large.
Re: Maps and Memory Leaks in Go
#13what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?
Yeah, that makes the title pretty much clickbait, because a memory leak in a memory-safe language would really be a big deal...
Re: Maps and Memory Leaks in Go
#14i don't see any issue here. same behavior can be achieved with slices: foo := make([]int, 0, 1000) for { for k := range bar { foo = append(foo, k) } foo = foo[:0] } the slice will grow as much as the largest dataset. map will be the same. you need to let go of it to be GCd and create a new one.
The point is that you can remove the entries from the map, and the map won't ever shrink. If you're using large value type in the map[1], the map's dead storage will be large - by a functionally unbound amount. Most sane collection libraries shrink their backing store after some sufficiently large portion becomes dead. [1] I would argue a general purpose hash table/map should really switch to using a hash code=>index…
The map will shrink, just not by as much as you might expect. You could also argue that they have picked a pathological case for their example because they use 128 byte entries ("If a key or a value is over 128 bytes, Go won’t store it directly in the map bucket. Instead, Go stores a pointer to reference the key or the value" - which probably leads to less memory consumption).
Re: Maps and Memory Leaks in Go
#15what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?
Yeah, that makes the title pretty much clickbait, because a memory leak in a memory-safe language would really be a big deal...
> a memory leak in a memory-safe language would really be a big deal...
It is not.
Let me show you a memory leak in the memory safe language, rust:
let vec: Vec = Vec::with_capacity(1024);
std::mem::forget(vec);
_ = time.Tick(1 * time.Second)
You can also, if you want to leak memory in go, set the environment variable GOGC=off, and there you go, instant memory leak.
Practically any language, memory safe or otherwise, will let you create a memory leak.
Re: Maps and Memory Leaks in Go
#16what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?
Yeah, that makes the title pretty much clickbait, because a memory leak in a memory-safe language would really be a big deal...
Go is just memory safe until you have a race, or so I have heard.
Re: Maps and Memory Leaks in Go
#17FWIW i am pretty sure Java's HashMap has the same behaviour - it grows the table, but never shrinks it. Even if you call .clear(), it just clears out the table, rather than throwing the table away.
I imagine there are lots of scenarios in which this is what you want, because after emptying the map, you're going to re-fill it, and it saves reallocating the table. But it would be frustrating in a scenario when that isn't what you want.
If a map has this behaviour, i would say that the most important thing is that it should be clearly documented (Java's isn't). The second most important thing is that there should be a way to get round it - either a .clearHarder() method which throws away the table, or a .compact() method which downsizes it while retaining the content.
Re: Maps and Memory Leaks in Go
#18This all looks like reasonable implementation behavior which will give optimal runtime performance in most "common" cases. If one really wants or needs a map that'll free memory as it shrinks (and sure, for some folks, that'd be super useful) one's always free to just implement your own.
Re: Maps and Memory Leaks in Go
#19what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?
Those pretty much can't happen in garbage collected languages, so the usage of the term has been widened to include things like this. I agree it's a shame.
Roedy Green coined the name "packratting" for this modern kind of memory leak: https://www.mindprod.com/jgloss/packratting.html
Re: Maps and Memory Leaks in Go
#20FWIW i am pretty sure Java's HashMap has the same behaviour - it grows the table, but never shrinks it. Even if you call .clear(), it just clears out the table, rather than throwing the table away. I imagine there are lots of scenarios in which this is what you want, because after emptying the map, you're going to re-fill it, and it saves reallocating the table. But it would be frustrating in a scenario when that isn…
Wouldn't reallocating the map be very cheap when running in a JVM? Yes, it isn't something to do in the hot path, but surely it should be faster than a direct malloc(), right? I'm being very naive and ready to be proven wrong.