Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maps and Memory Leaks in Go

teivah.medium.com

Post reply on HN
Print view View original

31–40 of 72 posts

Re: Maps and Memory Leaks in Go

#31

Post by Gwypaas »

twic wrote: post #17

FWIW i am pretty sure Java's HashMap has the same behaviour - it grows the table, but never shrinks it. Even if you call .clear(), it just clears out the table, rather than throwing the table away. I imagine there are lots of scenarios in which this is what you want, because after emptying the map, you're going to re-fill it, and it saves reallocating the table. But it would be frustrating in a scenario when that isn…

Rust uses "shrink_to_fit()". Personally never had to use it, but you always end up scrolling by the backing allocation management for all the standard collections when looking through the docs.

> Shrinks the capacity of the map as much as possible. It will drop down as much as possible while maintaining the internal rules and possibly leaving some space in accordance with the resize policy.

https://doc.rust-lang.org/std/collections/struct.HashMap.htm...

And the docs makes it clear that "clear()" only removes the elements. Giving a hint of where to go next if you stumble upon the issue in the OP.

> "Clears the map, removing all key-value pairs. Keeps the allocated memory for reuse."

https://doc.rust-lang.org/std/collections/struct.HashMap.htm...

Re: Maps and Memory Leaks in Go

#32

Post by throwaway894345 »

jimbokun wrote: post #28
tapirl wrote: post #25

Yes, Go maps never shrink. This is good for most use cases in practice. Because in practice, map entry deletions happen seldom. And when map entry deletions are needed, users often hope maps don't shrink, to avoid potential later unnecessary memory allocations and entry moves. For example, I only do map entry deletions in one of my projects, In the project, I clear all entries of a map and re-use the map to avoid mak…

The article gives a common place example where this could be an issue, so I don’t know what you mean by “in practice” here.

The parent said “most” so both can be true: most maps don’t need to shrink and the example in the article is a valid case where shrinking a map would be desirable. This seems obvious so I’m confused by your confusion. :)

Re: Maps and Memory Leaks in Go

#33

Post by skywhopper »

SeanLuke wrote: post #27

Let's be kind and assume that prior to removal, the map has just trebled in size and the extra space wasn't used. Doesn't this imply that a map has an overhead of about 100 bytes per key/value pair? How can this be so?

There's some some waste involved in not pre-allocating the map to begin with. Check the output of this variation (https://go.dev/play/p/vQwg3GajzXx -- n shrunk to 1,000 to stay within the playground's memory constraints) which fills the map again after the GC. You'll see the map doesn't grow back to the max size.

And if you specify the size of the map up front in the `make()` function, it never grows or shrinks (or not significantly): https://go.dev/play/p/AGW35kMMOc5

IIRC, what's happening is that whenever the map hits what was pre-allocated, it expands by a certain growth factor of the current size. So at some point between 1 and 1,000,000 a big chunk was allocated that went over what was necessary for 1,000,000 entries. In my testing it happened around i == 851_500:

   i == 851000: 258507 KB
   i == 852000: 479212 KB

Re: Maps and Memory Leaks in Go

#34

Post by pclmulqdq »

sigg3 wrote: post #16
rob74 wrote: post #13

Earlier quoted context omitted.

Yeah, that makes the title pretty much clickbait, because a memory leak in a memory-safe language would really be a big deal...

Go is just memory safe until you have a race, or so I have heard.

The same goes for Rust and most other "safe" languages. They all have synchronization primitives that make it safe, but you need to use them - the compiler won't always tell you.

Re: Maps and Memory Leaks in Go

#35

Post by creata »

twic wrote: post #17

FWIW i am pretty sure Java's HashMap has the same behaviour - it grows the table, but never shrinks it. Even if you call .clear(), it just clears out the table, rather than throwing the table away. I imagine there are lots of scenarios in which this is what you want, because after emptying the map, you're going to re-fill it, and it saves reallocating the table. But it would be frustrating in a scenario when that isn…

> If a map has this behaviour, i would say that the most important thing is that it should be clearly documented

Iirc, most hash table implementations don't automatically shrink. More documentation is always nice, but isn't not automatically shrinking kind of the expected "default" behavior?

Re: Maps and Memory Leaks in Go

#37

Post by avianlyric »

sys_64738 wrote: post #36

I thought GO had garbage collection or is that just trash talk?

It does, and as the article mentions, it’ll collect the map elements. But it doesn’t collect the map infrastructure that grew to accommodate the elements, and doesn’t shrink when the elements are removed, because the map never stops referencing them.

Re: Maps and Memory Leaks in Go

#38

Post by esprehn »

tapirl wrote: post #25

Yes, Go maps never shrink. This is good for most use cases in practice. Because in practice, map entry deletions happen seldom. And when map entry deletions are needed, users often hope maps don't shrink, to avoid potential later unnecessary memory allocations and entry moves. For example, I only do map entry deletions in one of my projects, In the project, I clear all entries of a map and re-use the map to avoid mak…

That's easily resolved with an ensureCapacity(size) method, or the other way around with shrinkToFit(size).

maps being magic without a real interface means exposing new API surface is difficult though.

Re: Maps and Memory Leaks in Go

#39

Post by suremarc »

pclmulqdq wrote: post #34
sigg3 wrote: post #16

Earlier quoted context omitted.

Go is just memory safe until you have a race, or so I have heard.

The same goes for Rust and most other "safe" languages. They all have synchronization primitives that make it safe, but you need to use them - the compiler won't always tell you.

I have no idea what you mean here. Data races are next to impossible in safe Rust.

Re: Maps and Memory Leaks in Go

#40

Post by EdiX »

SeanLuke wrote: post #27

Let's be kind and assume that prior to removal, the map has just trebled in size and the extra space wasn't used. Doesn't this imply that a map has an overhead of about 100 bytes per key/value pair? How can this be so?

If you are referring to the change from 461 MB to 293 MB that's because he didn't call runtime.GC right after the insertion loop.
Post reply on HN