Live data from Hacker News

Maps and Memory Leaks in Go

teivah.medium.com

21–30 of 72 posts

Re: Maps and Memory Leaks in Go

#21

This all looks like reasonable implementation behavior which will give optimal runtime performance in most "common" cases. If one really wants or needs a map that'll free memory as it shrinks (and sure, for some folks, that'd be super useful) one's always free to just implement your own.

Go doesn't really allow you to create a hashmap with the same generic possibilities as the built in one.

Re: Maps and Memory Leaks in Go

#22
post #11

what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?

That's the most used example of a memory leak but it is not the definition of a memory leak.

If you put data into a hash map and forget the key, you leaked.

Re: Maps and Memory Leaks in Go

#23
post #11

what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?

That's the most used example of a memory leak but it is not the definition of a memory leak. If you put data into a hash map and forget the key, you leaked.

[deleted]

Re: Maps and Memory Leaks in Go

#24
post #12
post #9

i don't see any issue here. same behavior can be achieved with slices: foo := make([]int, 0, 1000) for { for k := range bar { foo = append(foo, k) } foo = foo[:0] } the slice will grow as much as the largest dataset. map will be the same. you need to let go of it to be GCd and create a new one.

The point is that you can remove the entries from the map, and the map won't ever shrink. If you're using large value type in the map[1], the map's dead storage will be large - by a functionally unbound amount. Most sane collection libraries shrink their backing store after some sufficiently large portion becomes dead. [1] I would argue a general purpose hash table/map should really switch to using a hash code=>index…

> Most sane collection libraries shrink their backing store after some sufficiently large portion becomes dead.

If you exclude Java, and C++ and C# I think, stdlibs from being sane, sure.

Re: Maps and Memory Leaks in Go

#25
Yes, Go maps never shrink. This is good for most use cases in practice. Because in practice, map entry deletions happen seldom. And when map entry deletions are needed, users often hope maps don't shrink, to avoid potential later unnecessary memory allocations and entry moves. For example, I only do map entry deletions in one of my projects, In the project, I clear all entries of a map and re-use the map to avoid making new allocations. The current design satisfies my need well.

This is more an optimization than a memory leak.

To avoid the kind-of memory leak, just make a new map and discard the old one.

Re: Maps and Memory Leaks in Go

#26
post #21

This all looks like reasonable implementation behavior which will give optimal runtime performance in most "common" cases. If one really wants or needs a map that'll free memory as it shrinks (and sure, for some folks, that'd be super useful) one's always free to just implement your own.

Go doesn't really allow you to create a hashmap with the same generic possibilities as the built in one.

Why is it? Or rather what limitations are there to enforced by go compiler that won’t allow someone to implement their own hash map that can free memory with the same generic possibilities, even with the recent introduction of generics?

Re: Maps and Memory Leaks in Go

#27
Let's be kind and assume that prior to removal, the map has just trebled in size and the extra space wasn't used. Doesn't this imply that a map has an overhead of about 100 bytes per key/value pair? How can this be so?

Re: Maps and Memory Leaks in Go

#28
post #25

Yes, Go maps never shrink. This is good for most use cases in practice. Because in practice, map entry deletions happen seldom. And when map entry deletions are needed, users often hope maps don't shrink, to avoid potential later unnecessary memory allocations and entry moves. For example, I only do map entry deletions in one of my projects, In the project, I clear all entries of a map and re-use the map to avoid mak…

The article gives a common place example where this could be an issue, so I don’t know what you mean by “in practice” here.

Re: Maps and Memory Leaks in Go

#29
post #11

what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?

That's the most used example of a memory leak but it is not the definition of a memory leak. If you put data into a hash map and forget the key, you leaked.

This is the use case for weak maps, which both Java and JavaScript have. In the latter case, the map is not iterable, so one cannot observe JavaScript GC (through WeakMap at least).

Re: Maps and Memory Leaks in Go

#30
post #21

This all looks like reasonable implementation behavior which will give optimal runtime performance in most "common" cases. If one really wants or needs a map that'll free memory as it shrinks (and sure, for some folks, that'd be super useful) one's always free to just implement your own.

Go doesn't really allow you to create a hashmap with the same generic possibilities as the built in one.

Is that still true with the support Go added for generics?
Post reply on HN