Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maps and Memory Leaks in Go

teivah.medium.com

Post reply on HN
Print view View original

61–70 of 72 posts

Re: Maps and Memory Leaks in Go

#61

Post by SeanLuke »

skywhopper wrote: post #33
SeanLuke wrote: post #27

Let's be kind and assume that prior to removal, the map has just trebled in size and the extra space wasn't used. Doesn't this imply that a map has an overhead of about 100 bytes per key/value pair? How can this be so?

There's some some waste involved in not pre-allocating the map to begin with. Check the output of this variation ( https://go.dev/play/p/vQwg3GajzXx -- n shrunk to 1,000 to stay within the playground's memory constraints) which fills the map again after the GC. You'll see the map doesn't grow back to the max size. And if you specify the size of the map up front in the `make()` function, it never grows or shrinks (or…

Hashtables often treble when they reach 50% utilization. I think I'm already factoring that in. It's still 100 bytes overhead!

Re: Maps and Memory Leaks in Go

#62

Post by morelisp »

esprehn wrote: post #38
tapirl wrote: post #25

Yes, Go maps never shrink. This is good for most use cases in practice. Because in practice, map entry deletions happen seldom. And when map entry deletions are needed, users often hope maps don't shrink, to avoid potential later unnecessary memory allocations and entry moves. For example, I only do map entry deletions in one of my projects, In the project, I clear all entries of a map and re-use the map to avoid mak…

That's easily resolved with an ensureCapacity(size) method, or the other way around with shrinkToFit(size). maps being magic without a real interface means exposing new API surface is difficult though.

A few times I have wished for a way to grow maps, but generally to shrink maps I don't think there's any big advantage to a built-in rather than your own function making a new one. (It would only cover the case where you know you want to shrink but don't know if you need to shrink because you don't know how many elements you removed or overestimated by, which I think is pretty unusual.)

Re: Maps and Memory Leaks in Go

#63

Post by morelisp »

jimbokun wrote: post #30
eptcyka wrote: post #21

Earlier quoted context omitted.

Go doesn't really allow you to create a hashmap with the same generic possibilities as the built in one.

Is that still true with the support Go added for generics?

You probably won't be able to match the built-in's performance without a similarly large surface area of unsafe usage. And Go's maintainers won't maintain your unsafe code for you as they do the default, nor consider the impact of other language changes on your micro-optimizations.

https://github.com/golang/go/blob/master/src/runtime/map.go

You also won't have `for k, v := range` iteration, but that is also likely being addressed within the next few versions.

But - no, there's not really any major ergonomic issues to basic lookups given generics these days.

Re: Maps and Memory Leaks in Go

#65

Post by suremarc »

Thaxll wrote: post #60
sigg3 wrote: post #16

Earlier quoted context omitted.

Go is just memory safe until you have a race, or so I have heard.

Every GC language by definition are memory safe, memory safety in programming does not mean than accessing the same resources from two thread should be safe.

I don't know how it works in other languages, but accessing a partially overwritten slice in Go (as will happen in the presence of data races) can cause your code to access out-of-bounds memory. And as we all know, once you have read/write access to arbitrary areas in memory, you've basically opened up Pandora's box.

Re: Maps and Memory Leaks in Go

#66

Post by saghm »

fnord123 wrote: post #22
mjpa86 wrote: post #11

what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?

That's the most used example of a memory leak but it is not the definition of a memory leak. If you put data into a hash map and forget the key, you leaked.

This is also why valgrind classifies the leaks it reports with stuff like "still reachable" or "possibly still in use" (I might be remembering the exact phrasing incorrectly). It would be pretty hard to programmatically determine whether the memory that's still kept around was intended to be kept around or not, which is why valgrind supports generating "suppressions" (and specifying them in subsequent runs to be ignored).

Re: Maps and Memory Leaks in Go

#67

Post by olliej »

morelisp wrote: post #24
olliej wrote: post #12

Earlier quoted context omitted.

The point is that you can remove the entries from the map, and the map won't ever shrink. If you're using large value type in the map[1], the map's dead storage will be large - by a functionally unbound amount. Most sane collection libraries shrink their backing store after some sufficiently large portion becomes dead. [1] I would argue a general purpose hash table/map should really switch to using a hash code=>index…

> Most sane collection libraries shrink their backing store after some sufficiently large portion becomes dead. If you exclude Java, and C++ and C# I think, stdlibs from being sane, sure.

womp womp, you are indeed correct. I'd swear that Java and .NET's did, but I assume that's bad memory at fault :(

I will say though that I don't consider C++'s various maps to be sane :D

Re: Maps and Memory Leaks in Go

#68

Post by EdwardDiego »

twic wrote: post #17

FWIW i am pretty sure Java's HashMap has the same behaviour - it grows the table, but never shrinks it. Even if you call .clear(), it just clears out the table, rather than throwing the table away. I imagine there are lots of scenarios in which this is what you want, because after emptying the map, you're going to re-fill it, and it saves reallocating the table. But it would be frustrating in a scenario when that isn…

You're right that Java's Hashmap only ever resizes upwards - the most common use case. However, if there's a need to remove that memory after clearing() then the typical use case is to do something that allows the GC to sort it.

E.g.,

    var map = new HashMap();
    // Many things added to map
    // Many but not all things removed from map
    // to shrink it to size of remaining items
    map = new HashMap(map);
Yeah, it's not nice, but the people who wrote the JDK are far smarter than me, so I figure they optimised for the 99% use-case, not the 1%.

Re: Maps and Memory Leaks in Go

#69

Post by EdwardDiego »

rob74 wrote: post #13
mjpa86 wrote: post #11

what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?

Yeah, that makes the title pretty much clickbait, because a memory leak in a memory-safe language would really be a big deal...

Plenty easy to leak memory in memory-safe languages. I'm assuming we're including GC-ed languages in that set.

Re: Maps and Memory Leaks in Go

#70

Post by mariusor »

EdwardDiego wrote: post #68
twic wrote: post #17

FWIW i am pretty sure Java's HashMap has the same behaviour - it grows the table, but never shrinks it. Even if you call .clear(), it just clears out the table, rather than throwing the table away. I imagine there are lots of scenarios in which this is what you want, because after emptying the map, you're going to re-fill it, and it saves reallocating the table. But it would be frustrating in a scenario when that isn…

You're right that Java's Hashmap only ever resizes upwards - the most common use case. However, if there's a need to remove that memory after clearing() then the typical use case is to do something that allows the GC to sort it. E.g., var map = new HashMap (); // Many things added to map // Many but not all things removed from map // to shrink it to size of remaining items map = new HashMap (map); Yeah, it's not nice…

I think that's exactly what TFA is proposing by recreating the map

    m := make(map[Bla]Bla)
    // add to map
    // mark map as ready for GC
    m = make(map[Bla]Bla)
Post reply on HN