Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maps and Memory Leaks in Go

teivah.medium.com

Post reply on HN
Print view View original

41–50 of 72 posts

Re: Maps and Memory Leaks in Go

#41

Post by _old_dude_ »

twic wrote: post #17

FWIW i am pretty sure Java's HashMap has the same behaviour - it grows the table, but never shrinks it. Even if you call .clear(), it just clears out the table, rather than throwing the table away. I imagine there are lots of scenarios in which this is what you want, because after emptying the map, you're going to re-fill it, and it saves reallocating the table. But it would be frustrating in a scenario when that isn…

The main difference with Go is that the table of a HashMap in Java is a table of pointers, not a table of structs, so the overhead to not shrink the table is less an issue.

Re: Maps and Memory Leaks in Go

#42

Post by suremarc »

sigg3 wrote: post #16
rob74 wrote: post #13

Earlier quoted context omitted.

Yeah, that makes the title pretty much clickbait, because a memory leak in a memory-safe language would really be a big deal...

Go is just memory safe until you have a race, or so I have heard.

Indeed, this person showed that you can read/write to arbitrary memory addresses inside of a Go program: https://blog.stalkr.net/2022/01/universal-go-exploit-using-d...

Although, it's pretty useless as an exploit, since it requires you to be able to run arbitrary Go code to begin with (the author admits as much). It's _very_ unlikely that a remote attacker could exploit a data race in a regular Go program.

Re: Maps and Memory Leaks in Go

#43

Post by [deleted] »

eptcyka wrote: post #21
eschneider wrote: post #18

This all looks like reasonable implementation behavior which will give optimal runtime performance in most "common" cases. If one really wants or needs a map that'll free memory as it shrinks (and sure, for some folks, that'd be super useful) one's always free to just implement your own.

Go doesn't really allow you to create a hashmap with the same generic possibilities as the built in one.

[deleted]

Re: Maps and Memory Leaks in Go

#44

Post by ufo »

twic wrote: post #19
mjpa86 wrote: post #11

what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?

Those pretty much can't happen in garbage collected languages, so the usage of the term has been widened to include things like this. I agree it's a shame. Roedy Green coined the name "packratting" for this modern kind of memory leak: https://www.mindprod.com/jgloss/packratting.html

I've also seen the term "memory bloat".

Re: Maps and Memory Leaks in Go

#45

Post by tapirl »

esprehn wrote: post #38
tapirl wrote: post #25

Yes, Go maps never shrink. This is good for most use cases in practice. Because in practice, map entry deletions happen seldom. And when map entry deletions are needed, users often hope maps don't shrink, to avoid potential later unnecessary memory allocations and entry moves. For example, I only do map entry deletions in one of my projects, In the project, I clear all entries of a map and re-use the map to avoid mak…

That's easily resolved with an ensureCapacity(size) method, or the other way around with shrinkToFit(size). maps being magic without a real interface means exposing new API surface is difficult though.

Personally, I am not against the proposal to add a "shrink(aMapOrSlice)` builtin function. In fact, there are more builtin functions on my wish list, such as "concat(slices)" and "decap(aSlice)".

It is just that the Go core team have their own philosophy.

Re: Maps and Memory Leaks in Go

#46

Post by eptcyka »

ironick09 wrote: post #26
eptcyka wrote: post #21

Earlier quoted context omitted.

Go doesn't really allow you to create a hashmap with the same generic possibilities as the built in one.

Why is it? Or rather what limitations are there to enforced by go compiler that won’t allow someone to implement their own hash map that can free memory with the same generic possibilities, even with the recent introduction of generics?

Seems like I was wrong about my assumptions about Go's generics - it does seem like they're specialized at build time and it is possible to operate over generic values rather than fat pointers. So it is possible to implement a fully featured hash map without extra pointer hopping now. I stand corrected.

Re: Maps and Memory Leaks in Go

#47

Post by Dobbs »

pclmulqdq wrote: post #34
sigg3 wrote: post #16

Earlier quoted context omitted.

Go is just memory safe until you have a race, or so I have heard.

The same goes for Rust and most other "safe" languages. They all have synchronization primitives that make it safe, but you need to use them - the compiler won't always tell you.

That is basically the entire shtick of rust. That data is "owned", and only the owner can write. You can "borrow" something for read access, but if something is borrowed it can't be written to.

There are of course workarounds for this like reference counted wrappers and so on.

Re: Maps and Memory Leaks in Go

#48

Post by simiones »

twic wrote: post #19
mjpa86 wrote: post #11

what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?

Those pretty much can't happen in garbage collected languages, so the usage of the term has been widened to include things like this. I agree it's a shame. Roedy Green coined the name "packratting" for this modern kind of memory leak: https://www.mindprod.com/jgloss/packratting.html

Memory leaks have always meant "failing to free memory that is no longer needed".

Garbage collection literature often stresses the difference between "no longer needed" and "not reachable", noting that the former is not automatically enforceable (it amounts to solving the halting problem), but the latter is only a heuristic. So, the fact that garbage collectors can't prevent all memory leaks is always stressed by the literature.

Re: Maps and Memory Leaks in Go

#49

Post by twic »

simiones wrote: post #48
twic wrote: post #19

Earlier quoted context omitted.

Those pretty much can't happen in garbage collected languages, so the usage of the term has been widened to include things like this. I agree it's a shame. Roedy Green coined the name "packratting" for this modern kind of memory leak: https://www.mindprod.com/jgloss/packratting.html

Memory leaks have always meant "failing to free memory that is no longer needed ". Garbage collection literature often stresses the difference between "no longer needed" and "not reachable", noting that the former is not automatically enforceable (it amounts to solving the halting problem), but the latter is only a heuristic. So, the fact that garbage collectors can't prevent all memory leaks is always stressed by th…

> Memory leaks have always meant "failing to free memory that is no longer needed".

Citation needed - that sounds reasonable, but i have never come across that formulation before.

Re: Maps and Memory Leaks in Go

#50

Post by simiones »

mjpa86 wrote: post #11

what happened to a memory leak being some memory that was allocated but had no reference to it so couldn't be freed? If you can copy the map and release it and the memory usage drops, there is no leak?

A memory leak has always meant "this program keeps allocating more memory as it runs, even though it's not being asked to store anything new". That is equivalent to saying that a program has a memory leak when it fails to free memory that is no longer needed, not just memory that is no longer reachable.

For example, a common example of a memory leak is adding items to a "cache" without any mechanism that evicts items from the cache in any scenario. The "cache" is thus not a cache, but a memory leak (a common implementation of this leaking scenario is that items are put in a map, but never removed from the map).

Memory leak has never, as far as I know, referred to the specific case of memory that is no longer accessible from program code to be freed. In fact, this definition doesn't even make sense from a runtime system perspective, since even in C, the memory is actually always still reachable - from malloc()'s internal data structures.

Post reply on HN