Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maps and Memory Leaks in Go

teivah.medium.com

Post reply on HN
Print view View original

51–60 of 72 posts

Re: Maps and Memory Leaks in Go

#51

Post by bilkow »

pclmulqdq wrote: post #34
sigg3 wrote: post #16

Earlier quoted context omitted.

Go is just memory safe until you have a race, or so I have heard.

The same goes for Rust and most other "safe" languages. They all have synchronization primitives that make it safe, but you need to use them - the compiler won't always tell you.

For Rust specifically, the compiler does force safe programs to have no data races. That's actually what the ownership system, Send and Sync are about. If you manage to corrupt memory or have undefined behavior in safe Rust, that should be a compiler or library bug.

See https://doc.rust-lang.org/nomicon/races.html

Re: Maps and Memory Leaks in Go

#52

Post by foldr »

twic wrote: post #17

FWIW i am pretty sure Java's HashMap has the same behaviour - it grows the table, but never shrinks it. Even if you call .clear(), it just clears out the table, rather than throwing the table away. I imagine there are lots of scenarios in which this is what you want, because after emptying the map, you're going to re-fill it, and it saves reallocating the table. But it would be frustrating in a scenario when that isn…

>The second most important thing is that there should be a way to get round it - either a .clearHarder() method which throws away the table, or a .compact() method which downsizes it while retaining the content.

Wouldn't just creating a new map be a pretty good way of doing it? If the reduction in size is significant, then the cost of the new allocation will be small compared to the cost of all the allocations you've done to build the original map.

Re: Maps and Memory Leaks in Go

#53

Post by simiones »

twic wrote: post #49
simiones wrote: post #48

Earlier quoted context omitted.

Memory leaks have always meant "failing to free memory that is no longer needed ". Garbage collection literature often stresses the difference between "no longer needed" and "not reachable", noting that the former is not automatically enforceable (it amounts to solving the halting problem), but the latter is only a heuristic. So, the fact that garbage collectors can't prevent all memory leaks is always stressed by th…

> Memory leaks have always meant "failing to free memory that is no longer needed". Citation needed - that sounds reasonable, but i have never come across that formulation before.

I read about this in the Garbage Collection Handbook [1], which is an excellent overview of the entire field (at least up to ~2016), which discusses the distinction at large. I don't have it on me to quote, but a very clear distinction is made between "live objects" and "reachable objects", with reachabillity acting as a computable proxy for the uncomputable property of liveness. Liveness is defined as "this object will be used again by the program in some way", and a memory leak is defined as "failing to free an object that is no longer live". An unreachable object can't be live, but there are many ways of having a reachable object that is not live.

To prove that this is used in the literature at large, here is the abstract of a random GC paper I found [0]:

> Functional languages manage heap data through garbage collection. Since static analysis of heap data is difficult, garbage collectors conservatively approximate the liveness of heap objects by reachability i.e. every object that is reachable from the root set is considered live. Consequently, a large amount of memory that is reachable but not used further during execution is left uncollected by the collector.

[0] https://dl.acm.org/doi/10.1145/3381898.3397208

[1] https://www.google.com/books/edition/_/TKOfDQAAQBAJ?hl=en&gb... (you may try to search for live/reachable/leak to get some idea here as well)

Re: Maps and Memory Leaks in Go

#54

Post by slantedview »

tapirl wrote: post #25

Yes, Go maps never shrink. This is good for most use cases in practice. Because in practice, map entry deletions happen seldom. And when map entry deletions are needed, users often hope maps don't shrink, to avoid potential later unnecessary memory allocations and entry moves. For example, I only do map entry deletions in one of my projects, In the project, I clear all entries of a map and re-use the map to avoid mak…

> Yes, Go maps never shrink. This is good for most use cases in practice.

Why is it that every shortcoming of Go is spun by its advocates into something that is actually a good thing?

Re: Maps and Memory Leaks in Go

#55

Post by philosopher1234 »

slantedview wrote: post #54
tapirl wrote: post #25

Yes, Go maps never shrink. This is good for most use cases in practice. Because in practice, map entry deletions happen seldom. And when map entry deletions are needed, users often hope maps don't shrink, to avoid potential later unnecessary memory allocations and entry moves. For example, I only do map entry deletions in one of my projects, In the project, I clear all entries of a map and re-use the map to avoid mak…

> Yes, Go maps never shrink. This is good for most use cases in practice. Why is it that every shortcoming of Go is spun by its advocates into something that is actually a good thing?

Why is it that haters of go refuse to engage with the actual argument?

Edit: rephrasing your comment “why is it that every time I say something misleading about go people disagree with me?”

Re: Maps and Memory Leaks in Go

#56

Post by morelisp »

slantedview wrote: post #54
tapirl wrote: post #25

Yes, Go maps never shrink. This is good for most use cases in practice. Because in practice, map entry deletions happen seldom. And when map entry deletions are needed, users often hope maps don't shrink, to avoid potential later unnecessary memory allocations and entry moves. For example, I only do map entry deletions in one of my projects, In the project, I clear all entries of a map and re-use the map to avoid mak…

> Yes, Go maps never shrink. This is good for most use cases in practice. Why is it that every shortcoming of Go is spun by its advocates into something that is actually a good thing?

In what languages do maps shrink? You may be surprised.

Re: Maps and Memory Leaks in Go

#57

Post by Someone »

throwaway894345 wrote: post #32
jimbokun wrote: post #28

Earlier quoted context omitted.

The article gives a common place example where this could be an issue, so I don’t know what you mean by “in practice” here.

The parent said “most” so both can be true: most maps don’t need to shrink and the example in the article is a valid case where shrinking a map would be desirable. This seems obvious so I’m confused by your confusion. :)

I think you’re referring to

> However, let’s say we want to store one hour of data. Meanwhile, our company has decided to have a big promotion for Black Friday: in one hour, we may have millions of customers connected to our system. But a few days after Black Friday, our map will contain the same number of buckets as during the peak time. This explains why we can experience high memory consumption that doesn’t significantly decrease in such a scenario.

> What are the solutions if we don’t want to manually restart our service to clean the amount of memory consumed by the map?

I don’t see that as a strong argument for making the implementation of maps more complex by adding some auto- shrinking. If your process survived Black Friday with a map full of items, why would it run out of memory after it with a map that’s mostly empty but didn’t return bookkeeping memory to the heap? Your process likely will keep running. There may be a slight performance drop because cache lookups become more likely to lead to cache misses, but otherwise, the typical system won’t care much about the use of virtual memory space.

Re: Maps and Memory Leaks in Go

#58

Post by jayd16 »

morelisp wrote: post #56
slantedview wrote: post #54

Earlier quoted context omitted.

> Yes, Go maps never shrink. This is good for most use cases in practice. Why is it that every shortcoming of Go is spun by its advocates into something that is actually a good thing?

In what languages do maps shrink? You may be surprised.

C# has trim excess but I do t think Java does, ...just the two I checked.

https://learn.microsoft.com/en-us/dotnet/api/system.collecti...

Re: Maps and Memory Leaks in Go

#59

Post by Thaxll »

jayd16 wrote: post #58
morelisp wrote: post #56

Earlier quoted context omitted.

In what languages do maps shrink? You may be surprised.

C# has trim excess but I do t think Java does, ...just the two I checked. https://learn.microsoft.com/en-us/dotnet/api/system.collecti...

Java HashMap does not shrink.

Re: Maps and Memory Leaks in Go

#60

Post by Thaxll »

sigg3 wrote: post #16
rob74 wrote: post #13

Earlier quoted context omitted.

Yeah, that makes the title pretty much clickbait, because a memory leak in a memory-safe language would really be a big deal...

Go is just memory safe until you have a race, or so I have heard.

Every GC language by definition are memory safe, memory safety in programming does not mean than accessing the same resources from two thread should be safe.
Post reply on HN