Manual Memory Management in Go
deferpanic.com
Manual Memory Management in Go
1–10 of 14 posts
Re: Manual Memory Management in Go
#2The article references a blog post that I wrote a long while ago about how we do memory management for some Go programs at CloudFlare: https://blog.cloudflare.com/recycling-memory-buffers-in-go/ We've switched most of that stuff to use sync.Pool now. And we've done a lot of pprof work to simply reduce garbage in the first place.
We can't wait for Go 1.5 where the Stop-The-World GC goes away: https://docs.google.com/document/d/1wmjrocXIWTr1JxU-3EQBI6BK...
Re: Manual Memory Management in Go
#3The article references a blog post that I wrote a long while ago about how we do memory management for some Go programs at CloudFlare: https://blog.cloudflare.com/recycling-memory-buffers-in-go/ We've switched most of that stuff to use sync.Pool now. And we've done a lot of pprof work to simply reduce garbage in the first place. We can't wait for Go 1.5 where the Stop-The-World GC goes away: https://docs.google.com/d…
Nice article.
Regardless of my opinion about some Go issues, I really look forward to see the language becoming widespread.
My experience with Oberon back in the mid-90's, made me aware that it is possible to have systems languages with GC.
The main problem is that many developers don't experience the variety from GC implementations that exist out there, and equate all implementations alike.
In Oberon's case it wasn't the best GC in the world, specially in mid-90's PC hardware, but coupled together with value types it was good enough to fully implement an OS in Oberon (and its derivatives).
Re: Manual Memory Management in Go
#4"Imagine doing lots of small allocations - you can cause a lot of fragmentation resulting in needlessly having to resize your heap which in dire scenarios can result in thrashing."
The article repeatedly talks about how managing memory manually increases the risk of fragmentation. And that this risk somehow goes away with gc managed heaps.
...so garbage collectors don't also have to manage their own internal heaps and have fragmentation issues? Hm, not sure I buy this.
Re: Manual Memory Management in Go
#5The title seems a bit misleading, the author goes on to talk about possible performance issues and how people view GC's and issues with manual memory management, but never actually touches on how to actually use any of the alternatives mentioned.
.. So it's actually just an advertisement for their product, the title should really be something else, I feel a bit mislead initially assuming something useful might be had from reading the article.
Re: Manual Memory Management in Go
#6"Imagine doing lots of small allocations - you can cause a lot of fragmentation resulting in needlessly having to resize your heap which in dire scenarios can result in thrashing." The article repeatedly talks about how managing memory manually increases the risk of fragmentation. And that this risk somehow goes away with gc managed heaps. ...so garbage collectors don't also have to manage their own internal heaps an…
GC's can compact memory. Go does this for stacks, but not (yet) for heaps.
Re: Manual Memory Management in Go
#7"Imagine doing lots of small allocations - you can cause a lot of fragmentation resulting in needlessly having to resize your heap which in dire scenarios can result in thrashing." The article repeatedly talks about how managing memory manually increases the risk of fragmentation. And that this risk somehow goes away with gc managed heaps. ...so garbage collectors don't also have to manage their own internal heaps an…
Compacting garbage collectors don't have fragmentation issues, but of course Go doesn't have one. A GC isn't even strictly a prerequisite for compaction. Manually combining memory allocations to reduce allocator overhead and fragmentation is a lot of work, but not unheard of.
Re: Manual Memory Management in Go
#8Not a very useful article from the perspective of you are the point where you need to take some control over memory management in Go. The article spends a lot of time telling you should probably don't have that problem or you are not going to do any better than the Go. However, it is pretty easy to start proving you have that problem. Furthmore, you can make a real dent in the problem if you spend enough time working on it.
I should probably not just complain but explain
1. How to prove via pprof and other tools that you have problems related to memory allocation or GC.
2. Some strategies for addressing that problem that don't involve writing an allocator. Using sync.Pool or a recycler pattern. Keeping allocations on the stack etc...
3. How to write a allocation interface in Go for allocating arbitrary objects. Aka emulating New and Make.
4. How to write a allocator backed by a memory mapped files.
I have been working for a while in this space because of my academic research which involves frequent subgraph mining. That particular data mining problem is exponential and certain techniques can use a lot of memory. In order to scale my solution I have need to get the memory allocations under tight control. Sometimes I think I should just re-write in C++ or Rust but Go has provided me a lot of benefits from the concurrency angle so it isn't a win/win to move languages.
Maybe I will write up all of this stuff at some point. It is a bit much for an HN comment.
Re: Manual Memory Management in Go
#9> This is at the point when another developer on the team will pipe in stating their preference for another langauge, that if you might not be a software engineer, you might confuse w/the chemical compound of Fe2O3.
Or a certain fungi. :)
http://stackoverflow.com/questions/16494822/why-is-it-called...
Re: Manual Memory Management in Go
#10Not a very useful article from the perspective of you are the point where you need to take some control over memory management in Go. The article spends a lot of time telling you should probably don't have that problem or you are not going to do any better than the Go. However, it is pretty easy to start proving you have that problem. Furthmore, you can make a real dent in the problem if you spend enough time working…
I agree that this doesn't seem to actually cover practical memory management. In Go (and perhaps most languages) this seems to mean avoid allocations, and if you have to allocate, avoid the heap.
Maybe some information about how heap size and mutation rate affect GC would be interesting, as well as the future of the Go GC.
One interesting thing about Go is that if your objects contain no pointers, the Go GC does not have to scan them. So you can supposedly create huge slices of any type containing no pointers and the GC will have a much lower load on it.
There's a really good article from Dmitry Vyukov including memory related stuff here: https://software.intel.com/en-us/blogs/2014/05/10/debugging-...