Techniques for investigating untypical Go memory leaks
cossacklabs.com
Techniques for investigating untypical Go memory leaks
1–10 of 10 posts
Re: Techniques for investigating untypical Go memory leaks
#2Techniques for noticing you're almost shooting yourself in the leg, then suddenly saving the situation very last moment, mi amigo :)
Re: Techniques for investigating untypical Go memory leaks
#3Don't use cgo (or any dependencies that use it) if you can at all avoid doing so.
Re: Techniques for investigating untypical Go memory leaks
#4"When we've initially started [..], dominant was Go 1.3, which had mechanisms to detect memory blocks allocated inside used libraries and all such blocks were to be freed by GC. Go 1.6 significantly improved its memory management, however as part of those improvements memory allocated by external libraries was no longer garbage collected by the runtime."
This looks like a serious API break.
To be fair, the whole idea of the runtime automagically managing external memory gives me shivers, and this looks like moving in the right direction - I just think it was not very wise to go with that initial decision in the first place.
Re: Techniques for investigating untypical Go memory leaks
#5This remember me one of the Go Proverbs, Cgo is not Go https://www.youtube.com/watch?v=PAAkCSZUG1c&t=12m37s
Re: Techniques for investigating untypical Go memory leaks
#6"var output = (*byte)(unsafe.Pointer(m))" This will give you the pointer to the byte slice object which may have its own structure which is opaque to you and this is not same as "getting the pointer to C byte array". It is weird that it was producing correct results at all :)
Re: Techniques for investigating untypical Go memory leaks
#7"When we've initially started [..], dominant was Go 1.3, which had mechanisms to detect memory blocks allocated inside used libraries and all such blocks were to be freed by GC. Go 1.6 significantly improved its memory management, however as part of those improvements memory allocated by external libraries was no longer garbage collected by the runtime." This looks like a serious API break. To be fair, the whole idea…
Well, it is true: it was interesting side effect that 'just worked', we were like 'hmm, if GC is able to control external memory allocation, well, fine, maybe Go is even better than we've thought initially'.
At some point in time, someone behind Go/CGo realized that automatically managing pointer math is an exercise that is being attempted for last 15 years without success (there always will be absolutely normal situations where you can't measure allocation correctly), and just limited visibility scope to things Go can control.
"This looks like a serious API break."
So thought we, being a bit annoyed. But when you think of how many problems it actually prevents,- it's a worthy move, in the end.
Re: Techniques for investigating untypical Go memory leaks
#8This remember me one of the Go Proverbs, Cgo is not Go https://www.youtube.com/watch?v=PAAkCSZUG1c&t=12m37s
But, as he says, 'sometimes you have to'. This is a time like this.
Re: Techniques for investigating untypical Go memory leaks
#9Don't use cgo (or any dependencies that use it) if you can at all avoid doing so.
If we could avoid doing so - we would, because we understand how hard earned Go's guarantees are. But, things are the way they are.
Re: Techniques for investigating untypical Go memory leaks
#10"When we've initially started [..], dominant was Go 1.3, which had mechanisms to detect memory blocks allocated inside used libraries and all such blocks were to be freed by GC. Go 1.6 significantly improved its memory management, however as part of those improvements memory allocated by external libraries was no longer garbage collected by the runtime." This looks like a serious API break. To be fair, the whole idea…
May not have been a "decision" at all. The simplest GC just needs to hijack malloc to keep track of allocations and every now and then scan the process memory for values that look like pointers into those allocations. Of course that is both slow and leaks bad enough that early Go versions had issues with 32bit address space.