Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory Profiling: Introduction

easyperf.net

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Memory Profiling: Introduction

#11

Post by milianw »

malkia wrote: post #4

Some of our tools do so much allocation that capturing all the information (using MTuner) to disk and later loading require by itself hundreths of GB. Instead I've added random sampling - e.g. if ptr % modulo > level - output it or not. Another factor that slows down is doing a callstack capture. It's not for free at all, like on Windows it has to go through the exception handlers, etc. I think perfetto simply captur…

Heaptrack can handle millions of allocations per second, and thanks to the deduplication built into the trace file format combined with zstd compression, the overhead is pretty manageable. It's pretty hard to get "hundreds of GB" of data recorded.

Another thing: you can always just runtime attach and profile a partial time to reduce the amount of data recorded.

Finally: if your tools do so many allocations, maybe you should consider optimizing them...

Re: Memory Profiling: Introduction

#12

Post by kohlerm »

milianw wrote: post #10
kohlerm wrote: post #2

http://kohlerm.blogspot.com/2009/02/how-to-really-measure-me... do we really still not have a decent memory usage analysis tool for c/ C++ ?

What is not decent about heaptrack or similar tools?

It does not really allow you to analyze what contributes to memory usage in a graph of "objects".

Re: Memory Profiling: Introduction

#13

Post by kohlerm »

kouteiheika wrote: post #8
kohlerm wrote: post #2

http://kohlerm.blogspot.com/2009/02/how-to-really-measure-me... do we really still not have a decent memory usage analysis tool for c/ C++ ?

> do we really still not have a decent memory usage analysis tool for c/ C++ ? Shameless plug: not sure if I'd call it decent, but you might want to check my Bytehound for more in-depth analysis: https://github.com/koute/bytehound

It is like others relying on tracking memory allocations. Which has it"s place, but to really analyze memory usage you would need to be able to analyze which "objects" are keeping other objects alive (dominator tree in MAT)

Re: Memory Profiling: Introduction

#14

Post by milianw »

kohlerm wrote: post #12
milianw wrote: post #10

Earlier quoted context omitted.

What is not decent about heaptrack or similar tools?

It does not really allow you to analyze what contributes to memory usage in a graph of "objects".

So more something like dhat from valgrind? see https://valgrind.org/docs/manual/dh-manual.html ?

Re: Memory Profiling: Introduction

#15

Post by milianw »

kohlerm wrote: post #13
kouteiheika wrote: post #8

Earlier quoted context omitted.

> do we really still not have a decent memory usage analysis tool for c/ C++ ? Shameless plug: not sure if I'd call it decent, but you might want to check my Bytehound for more in-depth analysis: https://github.com/koute/bytehound

It is like others relying on tracking memory allocations. Which has it"s place, but to really analyze memory usage you would need to be able to analyze which "objects" are keeping other objects alive (dominator tree in MAT)

IIUC then you might be looking for https://github.com/facebookexperimental/object-introspection - very new and not yet easily usable outside of the Meta sphere, but goes into that direction, or?

Re: Memory Profiling: Introduction

#17

Post by malkia »

kouteiheika wrote: post #9
malkia wrote: post #4

Some of our tools do so much allocation that capturing all the information (using MTuner) to disk and later loading require by itself hundreths of GB. Instead I've added random sampling - e.g. if ptr % modulo > level - output it or not. Another factor that slows down is doing a callstack capture. It's not for free at all, like on Windows it has to go through the exception handlers, etc. I think perfetto simply captur…

> Instead I've added random sampling - e.g. if ptr % modulo > level - output it or not. A good alternative that also cuts down on the amount of data significantly is to strip away temporary allocations, say, only emit those allocations which were alive for at least X seconds.

Ah good idea! It kind of calls for out-of-process to filter this out, or post-process (maybe there is common tooling around).

Re: Memory Profiling: Introduction

#18

Post by malkia »

milianw wrote: post #11
malkia wrote: post #4

Some of our tools do so much allocation that capturing all the information (using MTuner) to disk and later loading require by itself hundreths of GB. Instead I've added random sampling - e.g. if ptr % modulo > level - output it or not. Another factor that slows down is doing a callstack capture. It's not for free at all, like on Windows it has to go through the exception handlers, etc. I think perfetto simply captur…

Heaptrack can handle millions of allocations per second, and thanks to the deduplication built into the trace file format combined with zstd compression, the overhead is pretty manageable. It's pretty hard to get "hundreds of GB" of data recorded. Another thing: you can always just runtime attach and profile a partial time to reduce the amount of data recorded. Finally: if your tools do so many allocations, maybe you…

Thanks! Should look into heaptrack - reading that it's linux only, but probably can be adopted for Windows?

Re: Memory Profiling: Introduction

#19

Post by milianw »

malkia wrote: post #18
milianw wrote: post #11

Earlier quoted context omitted.

Heaptrack can handle millions of allocations per second, and thanks to the deduplication built into the trace file format combined with zstd compression, the overhead is pretty manageable. It's pretty hard to get "hundreds of GB" of data recorded. Another thing: you can always just runtime attach and profile a partial time to reduce the amount of data recorded. Finally: if your tools do so many allocations, maybe you…

Thanks! Should look into heaptrack - reading that it's linux only, but probably can be adopted for Windows?

MTuner exists for Windows which is similar but different. Porting heaptrack is not straight forward, as it heavily relies on libraries that do not work on Windows or do not support Windows executables and debugging information, such as libunwind and elfutils.
Post reply on HN