Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memray: a memory profiler for Python

github.com

Post reply on HN
Print view View original

21–30 of 53 posts

Re: Memray: a memory profiler for Python

#21

Post by pablogsal »

itamarst wrote: post #19
pablogsal wrote: post #16

Earlier quoted context omitted.

It does much more than that! It tracks every single allocation and dumps it to a file that can later be analysed in many ways. Currently our reporters report peak memory (and leaked memory at the end of the execution) but technically any other reporter can be used. For example, we plan to allow to generate flame-graphs at arbitrary points in the execution and much more!

BTW I am starting a Slack for devs working on profilers, would be great to have you all join, I'd love to hear more about the ELF patching technique (Fil uses LD_PRELOAD and macOS equivalent).

Yeah, please hook me in! You know my email :)

Re: Memray: a memory profiler for Python

#22

Post by itamarst »

neves wrote: post #13

I've just put in production an app that intermittently can't allocate enough memory. Is this the best tool to debug it? I've never had to debug memory problems in Python.

Fil can dump memory profiling reports on failed allocations; as other commenter said, Linux by default happily just gives you memory even if doesn't have any, so failed allocation implies giant allocation.

https://pythonspeed.com/fil/docs/oom.html

Re: Memray: a memory profiler for Python

#23

Post by itamarst »

pablogsal wrote: post #21
itamarst wrote: post #19

Earlier quoted context omitted.

BTW I am starting a Slack for devs working on profilers, would be great to have you all join, I'd love to hear more about the ELF patching technique (Fil uses LD_PRELOAD and macOS equivalent).

Yeah, please hook me in! You know my email :)

I'm on vacation, will do Monday.

Re: Memray: a memory profiler for Python

#24

Post by pablogsal »

itamarst wrote: post #15

I'm excited to see more profiling tools for Python! This sounds like it does peak memory, which is critical for batch jobs, since that's the bottleneck. Memory is fundamentally different than performance in that it's a limited resource, instead of cumulative cost; making any part of the program faster almost always helps speed up the program (at least a little, or at least reduces CPU load), but optimizing non-peak m…

One thing to note is that we support tracking forked/child processes as well!

>> - Much lower overhead, sounds like.

I actually think Fil has the potential to be faster in some situations because it seems that aggregates the flamegraph in memory. Memray needs to do a ton of I/O to disk holding a lock so if is under very heavy pressure it will be a bit slow.

Here are some non-very scientific tests running the "test_list" file from the CPython test suite:

* With pymalloc active (not a lot of heavy presure):

$ fil-profile run -m test test_list ... Total duration: 278 ms

$ memray3.10 run -m test test_list ... Total duration: 128 ms

* With pymalloc not active (heavy presure):

$ PYTHONMALLOC=malloc fil-profile run -m test test_list ... Total duration: 278 ms

$ PYTHONMALLOC=malloc memray3.10 run -m test test_list

Total duration: 344 ms

So as you can see Fil is 20% faster than memray in this scenario. This means that Fil is doing a fantastic job! We spent a lot of time optimizing memray and the fact that Fil can beat it is a testament to Fil's quality :)

Re: Memray: a memory profiler for Python

#25

Post by noitpmeder »

Want to chime in to say it's awesome that Bloomberg is releasing this product to the world. I don't think many financial organizations would even consider releasing anything as open source, even if it is as general purpose as this tool. My current one (HFT,MM) wouldn't go for it.

It also shows good engineering practices. I'd venture a guess that they are probably one of the better ones in the space.

Makes me want to work there! Maybe one day...

Re: Memray: a memory profiler for Python

#26

Post by noitpmeder »

neves wrote: post #13

I've just put in production an app that intermittently can't allocate enough memory. Is this the best tool to debug it? I've never had to debug memory problems in Python.

Is this due to memory constraints on your system? If so, then this would definitely help you profile _what_ in your code is trying to claim so much memory.

Re: Memray: a memory profiler for Python

#27

Post by foota »

itamarst wrote: post #15

I'm excited to see more profiling tools for Python! This sounds like it does peak memory, which is critical for batch jobs, since that's the bottleneck. Memory is fundamentally different than performance in that it's a limited resource, instead of cumulative cost; making any part of the program faster almost always helps speed up the program (at least a little, or at least reduces CPU load), but optimizing non-peak m…

>> but optimizing non-peak memory has no impact. You have to be able to identify the peak in order to reduce memory usage.

In some sense this is only true if you're the end user of a platform, if you're trying to pack jobs onto machines then you actually do care about the utilization at any given time, since you can oversubscribe based on someone's max usage.

E.g., you can give everyone a limit based on their peak memory, but then bin pack based on their actual usage (and evict when you're wrong)

Re: Memray: a memory profiler for Python

#28

Post by pid-1 »

itamarst wrote: post #15

I'm excited to see more profiling tools for Python! This sounds like it does peak memory, which is critical for batch jobs, since that's the bottleneck. Memory is fundamentally different than performance in that it's a limited resource, instead of cumulative cost; making any part of the program faster almost always helps speed up the program (at least a little, or at least reduces CPU load), but optimizing non-peak m…

Unrelated, but thank you for your awesome writings!

Re: Memray: a memory profiler for Python

#29

Post by dbenamy »

neves wrote: post #13

I've just put in production an app that intermittently can't allocate enough memory. Is this the best tool to debug it? I've never had to debug memory problems in Python.

Datadog's profiler [0] might help. The python profiler has a heap profile type which shows (the call site which allocates) data using memory [1]. Feedback welcome!

Disclosure: I work on this :-)

[0] https://app.datadoghq.com/profiling

[1] From https://docs.datadoghq.com/tracing/profiler/search_profiles/

    > Heap Live Size
    > Shows the amount of heap memory allocated by each function that has not been garbage collected (yet). This is useful for investigating the overall memory usage of your service and identifying potential memory leaks.

Re: Memray: a memory profiler for Python

#30

Post by vdfs »

neves wrote: post #13

I've just put in production an app that intermittently can't allocate enough memory. Is this the best tool to debug it? I've never had to debug memory problems in Python.

Yes, I just give this tool a try and reduced our app initial memory allocation from 120MB to about 80MB in few minutes by moving imports around, ex: rarely called functions importing big 3rd party library.

You can use this to figure out what need to be optimized, but in your case optimization might not need to be easy, in above example you would have to re-implement the 3rd party library in memory efficient way or find an other one

Post reply on HN