Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memray: a memory profiler for Python

github.com

Post reply on HN
Print view View original

11–20 of 53 posts

Re: Memray: a memory profiler for Python

#11

Post by knlb2022 »

erwincoumans wrote: post #8
uniqueuid wrote: post #3

Tangential question: Does anybody have a good recommendation for a profiler that works well with massively async codebases? My experience has been that the concurrent nature of coroutines can make it hard to reason about what's going on at a particular point in time. If you don't know how many things you're awaiting on at a specific moment (and what potential external stuff they may be interacting with), it's not exa…

I like open source Perfetto UI (formerly Google Chrome about::/tracing) Wrote my own timing trace json export for Python and C++, across threads and processes. The docs have some pointers to creating your own traces with the Tracing SDK. See https://ui.perfetto.dev

I built https://github.com/kunalb/panopticon to export perfetto/chrome compatible traces and also draw arrows between async functions. (I think the arrows are only supported in about://tracing though).

Re: Memray: a memory profiler for Python

#12

Post by kungfufrog »

uniqueuid wrote: post #3

Tangential question: Does anybody have a good recommendation for a profiler that works well with massively async codebases? My experience has been that the concurrent nature of coroutines can make it hard to reason about what's going on at a particular point in time. If you don't know how many things you're awaiting on at a specific moment (and what potential external stuff they may be interacting with), it's not exa…

https://github.com/plasma-umass/scalene check out scalene!

Re: Memray: a memory profiler for Python

#15

Post by itamarst »

I'm excited to see more profiling tools for Python!

This sounds like it does peak memory, which is critical for batch jobs, since that's the bottleneck. Memory is fundamentally different than performance in that it's a limited resource, instead of cumulative cost; making any part of the program faster almost always helps speed up the program (at least a little, or at least reduces CPU load), but optimizing non-peak memory has no impact. You have to be able to identify the peak in order to reduce memory usage.

If you want peak memory profiling for Python that also runs on macOS, check out https://pythonspeed.com/fil/ (ARM support has some issues, but once I unpack my new Mac Mini I plan to fix it.)

Ways memray is better than Fil:

- Native callstacks.

- More kinds of reports, and ability to do custom post-processing of data.

- Much lower overhead (but not always, see reply).

- Subprocess support.

Fil I suspect has better flamegraphs: https://pythonspeed.com/articles/a-better-flamegraph/

And if you're running Python batch jobs, and want both peak memory and performance profiling in production, check out Sciagraph: https://pythonspeed.com/sciagraph/

(You can probably cobble together something like Sciagraph with py-spy + memray, but you won't e.g. get timeline reports designed with batch jobs in mind.)

Re: Memray: a memory profiler for Python

#16

Post by pablogsal »

itamarst wrote: post #15

I'm excited to see more profiling tools for Python! This sounds like it does peak memory, which is critical for batch jobs, since that's the bottleneck. Memory is fundamentally different than performance in that it's a limited resource, instead of cumulative cost; making any part of the program faster almost always helps speed up the program (at least a little, or at least reduces CPU load), but optimizing non-peak m…

It does much more than that! It tracks every single allocation and dumps it to a file that can later be analysed in many ways. Currently our reporters report peak memory (and leaked memory at the end of the execution) but technically any other reporter can be used. For example, we plan to allow to generate flame-graphs at arbitrary points in the execution and much more!

Re: Memray: a memory profiler for Python

#17

Post by orf »

neves wrote: post #13

I've just put in production an app that intermittently can't allocate enough memory. Is this the best tool to debug it? I've never had to debug memory problems in Python.

If this isn't due to latent memory pressure, it's usually because _something_ in your code is trying to allocate a gigantic amount of memory due to a bug. Think a 1 billion x 1 billion numpy array, or a billion element list.

Re: Memray: a memory profiler for Python

#18

Post by itamarst »

pablogsal wrote: post #16
itamarst wrote: post #15

I'm excited to see more profiling tools for Python! This sounds like it does peak memory, which is critical for batch jobs, since that's the bottleneck. Memory is fundamentally different than performance in that it's a limited resource, instead of cumulative cost; making any part of the program faster almost always helps speed up the program (at least a little, or at least reduces CPU load), but optimizing non-peak m…

It does much more than that! It tracks every single allocation and dumps it to a file that can later be analysed in many ways. Currently our reporters report peak memory (and leaked memory at the end of the execution) but technically any other reporter can be used. For example, we plan to allow to generate flame-graphs at arbitrary points in the execution and much more!

Cool! Fil also tracks every allocation too, although it doesn't dump that at the moment, just the resulting report.

Re: Memray: a memory profiler for Python

#19

Post by itamarst »

pablogsal wrote: post #16
itamarst wrote: post #15

I'm excited to see more profiling tools for Python! This sounds like it does peak memory, which is critical for batch jobs, since that's the bottleneck. Memory is fundamentally different than performance in that it's a limited resource, instead of cumulative cost; making any part of the program faster almost always helps speed up the program (at least a little, or at least reduces CPU load), but optimizing non-peak m…

It does much more than that! It tracks every single allocation and dumps it to a file that can later be analysed in many ways. Currently our reporters report peak memory (and leaked memory at the end of the execution) but technically any other reporter can be used. For example, we plan to allow to generate flame-graphs at arbitrary points in the execution and much more!

BTW I am starting a Slack for devs working on profilers, would be great to have you all join, I'd love to hear more about the ELF patching technique (Fil uses LD_PRELOAD and macOS equivalent).

Re: Memray: a memory profiler for Python

#20

Post by pablogsal »

itamarst wrote: post #18
pablogsal wrote: post #16

Earlier quoted context omitted.

It does much more than that! It tracks every single allocation and dumps it to a file that can later be analysed in many ways. Currently our reporters report peak memory (and leaked memory at the end of the execution) but technically any other reporter can be used. For example, we plan to allow to generate flame-graphs at arbitrary points in the execution and much more!

Cool! Fil also tracks every allocation too, although it doesn't dump that at the moment, just the resulting report.

Nice! Fil looks like an awesome profiler and is fantastic that works in other platforms. I am super excited to see more cool features and https://pythonspeed.com/sciagraph/ looks fantastic :)
Post reply on HN