Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memray – A Memory Profiler for Python

github.com

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Memray – A Memory Profiler for Python

#21

Post by irogers »

albertzeyer wrote: post #14

I collected a list of profilers (also memory profilers, also specifically for Python) here: https://github.com/albertz/wiki/blob/master/profiling.md Currently I actually need a Python memory profiler, because I want to figure out whether there is some memory leak in my application (PyTorch based training script), and where exactly (in this case, it's not a problem of GPU memory, but CPU memory). I tried Scalene ( htt…

For a collection of profilers you can also check out the Profilerpedia: https://profilerpedia.markhansen.co.nz/

Re: Memray – A Memory Profiler for Python

#22

Post by EasyMark »

ezequiel-garzon wrote: post #6

For those Python programmers out there, if you don't mind sharing your experiences, do you spend any time at all on the REPL? If so, what fraction of the time, approximately? Using IPython, JupyterLab, or something else? Or do you just run it directly from an VS Code or PyCharm? Anything you may want to add about your routine would be appreciated. Oh, if you (experienced programmer or not) happen to know about a good…

for python: 90/100 times, I just run the code/tests/debugger, 8/100 times I'll pull out the code and step through it with my own inputs, 2/100 use a repl and break on areas of interest to debug because the debugger just isn't cooperating. It's just too easy to use the debugger in something like vscode to run a module, especially in python since you can just right click run any old module a lot of the time just make a __main__ and feed some parameters, step through it, unlike with a static language.

Re: Memray – A Memory Profiler for Python

#23

Post by [deleted] »

albertzeyer wrote: post #14

I collected a list of profilers (also memory profilers, also specifically for Python) here: https://github.com/albertz/wiki/blob/master/profiling.md Currently I actually need a Python memory profiler, because I want to figure out whether there is some memory leak in my application (PyTorch based training script), and where exactly (in this case, it's not a problem of GPU memory, but CPU memory). I tried Scalene ( htt…

[deleted]

Re: Memray – A Memory Profiler for Python

#24

Post by codedokode »

I have used memory profilers in PHP. In my experience, it is difficult to analyze why application uses lot of RAM just by looking at stack traces. It is better when you have a graph of references: which variable references which memory block. So that you can see, for example, that `cache.users[23].comments[56].attachments[2].binary_data` uses 10 Mb of RAM.

Re: Memray – A Memory Profiler for Python

#25

Post by sevagh »

metadat wrote: post #11
loeg wrote: post #3

Interesting testimonials section. Two out of five are "this looks interesting" and "this might be useful." Another two seem unclear on if they've actually used it. The last one does seem to be from an actual user (and is very positive, for what it's worth).

Where do you see the testimonials section?

There are 5 images of testimonials in the top part of the README

Re: Memray – A Memory Profiler for Python

#26

Post by colechristensen »

codedokode wrote: post #24

I have used memory profilers in PHP. In my experience, it is difficult to analyze why application uses lot of RAM just by looking at stack traces. It is better when you have a graph of references: which variable references which memory block. So that you can see, for example, that `cache.users[23].comments[56].attachments[2].binary_data` uses 10 Mb of RAM.

And flame graphs excel and this kind of thing

https://www.brendangregg.com/flamegraphs.html

Re: Memray – A Memory Profiler for Python

#27

Post by kgm »

When it comes to profiling in Python, never underestimate the power of the standard library's profiler. You can supply it with a custom timing function when instantiating the Profile type [1], and as far as the module is concerned, this can be any function which returns a monotonically-increasing counter.

This means that you can turn the standard profiler into a memory profiler by providing a timing function which reports either total memory allocation or a meaningful proxy for allocation. I've had good results in the past using a timing function which returns the number of minor page faults (via resource.getrusage).

[1] https://docs.python.org/3/library/profile.html#profile.Profi...

Re: Memray – A Memory Profiler for Python

#28

Post by dang »

Discussed recently here:

Memray: Python memory profiler - https://news.ycombinator.com/item?id=38561682 - Dec 2023 (21 comments)

(Reposts are fine after a year or so. This is in the FAQ: https://news.ycombinator.com/newsfaq.html.)

Also related:

Memray - https://news.ycombinator.com/item?id=31102918 - April 2022 (2 comments)

Memray: a memory profiler for Python - https://news.ycombinator.com/item?id=31102089 - April 2022 (48 comments)

Re: Memray – A Memory Profiler for Python

#29

Post by horaborza »

breatheoften wrote: post #17

I used this tool recently to debug some issues where one of our batch tasks was using more memory than it should and sometimes OOMing in an environment where it shouldn't come close to hitting a memory ceiling. Found the issue almost immediately with the high watermark analysis which provides visibility into the part of the codebase/call stack that allocated every bit of allocated memory that was still live at the hi…

Ah xarray, both incredibly useful and an extremely obnoxious nightmare

Re: Memray – A Memory Profiler for Python

#30

Post by pablogsal »

cozos wrote: post #15

Ive used Memray and it works great locally. But when I deployed my application over long running processes (i.e. in production) because I want to see memory usage over a long period of time, the profiler outputs get really large, like hundreds of gbs. They cause disk outages and also take forever to download and visualize with the flamegraphs. What do people use to understand memory usage of long running workloads in…

Have you tried aggregated capture files? https://bloomberg.github.io/memray/run.html#aggregated-captu...

With that option the files are much much smaller and much easier to analyse

Post reply on HN