Detect memory leaks of C extensions with psutil and psleak
1–10 of 11 posts
Re: Detect memory leaks of C extensions with psutil and psleak
#2I have released new psutil 7.2.0, which includes 2 new APIs to inspect C heap memory allocations.
I have also released a new tool called psleak, which detects memory leaks in C extension modules.
Re: Detect memory leaks of C extensions with psutil and psleak
#3Hey, thank you so much for all your hard work on psutil, it is my goto import!
And this looks like a really interesting development you have created to add to the available tools for detecting those tricky memory leaks.
I have done some work in this space as well such as how to write a Python C extension without leaks in the first place [0]. This also contains a section on memory leaks, using tools like your psutil [1] This also shows where the Python documentation for C extensions contains errors, omissions or is misleading. These errors can trap the unwary.
Also I developed a Python memory tracer, pymemtrace, that uses a variety of techniques (including using your psutil!) to track memory usage at different costs and levels of granularity [2].
Now psutil 7.2.0 is out I'll update both those projects.
Thanks again for all your hard work.
[0] Project: https://github.com/paulross/PythonExtensionPatterns Docs: https://pythonextensionpatterns.readthedocs.io/en/latest/ind...
[1] https://pythonextensionpatterns.readthedocs.io/en/latest/mem...
[2] Project: https://github.com/paulross/pymemtrace Docs: https://pymemtrace.readthedocs.io/
Re: Detect memory leaks of C extensions with psutil and psleak
#4This is awesome, thanks a lot! I'm going to introduce this in the test suites of the extension modules I maintain [0][1] and, if all goes well, eventually at work [2].
I really appreciate the support for Windows as that platform is currently underserved [3] when it comes to such memory tooling.
[0]: https://github.com/jcrist/msgspec
[1]: https://github.com/ofek/coincurve
Re: Detect memory leaks of C extensions with psutil and psleak
#5The attached post feels AI-generated/AI-edited. I don’t know why that detracts from the usefulness of this utility, but it — at least how I’m reacting to it — seems to.
For me, it reads as not having any soul to it, as well as changing who its target reader is throughout the post. The choice of emphasis comes across weirdly too.
I think this is a wonderful tool, but whether it’s because the author uses AI to help them write this post, or that there’s AI edited content everywhere now, I’m weary of reading and trusting something that reads like this post does.
I don’t know if this is overblown or meta-commentary on the state of online posts these days, but it’s how I’m feeling, more and more, when I see online writings.
Re: Detect memory leaks of C extensions with psutil and psleak
#6Hey, thank you so much for all your hard work on psutil, it is my goto import! And this looks like a really interesting development you have created to add to the available tools for detecting those tricky memory leaks. I have done some work in this space as well such as how to write a Python C extension without leaks in the first place [0]. This also contains a section on memory leaks, using tools like your psutil […
Your C extension guide looks very useful and I quite like the foreword/history behind it. Have you considered updating the resource to account for the freethreaded mode (which will eventually become the default) on 3.14+?
Re: Detect memory leaks of C extensions with psutil and psleak
#7The attached post feels AI-generated/AI-edited. I don’t know why that detracts from the usefulness of this utility, but it — at least how I’m reacting to it — seems to. For me, it reads as not having any soul to it, as well as changing who its target reader is throughout the post. The choice of emphasis comes across weirdly too. I think this is a wonderful tool, but whether it’s because the author uses AI to help the…
The post is entirely authentic; it matches the writing style of the author from before the LLM boom and discusses work that the human author recently released. Can you pinpoint what makes you feel that way?
edit: I asked for explanation before the post was edited to expand on that. I disagree but am sympathetic to the weariness of reading content now that AI use is widespread.
Re: Detect memory leaks of C extensions with psutil and psleak
#8Hey, thank you so much for all your hard work on psutil, it is my goto import! And this looks like a really interesting development you have created to add to the available tools for detecting those tricky memory leaks. I have done some work in this space as well such as how to write a Python C extension without leaks in the first place [0]. This also contains a section on memory leaks, using tools like your psutil […
Your C extension guide looks very useful and I quite like the foreword/history behind it. Have you considered updating the resource to account for the freethreaded mode (which will eventually become the default) on 3.14+?
For sure, it is on my TODO list. It takes a lot of work to explore each new Python C API and I'll get around to it when I can.
Re: Detect memory leaks of C extensions with psutil and psleak
#9The attached post feels AI-generated/AI-edited. I don’t know why that detracts from the usefulness of this utility, but it — at least how I’m reacting to it — seems to. For me, it reads as not having any soul to it, as well as changing who its target reader is throughout the post. The choice of emphasis comes across weirdly too. I think this is a wonderful tool, but whether it’s because the author uses AI to help the…
These kinds of comments (accusing OC of being ai gen) seem like they're showing up in every thread and they're always useless and insulting.
HN needs a rule to ban these
Re: Detect memory leaks of C extensions with psutil and psleak
#10In a pinch one way to deal with memory leaks is to wrap the offending work in a multiprocessing pool, forcing it to live in an ephemeral process