Dangit. I was thinking of writing an AST inspector name pyspy.
Show HN: Py-spy – A new sampling profiler for Python programs
21–30 of 44 posts
Re: Show HN: Py-spy – A new sampling profiler for Python programs
#22This is really fantastic. I just managed to find a 3x speedup on a compute heavy job I run using Py-spy - there was an unneeded hotspot in a library I'm using that I didn't previously suspect. It would have taken a long time using kernprof to dig in through the call stack to find the issue.
Re: Show HN: Py-spy – A new sampling profiler for Python programs
#23Earlier quoted context omitted.
I've never understood why flame graphs are better than the normal presentation of inclusive and exclusive timings in performance tools, even if they're not "modern", but embody some decades' experience. Anyone care to explain?
I'm far from a performance expert, but my impression is: It shows the call paths to the functions and what part each path took, that's not so obvious from the typical table. On the other hand, finding functions that are called quite a lot all over the place and add up is easier in the table, so it's not become useless.
The latter can be accomplished with inverted flame graphs (sometimes called icicle graphs) which show the call stack inverted
Re: Show HN: Py-spy – A new sampling profiler for Python programs
#24Gave it a try, and wasn’t expecting to have to execute via sudo on macOS. I almost never use sudo, so this stands out as quite unexpected for a developer tool. Is there something off with my system, or are sudo privileges always going to be required on macOS?
Edit: I pip installed it into a virtualenv, if that matters.
Re: Show HN: Py-spy – A new sampling profiler for Python programs
#25Does it support python multiprocessing? Basically nothing out there does that I've found, and it's a really major pain-point for me.
It says you pass it a pid, so yes
Re: Show HN: Py-spy – A new sampling profiler for Python programs
#26Earlier quoted context omitted.
yappi? https://bitbucket.org/sumerc/yappi/
> If you want to profile a multi-threaded application, you must give an entry point to these profilers and then maybe merge the outputs. It basically boils down to (currently) doing multiprocessing profiling is a giant pain in the ass, you have to manually attach the profiler yourself if you ever launch another process, and every profiled process produces it's own output file. It's not impossible , it's just very ann…
As a heads-up if you hadn't already seen it 3.7 added fork callbacks for stuff like this - https://docs.python.org/3/library/os.html#os.register_at_for... - much nicer than patching-and-praying.
Re: Show HN: Py-spy – A new sampling profiler for Python programs
#27This is great!
I also took the liberty to add this (and setuptools_rust) to Arch Linux's AUR: https://aur.archlinux.org/packages/python-py-spy
Re: Show HN: Py-spy – A new sampling profiler for Python programs
#28Gave it a try, and wasn’t expecting to have to execute via sudo on macOS. I almost never use sudo, so this stands out as quite unexpected for a developer tool. Is there something off with my system, or are sudo privileges always going to be required on macOS? Edit: I pip installed it into a virtualenv, if that matters.
https://github.com/benfred/py-spy#when-do-you-need-to-run-as...
tl;dr - yes, it's a limitation(?) of macOS syscalls.
Re: Show HN: Py-spy – A new sampling profiler for Python programs
#29Earlier quoted context omitted.
If you're talking about the flame graphs, they're a fairly common feature of modern profilers. The oldest implementation I know of is at https://github.com/brendangregg/FlameGraph .
I've never understood why flame graphs are better than the normal presentation of inclusive and exclusive timings in performance tools, even if they're not "modern", but embody some decades' experience. Anyone care to explain?
Flame graphs have extra information in the form of seeing the caller and callees. That, and I find the graphical presentation to be easier to scan.
Re: Show HN: Py-spy – A new sampling profiler for Python programs
#30Earlier quoted context omitted.
> If you want to profile a multi-threaded application, you must give an entry point to these profilers and then maybe merge the outputs. It basically boils down to (currently) doing multiprocessing profiling is a giant pain in the ass, you have to manually attach the profiler yourself if you ever launch another process, and every profiled process produces it's own output file. It's not impossible , it's just very ann…
As a heads-up if you hadn't already seen it 3.7 added fork callbacks for stuff like this - https://docs.python.org/3/library/os.html#os.register_at_for... - much nicer than patching-and-praying.
Yeah, saw that while looking about for atfork stuff.
Ironically, I was involved in https://bugs.python.org/issue6721 which is one of the major bugs leading to the acceptance of https://bugs.python.org/issue16500, which is the patch including atfork().