Live data from Hacker News

Show HN: Py-spy – A new sampling profiler for Python programs

github.com

11–20 of 44 posts

Re: Show HN: Py-spy – A new sampling profiler for Python programs

#11
post #2

"top" for python programs. Thats pretty awesome - not sure if this has existed in other traces, but the output is great.

Rbspy, the inspiration for this tool, has a similar default output, which you can see in the documentation, https://rbspy.github.io/.

Re: Show HN: Py-spy – A new sampling profiler for Python programs

#13
The adaptations of HPC-type performance tools to Python and called non-Python, specifically parallel, libraries might be of interest:

TAU: https://www.cs.uoregon.edu/research/tau/docs/newguide/ch03s0... Extrae/Paraver: https://www.researchgate.net/publication/317485375_Performan... llel_Python_Applications Score-p/Scalasca: http://score-p.org https://github.com/score-p/scorep_binding_python

Re: Show HN: Py-spy – A new sampling profiler for Python programs

#15
post #2

"top" for python programs. Thats pretty awesome - not sure if this has existed in other traces, but the output is great.

If you're talking about the flame graphs, they're a fairly common feature of modern profilers. The oldest implementation I know of is at https://github.com/brendangregg/FlameGraph .

I've never understood why flame graphs are better than the normal presentation of inclusive and exclusive timings in performance tools, even if they're not "modern", but embody some decades' experience. Anyone care to explain?

Re: Show HN: Py-spy – A new sampling profiler for Python programs

#16
post #15

Earlier quoted context omitted.

If you're talking about the flame graphs, they're a fairly common feature of modern profilers. The oldest implementation I know of is at https://github.com/brendangregg/FlameGraph .

I've never understood why flame graphs are better than the normal presentation of inclusive and exclusive timings in performance tools, even if they're not "modern", but embody some decades' experience. Anyone care to explain?

I'm far from a performance expert, but my impression is:

It shows the call paths to the functions and what part each path took, that's not so obvious from the typical table. On the other hand, finding functions that are called quite a lot all over the place and add up is easier in the table, so it's not become useless.

Re: Show HN: Py-spy – A new sampling profiler for Python programs

#17
post #7

Wonderful. Can the data it produces be munged into something KCachegrind can show?

Not yet - but I'm hoping to have a version that supports this next week. Will update this issue when it's done: https://github.com/benfred/py-spy/issues/3

Re: Show HN: Py-spy – A new sampling profiler for Python programs

#18

Many thanks for building and releasing this. It's ridiculously easy to install (especially in virtualenvs) and very powerful. When I push '3' or '4', I get informative, stable output. Minor feature request: an explicit 'pause' button would make it easier to copy file paths from the output. Ctrl-S is a reasonable alternative, but it's a little hacky. Also, it would be nice to somehow eliminate time spent in poll() fro…

Thanks! Both of your suggestions totally make sense. I've created an issue to track the poll() issue here https://github.com/benfred/py-spy/issues/13 - I think that should be an easy fix.

Re: Show HN: Py-spy – A new sampling profiler for Python programs

#19
post #2

"top" for python programs. Thats pretty awesome - not sure if this has existed in other traces, but the output is great.

If you're talking about the flame graphs, they're a fairly common feature of modern profilers. The oldest implementation I know of is at https://github.com/brendangregg/FlameGraph .

This is the same script. The Rust code is just invoking Perl to generate it.

Re: Show HN: Py-spy – A new sampling profiler for Python programs

#20

Does it support python multiprocessing? Basically nothing out there does that I've found, and it's a really major pain-point for me.

yappi? https://bitbucket.org/sumerc/yappi/

> If you want to profile a multi-threaded application, you must give an entry point to these profilers and then maybe merge the outputs.

It basically boils down to (currently) doing multiprocessing profiling is a giant pain in the ass, you have to manually attach the profiler yourself if you ever launch another process, and every profiled process produces it's own output file.

It's not impossible, it's just very annoying. I've been vaguely meaning to write a thing which attaches to the fork() call and automatically starts the profiler in the child-process, and handles aggregating all the results back to a single output when all children exit.

Post reply on HN