PyInstrument – A statistical Python profile that focuses on the slow parts • Hacker News Board

Re: PyInstrument – A statistical Python profile that focuses on the slow parts

#21

Post by kapilvt » Wed, Oct 21, 2020, 12:07 AM UTC

jrichardshaw wrote: post #20

I'm a big fan of pyinstrument. Many of the newer profilers, (e.g. py-spy) attach to a process externally via SYS_PTRACE and though that seems great in many ways, it is very much a no-go when you're running code on an HPC cluster and you don't have root access.

ptrace is quite slow, py-spy reads memory and reconstructs stack frames which is much lower overhead for the profiled app.

from their github readme:

  py-spy works by directly reading the memory of the python program using the process_vm_readv system call on Linux, the vm_read call on OSX or the ReadProcessMemory call on Windows.

Re: PyInstrument – A statistical Python profile that focuses on the slow parts

#22

Post by gnufx » Wed, Oct 21, 2020, 9:21 AM UTC

jrichardshaw wrote: post #20

I'm a big fan of pyinstrument. Many of the newer profilers, (e.g. py-spy) attach to a process externally via SYS_PTRACE and though that seems great in many ways, it is very much a no-go when you're running code on an HPC cluster and you don't have root access.

On an HPC system, why wouldn't you use normal HPC-type performance tools? It will probably give you better analysis tools and allow you to profile/trace the libraries which are probably providing the performance. At least Score-p, Extrae, and TAU have Python support and have sophisticated viewers with support for call trees and inclusive/exclusive views. When I looked (maybe a couple of years ago) I couldn't find anything in the Python ecosystem to do that, though there may be now.

Re: PyInstrument – A statistical Python profile that focuses on the slow parts

#23

Post by gnufx » Wed, Oct 21, 2020, 9:40 AM UTC

formerly_proven wrote: post #12

abuckenheimer wrote: post #3

Very cool, py-spy[1] has been an invaluable tool in my development process since jvns blogged[2] about it. The power of being able to visualize where your code is spending its time is so obvious and I'm glad people are building tools to make that easier. As a quick compare and contrast between py-spy and pyinstrument it looks like py-spy has the advantage of being able to attach to an already running process which is…

I see a lot of tracing and sampling profilers for Python, but are there any profilers for manual instrumentation (goes by a bunch of names, frame profiling, performance telemetry, APM etc.)?

As an example, https://github.com/score-p/scorep_binding_python#user-region... is the support in the HPC system https://score-p.org (though it focuses on parallel programs). http://apps.fz-juelich.de/scalasca/releases/cube/4.5/docs/gu... is the associated profile analysis tool.