Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyInstrument – A statistical Python profile that focuses on the slow parts

github.com

Post reply on HN
Print view View original

11–20 of 23 posts

Re: PyInstrument – A statistical Python profile that focuses on the slow parts

#11

Post by baq »

abuckenheimer wrote: post #3

Very cool, py-spy[1] has been an invaluable tool in my development process since jvns blogged[2] about it. The power of being able to visualize where your code is spending its time is so obvious and I'm glad people are building tools to make that easier. As a quick compare and contrast between py-spy and pyinstrument it looks like py-spy has the advantage of being able to attach to an already running process which is…

py-spy is absolutely invaluable for the attach-to-process feature. also, nothing beats the adrenaline rush of attaching to a prod process and leaving it in a paused state after collecting a profile (true story! SIGCONT is your friend).

Re: PyInstrument – A statistical Python profile that focuses on the slow parts

#12

Post by formerly_proven »

abuckenheimer wrote: post #3

Very cool, py-spy[1] has been an invaluable tool in my development process since jvns blogged[2] about it. The power of being able to visualize where your code is spending its time is so obvious and I'm glad people are building tools to make that easier. As a quick compare and contrast between py-spy and pyinstrument it looks like py-spy has the advantage of being able to attach to an already running process which is…

I see a lot of tracing and sampling profilers for Python, but are there any profilers for manual instrumentation (goes by a bunch of names, frame profiling, performance telemetry, APM etc.)?

Re: PyInstrument – A statistical Python profile that focuses on the slow parts

#13

Post by formerly_proven »

oefrha wrote: post #2

> The standard Python profilers profile and cProfile show you a big list of functions, ordered by the time spent in each function. This is great, but it can be difficult to interpret why those functions are getting called. It's more helpful to know why those functions are called, and which parts of user code were involved. Note that you can use something like gprof2dot to convert pstats dump from cProfile to a visual…

pyprof2calltree is much better.

Re: PyInstrument – A statistical Python profile that focuses on the slow parts

#14

Post by MikeTheGreat »

The title for this post has a typo - it should be "profileR", not "profile" (er, without the capitalization on the R :) )

I could guess from context, but thought it might be good to point out.

Source: From the repo: "Pyinstrument is a Python profiler"

(Feel free to delete this comment after fixing the typo, or not :) )

Re: PyInstrument – A statistical Python profile that focuses on the slow parts

#15

Post by theptip »

I like how easy this is to wire up to a Django API, took me a few seconds to this hooked up on my local machine.

This gives nicer summarization/presentation than Django Debug Toolbar's profiler, so seems like a good one to have in the toolbox.

Re: PyInstrument – A statistical Python profile that focuses on the slow parts

#16

Post by nomel »

oefrha wrote: post #2

> The standard Python profilers profile and cProfile show you a big list of functions, ordered by the time spent in each function. This is great, but it can be difficult to interpret why those functions are getting called. It's more helpful to know why those functions are called, and which parts of user code were involved. Note that you can use something like gprof2dot to convert pstats dump from cProfile to a visual…

I've been using pyinstrument for a while.

The biggest problem with the standard profiler is that the reported times are not split by code path. For example, if you have two parts of your code that call the same library function, and you want to know which path is the slow path...you can't. The time reported for each line is a sum of all times/paths it was called. Worse, the visualization tools don't hint that this is the case, so you end up with very incorrect plots. Pyinstrument will give you the time, by path. Super useful, and a huge time saver!

Re: PyInstrument – A statistical Python profile that focuses on the slow parts

#17

Post by nomel »

pansa2 wrote: post #5

> Shows you why your code is slow! Because you wrote it in Python. Seriously, Python is probably the slowest mainstream language of all. If you’re building something where performance matters, you should be using a different language.

> If you’re building something where performance matters, you should be using a different language.

And the python ecosystem understands this, with those performance critical bits being implemented in C/fortran/whatever, not pure python.

Re: PyInstrument – A statistical Python profile that focuses on the slow parts

#18

Post by eliwe »

abuckenheimer wrote: post #3

Very cool, py-spy[1] has been an invaluable tool in my development process since jvns blogged[2] about it. The power of being able to visualize where your code is spending its time is so obvious and I'm glad people are building tools to make that easier. As a quick compare and contrast between py-spy and pyinstrument it looks like py-spy has the advantage of being able to attach to an already running process which is…

You can use https://github.com/4rtzel/tfg for an interactive console flamegraph. It supports opening pyspy files.

Re: PyInstrument – A statistical Python profile that focuses on the slow parts

#20

Post by jrichardshaw »

I'm a big fan of pyinstrument. Many of the newer profilers, (e.g. py-spy) attach to a process externally via SYS_PTRACE and though that seems great in many ways, it is very much a no-go when you're running code on an HPC cluster and you don't have root access.
Post reply on HN