Profiling Python with cProfile

#1

Post by ymichael » Sat, Mar 08, 2014, 1:36 PM UTC

Profiling Python with cProfile
yrmichael.com

Re: Profiling Python with cProfile

#2

Post by RogerL » Sat, Mar 08, 2014, 3:27 PM UTC

You'll perhaps take this as snark, but I am not going to read light gray text on a white background with a poorly rendered font (windows 7, firefox 27, 47yo eyes).

Re: Profiling Python with cProfile

#3

Post by nwenzel » Sat, Mar 08, 2014, 3:36 PM UTC

Very helpful to have a UI for digging through profiler results. I use the django debug toolbar but I think your profiler will be incredibly helpful for finding bottlenecks. Thanks for posting, thanks for sharing.

Re: Profiling Python with cProfile

#4

Post by nwenzel » Sat, Mar 08, 2014, 3:39 PM UTC

RogerL wrote: post #2

You'll perhaps take this as snark, but I am not going to read light gray text on a white background with a poorly rendered font (windows 7, firefox 27, 47yo eyes).

For less snark and more helpfulness, consider removing "I am not going to read". I think you'll be happier and less likely to be perceived as snarky.

Re: Profiling Python with cProfile

#5

Post by ymichael » Sat, Mar 08, 2014, 3:39 PM UTC

RogerL wrote: post #2

You'll perhaps take this as snark, but I am not going to read light gray text on a white background with a poorly rendered font (windows 7, firefox 27, 47yo eyes).

Ahh sorry about that. I'm like the most terrible UI person. ever... thanks for the feedback. made the text darker.

Re: Profiling Python with cProfile

#6

Post by tgb » Sat, Mar 08, 2014, 3:39 PM UTC

Kernprof and lineprof are also helpful.

Re: Profiling Python with cProfile

#7

Post by jbaiter » Sat, Mar 08, 2014, 3:50 PM UTC

nwenzel wrote: post #3

Very helpful to have a UI for digging through profiler results. I use the django debug toolbar but I think your profiler will be incredibly helpful for finding bottlenecks. Thanks for posting, thanks for sharing.

Then you might really enjoy runsnakerun: http://www.vrplumber.com/programming/runsnakerun/

Re: Profiling Python with cProfile

#8

Post by pekk » Sat, Mar 08, 2014, 3:54 PM UTC

I'm glad that there wasn't a "complain about the GIL" or "write everything in Go because Python is too slow" step in this optimization workflow

Re: Profiling Python with cProfile

#9

Post by habitue » Sat, Mar 08, 2014, 3:55 PM UTC

I'd say you had it right the first time with the list comprehension. List comprehensions in python are way faster than while/append loops because they're implemented in C. The issue is that you had the "not in" which is O(n) over a list (making the comprehension O(n^2) ). If the documents are hashable, I'd suggest making results a set. "not in" is constant time on a set. If a document is something not hashable, you can make results a dictionary and get the same constant time access.

In python function/method calls are expensive, so avoid them at all costs inside tight loops.

Re: Profiling Python with cProfile

#10

Post by m_ke » Sat, Mar 08, 2014, 4:19 PM UTC

habitue wrote: post #9

I'd say you had it right the first time with the list comprehension. List comprehensions in python are way faster than while/append loops because they're implemented in C. The issue is that you had the "not in" which is O(n) over a list (making the comprehension O(n^2) ). If the documents are hashable, I'd suggest making results a set. "not in" is constant time on a set. If a document is something not hashable, you c…

Thumbs up to this guy.

Function calls in tight loops are a killer. Property and global lookups are also "much" less efficient than stuff in the local scope.