Incrementally improving the performance of a Python script • Hacker News Board

Re: Incrementally improving the performance of a Python script

#21

Post by anaphor » Sat, Dec 15, 2018, 8:20 PM UTC

jacobsimon wrote: post #11

Python has some unusual performance behaviors. IIRC you can also speed up the performance of your program a lot by assigning intermediate variables instead of referencing properties, for example: [A.b[i] for i in range(100)] is a lot slower than: B = A.b [B[i] for i in range (100)]

Properties are not the same thing as attributes though. If `A.b` is a property (has a __get__ method associated with it), then this could easily break your code because `A.b` is supposed to change values. If it's an attribute that stays the same, then yeah I'll grant you that it could speed things up if it's being accessed frequently in a loop. There's not really any way for Python to tell if it can be optimized statically.

https://ideone.com/4R0suA

Re: Incrementally improving the performance of a Python script

#22

Post by sifoobar » Sat, Dec 15, 2018, 8:27 PM UTC

This is just a reflection from recent work on interpreter internals, but I figured it could be relevant to someone walking the same paths.

Allocating registers for all local vars statically means scopes have different sizes, which in turn complicates slab allocation and/or reuse. In return for being easier to reason about (except for the main scope issue) and saving space.

I opted for a fixed number of linearly assigned registers per scope in Snigl [0]; once the limit is reached, remaining variables are stored in a table. Which means I sort of get both, since additional scopes may be added using {} (it could make sense to add a scope: keyword to Python) if that becomes an issue.

It's all compromises, all the way down.

[0] https://gitlab.com/sifoo/snigl

Re: Incrementally improving the performance of a Python script

#23

Post by viraptor » Sat, Dec 15, 2018, 9:14 PM UTC

paavoova wrote: post #16

jacobsimon wrote: post #11

Python has some unusual performance behaviors. IIRC you can also speed up the performance of your program a lot by assigning intermediate variables instead of referencing properties, for example: [A.b[i] for i in range(100)] is a lot slower than: B = A.b [B[i] for i in range (100)]

I'm not sure about other implementations, but "optimizing" CPython ends up being optimizing against counter-intuitive interpreter internals rather than time-complexity of the code. For example, in CPython 3.6, n = 0 d = 100 for i in range(10**6): n += i if n >= d: n %= d is slower than n = (n + i) % d This counter to lower level languages, where dividing by a variable is costly, and the CPU can predict the pipeline t…

> can predict the pipeline to be false most of the time in the conditional and thus skip it.

I think there's a bug in the code then. It will be skipped a few times, but for i between 100 and 10^6, the condition is guaranteed true every time.

Re: Incrementally improving the performance of a Python script

#24

Post by kevinventullo » Sun, Dec 16, 2018, 12:58 AM UTC

Slightly OT from the main takeaway, but I wonder if this could be sped up further by only doing a single pass, and maintaining a stack (implicitly sorted) of elements which are greater than everything seen so far, and popping them off when they're greater than the current element.