Python Practices for Efficient Code: Performance, Memory, and Usability • Hacker News Board

Re: Python Practices for Efficient Code: Performance, Memory, and Usability

#41

Post by dom0 » Mon, Aug 21, 2017, 8:54 PM UTC

dr_zoidberg wrote: post #40

dom0 wrote: post #28

Earlier quoted context omitted.

The reason join is fast for these cases is because join basically converts all iterables to a list first and figures out how much it has to join exactly.

It walks the iterables (no need to convert them to lists), but its main trick is actually knowing how much memory it'll need for the end result, so no re-allocations happen, simply a one pass (through multiple iterables) copy.

No, it really converts all iterables to lists (there is a utility function in the C API to do just that, because it's kinda hairy to do by hand. PySequence_ToList or so). Otherwise it would not be possible to calculate how much space is required beforehand, since iterables cannot be walked twice.

Re: Python Practices for Efficient Code: Performance, Memory, and Usability

#42

Post by fnl » Sun, Aug 27, 2017, 9:01 AM UTC

dr_zoidberg wrote: post #14

davidwtbuxton wrote: post #10

> Use format instead of + for generating strings — In Python, str is immutable, so the left and right strings have to be copied into the new string for every pair of concatenations. It isn't always faster to use string formatting. $ python -m timeit -s 'a, b, c, d = "1234567890", "abcdefghij", "ABCDEFGHIJ", "0987654321"' 'a + b + c + d' 10000000 loops, best of 3: 0.181 usec per loop $ python -m timeit -s 'a, b, c, d…

On Python 2.7.10: In [2]: %timeit a+b+c+d The slowest run took 6.66 times longer than the fastest. This could mean that an intermediate result is being cached. 1000000 loops, best of 3: 247 ns per loop In [4]: %timeit "{}{}{}{}".format(a, b, c, d) The slowest run took 6.37 times longer than the fastest. This could mean that an intermediate result is being cached. 1000000 loops, best of 3: 709 ns per loop On Python 3.…

So the conclusion is: Have an existing list to join? Use str.join(list); Otherwise, operator+ might be your friend, but f-strings are probably cleaner to handle (but you only get those in Py3.6+). As to str.format(): Nah, not good.

Didn't expect str.format() to turn out to be "so" poor, though.

But this conclusion is taken w/o looking into the "old" formatting approach, `"%s%s%s%s" % (a, b, c, d)`, though.

Re: Python Practices for Efficient Code: Performance, Memory, and Usability

#43

Post by d0mine » Sun, Aug 27, 2017, 9:32 PM UTC

davidwtbuxton wrote: post #10

> Use format instead of + for generating strings — In Python, str is immutable, so the left and right strings have to be copied into the new string for every pair of concatenations. It isn't always faster to use string formatting. $ python -m timeit -s 'a, b, c, d = "1234567890", "abcdefghij", "ABCDEFGHIJ", "0987654321"' 'a + b + c + d' 10000000 loops, best of 3: 0.181 usec per loop $ python -m timeit -s 'a, b, c, d…

On pypy, + also wins for the 4x10 case:

  $ pypy -mperf timeit -s 'a, b, c, d = "1234567890", "abcdefghij", "ABCDEFGHIJ", "0987654321"' 'a + b + c + d'
  .........
  Mean +- std dev: 1.06 ns +- 0.04 ns
  $ pypy -mperf timeit -s 'a, b, c, d = "1234567890", "abcdefghij", "ABCDEFGHIJ", "0987654321"' '"{}{}{}{}".format(a, b, c, d)'
  ........
  Mean +- std dev: 45.8 ns +- 0.9 ns
  $ pypy -mperf timeit -s 'a, b, c, d = "1234567890", "abcdefghij", "ABCDEFGHIJ", "0987654321"' '"".join([a, b, c, d])'
  ........
  Mean +- std dev: 62.0 ns +- 4.8 ns
  $ pypy -mperf timeit -s 'a, b, c, d = "1234567890", "abcdefghij", "ABCDEFGHIJ", "0987654321"' '"%s%s%s%s" % (a, b, c, d)'
  ........
  Mean +- std dev: 78.3 ns +- 1.9 ns