Python Practices for Efficient Code: Performance, Memory, and Usability
1–10 of 43 posts
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#2Big fan of using Cython for things where you need significant performance optimization.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#3I think they're really good to be aware of, but think it's overreaching to advise "Use slots when defining a Python class."
I'm surprised there's no mention of exceptions. Constructing, throwing, catching, and discarding an exception can be relatively slow (especially in a tight loop). My usual advice is "exceptions should be the exceptional case."
In general, get familiar with inspection tools so your code is easy to measure so you can clean up hotspots. Trying to optimize code without inspecting it often makes code harder to read and may not address the slower-performing parts. Maybe everyone needs to spend hours trying to speed things up just to realize the slow parts were in a different part of the code and now it's harder for the next guy to reason what's going on.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#4The Numba JIT function decorator called with (nopython=True, nogil=True) is very useful for small functions with heavy use of NumPy. Often faster than Cython.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#5> Multiprocess, not Multi-thread
or Gevent - which is built on libev and provides constructs like queues, etc to make your multi-processing life much better.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#6Big fan of using Cython for things where you need significant performance optimization.
how do you develop in cython - doesnt the compile, load and test cycle break your flow inside IDE ?
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#7'Use static code analysis tools' is missing a mention of mypy and pytype.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#8> Multiprocess, not Multi-thread or Gevent - which is built on libev and provides constructs like queues, etc to make your multi-processing life much better.
I still find it terrifying monkey patching all the internal functions with Gevent; to be honest it caused a loads of weird bugs with celery that were impossible to debug. I’m pretty sure given my experience I’d choose to avoid it in future.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#9> On the other hand, you may find a lot of packages that only support Python2, and Python3 is not backward-compatible. This means that running your Python2 code on a Python3.x interpreter can possibly throw errors.
I've never found this situation, though I've found the inverse (Py3 but no Py2 support).
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#10> Use format instead of + for generating strings — In Python, str is immutable, so the left and right strings have to be copied into the new string for every pair of concatenations.
It isn't always faster to use string formatting.
$ python -m timeit -s 'a, b, c, d = "1234567890", "abcdefghij", "ABCDEFGHIJ", "0987654321"' 'a + b + c + d'
10000000 loops, best of 3: 0.181 usec per loop
$ python -m timeit -s 'a, b, c, d = "1234567890", "abcdefghij", "ABCDEFGHIJ", "0987654321"' '"{}{}{}{}".format(a, b, c, d)'
1000000 loops, best of 3: 0.447 usec per loop
$ python --version
Python 2.7.13