Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Practices for Efficient Code: Performance, Memory, and Usability

codementor.io

Post reply on HN
Print view View original

21–30 of 43 posts

Re: Python Practices for Efficient Code: Performance, Memory, and Usability

#21

Post by sandGorgon »

dr_zoidberg wrote: post #12
sandGorgon wrote: post #6

Earlier quoted context omitted.

how do you develop in cython - doesnt the compile, load and test cycle break your flow inside IDE ?

You can either compile-before-run (as you would any compiled language), our you can leverage Cythons compile-on-import mechanism. Which does exactly the same, but you don't have to think about having to compile. Compile time is usually down to a few seconds (or less), so it's not really a burden.

hey - thanks for this. what do you personally use ? It is more interesting to learn from someone who has been using it for a while.

I'm especially concerned about IDE support, debuggability and testability.

Re: Python Practices for Efficient Code: Performance, Memory, and Usability

#22

Post by dr_zoidberg »

cjbillington wrote: post #18
dr_zoidberg wrote: post #11

Earlier quoted context omitted.

One of the best tools for profiling I've used with python is gprof2dot, which analizes cProfiles output and generates a function call-graph that gives deep insights as to what is really taking execution time. It can help you see a lot clearer why things are slow. Though it has its nuissances, it significantly improves your understanding of where time is spent. It's the same data from cProfile, but much better organiz…

I made a wrapper of cProfile and gprof2dot, which can be found here: http://bprofile.readthedocs.io so that you don't have to remember how to call gprof2dot all the time.

Nice, it looks handy!

Re: Python Practices for Efficient Code: Performance, Memory, and Usability

#23

Post by mattbillenstein »

andy_ppp wrote: post #8
sandGorgon wrote: post #5

> Multiprocess, not Multi-thread or Gevent - which is built on libev and provides constructs like queues, etc to make your multi-processing life much better.

I still find it terrifying monkey patching all the internal functions with Gevent; to be honest it caused a loads of weird bugs with celery that were impossible to debug. I’m pretty sure given my experience I’d choose to avoid it in future.

I've used gevent monkey patching for years in real codebases with success although I have not used celery. Most problems I've seen with this come from not doing the monkey patching before importing anything else.

Also, generally avoiding c-extensions for talking to databases unless they have explicit support for greenlets is advisable.

My main webapp currently is Flask on gunicorn with async gevent workers. Mix in Flask-Sockets (using gevent-websocket) and I have websocket support as well and use redis pubsub as a broker to communicate intraprocess. It's a nice and capable system and you can reach into gevent if you want to do scatter-gather sorts of patterns without dealing with threads.

Re: Python Practices for Efficient Code: Performance, Memory, and Usability

#24

Post by dr_zoidberg »

sandGorgon wrote: post #21
dr_zoidberg wrote: post #12

Earlier quoted context omitted.

You can either compile-before-run (as you would any compiled language), our you can leverage Cythons compile-on-import mechanism. Which does exactly the same, but you don't have to think about having to compile. Compile time is usually down to a few seconds (or less), so it's not really a burden.

hey - thanks for this. what do you personally use ? It is more interesting to learn from someone who has been using it for a while. I'm especially concerned about IDE support, debuggability and testability.

For IDE support, I'm totally in love with PyCharm and been using it for years, first community and then I upgraded to Pro for Cython support. PyCharm Pro has syntax highlighting and does inspections (linting) on Cython code, so it's a wonderful tool, totally worth its price.

Can't really recommend any other IDE because I haven't used another one for over 4 years, though a friend uses (loves) Sublime Text and has high praise for it too. It has Cython syntax support, but there's really no linter for Cython -- what's more, you may have to turn off warnings if you're using a Python linter because it'll complain about cdefs and such.

As for the day to day use, I don't do that much Cython really, ocassionally to speed things up. I start from a Python function, then make a .pyx (or bunch of .pyxes) files and replicate what's needed. Compile, load both versions (Py and Cy) and compare outputs and speed from IPython or with benchmark scripts.

I did use a lot the Cython functionality to create the HTML view of the source source at the beggnining, though you get the hang of what transpiles well with time and use it less and less. Still, sometimes it's useful.

Re: Python Practices for Efficient Code: Performance, Memory, and Usability

#25

Post by TFortunato »

oliwarner wrote: post #19
pfranz wrote: post #3

I think they're really good to be aware of, but think it's overreaching to advise "Use slots when defining a Python class." I'm surprised there's no mention of exceptions. Constructing, throwing, catching, and discarding an exception can be relatively slow (especially in a tight loop). My usual advice is "exceptions should be the exceptional case." In general, get familiar with inspection tools so your code is easy t…

> exception[s] can be relatively slow That really depends on your hit rate. Say you have a generator returning objects and you don't know if they have an attribute. You can use hasattr(..) to check or just try to use the attribute and catch the AttributeError. One-on-one the exception is slower, but there comes a point where the exception is rare enough that actively checking every single item is slower. Eg if only 1…

Out of curiosity, is there any overhead to the python exception mechanism if you don't hit an exception (ie just by wrapping things in a try block)

I come from an embedded background, where in some c++ projects, we would disable exceptions for various reasons, including the memory overhead, which is why I ask..

Re: Python Practices for Efficient Code: Performance, Memory, and Usability

#27

Post by yen223 »

pfranz wrote: post #3

I think they're really good to be aware of, but think it's overreaching to advise "Use slots when defining a Python class." I'm surprised there's no mention of exceptions. Constructing, throwing, catching, and discarding an exception can be relatively slow (especially in a tight loop). My usual advice is "exceptions should be the exceptional case." In general, get familiar with inspection tools so your code is easy t…

Slots have benefits outside of the memory savings. By preventing arbitrary attribute assignment on an object, slots can provide nice guarantees about the shape of an object, making it easier to reason about.

Re: Python Practices for Efficient Code: Performance, Memory, and Usability

#28

Post by dom0 »

dr_zoidberg wrote: post #14
davidwtbuxton wrote: post #10

> Use format instead of + for generating strings — In Python, str is immutable, so the left and right strings have to be copied into the new string for every pair of concatenations. It isn't always faster to use string formatting. $ python -m timeit -s 'a, b, c, d = "1234567890", "abcdefghij", "ABCDEFGHIJ", "0987654321"' 'a + b + c + d' 10000000 loops, best of 3: 0.181 usec per loop $ python -m timeit -s 'a, b, c, d…

On Python 2.7.10: In [2]: %timeit a+b+c+d The slowest run took 6.66 times longer than the fastest. This could mean that an intermediate result is being cached. 1000000 loops, best of 3: 247 ns per loop In [4]: %timeit "{}{}{}{}".format(a, b, c, d) The slowest run took 6.37 times longer than the fastest. This could mean that an intermediate result is being cached. 1000000 loops, best of 3: 709 ns per loop On Python 3.…

The reason join is fast for these cases is because join basically converts all iterables to a list first and figures out how much it has to join exactly.

Re: Python Practices for Efficient Code: Performance, Memory, and Usability

#29

Post by pfranz »

swaggyBoatswain wrote: post #26
ram_rar wrote: post #15

Its 2017. After python 3.6, I hope the debate btw python2 vs 3 is put to an end.

I can't believe this is still a discussion, python 3 already practically every python 2 library http://py3readiness.org/

In general, Python 3 should have been what you start with for the past few years. In visual effects most of our apps have an embedded Python runtime. Those apps have a lot of internal code built around Python2 in addition to clients that have a decades worth of Python2 infrastructure. Everyone needs to move at the same time (and it's been hard even with minor versions of Python).

As an industry, we're only now talking about migrating to Python 3 in 2019: http://www.vfxplatform.com/ Looking around, most people I worked with just do what they're told and have probably never seen Python3 yet. The past few years the industry has pushed off Python3 because of other large changes requested for third-party apps: new versions of C++, gcc, boost, Qt.

It's actually been kind of tough to use Python 2 for the past couple years. I noticed the PyCon talks are a lot less relevant because they're all Python 3 oriented. Python bindings for Qt are a lot more difficult because of the old compiler used for Python 2 on Windows.

I wonder how many other industries are similarly stuck?

Re: Python Practices for Efficient Code: Performance, Memory, and Usability

#30

Post by dom0 »

TFortunato wrote: post #25
oliwarner wrote: post #19

Earlier quoted context omitted.

> exception[s] can be relatively slow That really depends on your hit rate. Say you have a generator returning objects and you don't know if they have an attribute. You can use hasattr(..) to check or just try to use the attribute and catch the AttributeError. One-on-one the exception is slower, but there comes a point where the exception is rare enough that actively checking every single item is slower. Eg if only 1…

Out of curiosity, is there any overhead to the python exception mechanism if you don't hit an exception (ie just by wrapping things in a try block) I come from an embedded background, where in some c++ projects, we would disable exceptions for various reasons, including the memory overhead, which is why I ask..

Exception handlers in CPython work by having an instruction store the offset to the start of the handler code which is stored on a stack attached to the frame (iirc). So the cost of a "try:" block is pretty low (for Python, anyway).

The handlers themselves are pretty basic, an "except SomeError" pretty much translates to "if isinstance(exc, SomeError):".

Post reply on HN