Big fan of using Cython for things where you need significant performance optimization.
Not a big fan of Cython here... https://hn.algolia.com/?query=dom0%20cython&sort=byPopularit...
Python Practices for Efficient Code: Performance, Memory, and Usability
31–40 of 43 posts
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#32Earlier quoted context omitted.
I still find it terrifying monkey patching all the internal functions with Gevent; to be honest it caused a loads of weird bugs with celery that were impossible to debug. I’m pretty sure given my experience I’d choose to avoid it in future.
for standalone programs, there is simply no better multi-processing library than gevent. Forget the libev performance thing, the api is super pleasant... even more so than asyncio. When you use gevent to run explicit green threads, you do NOT monkey patch - you call the library functions. You only monkey patch if you want stuff happening for free. Which also works pretty well in production with gunicorn.. and yes wit…
Sure, the example doesn’t monkey patch there though. My issues happened around complex multi part Chords and removing Gevent fixed stuff, but then eventually I moved away from celery too and just used Pikka and Rabbitmq directly. This gave me more control and allowed me to see what was happening more easily; celery sometimes seemed to hide exceptions from me. Glad you haven’t had issues.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#33I think they're really good to be aware of, but think it's overreaching to advise "Use slots when defining a Python class." I'm surprised there's no mention of exceptions. Constructing, throwing, catching, and discarding an exception can be relatively slow (especially in a tight loop). My usual advice is "exceptions should be the exceptional case." In general, get familiar with inspection tools so your code is easy t…
> exception[s] can be relatively slow That really depends on your hit rate. Say you have a generator returning objects and you don't know if they have an attribute. You can use hasattr(..) to check or just try to use the attribute and catch the AttributeError. One-on-one the exception is slower, but there comes a point where the exception is rare enough that actively checking every single item is slower. Eg if only 1…
That's what I was trying to get at. Not to avoid exceptions, but to write your code in such a way that exceptions are rare. I usually encounter it with file i/o situations, like like you're saying, an extra file stat in a loop would be really slow and open you to race conditions (file being modified before testing an acting). If an exception was thrown 90% of the time, you want to rethink your logic because performance might be equally poor. Catching the rare exception is ideal.
As an aside (since you mentioned generators), I wish exception handling in comprehensions were easier to deal with.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#34Its 2017. After python 3.6, I hope the debate btw python2 vs 3 is put to an end.
I stopped reading there.
In fact, I'm finding more and more libraries written for Python 3 exclusively, or for which Python 2 support has been discontinued. Stating "you may find a lot of packages that only support Python2" makes me think the author is just spouting off random things he's heard.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#35Earlier quoted context omitted.
hey - thanks for this. what do you personally use ? It is more interesting to learn from someone who has been using it for a while. I'm especially concerned about IDE support, debuggability and testability.
For IDE support, I'm totally in love with PyCharm and been using it for years, first community and then I upgraded to Pro for Cython support. PyCharm Pro has syntax highlighting and does inspections (linting) on Cython code, so it's a wonderful tool, totally worth its price. Can't really recommend any other IDE because I haven't used another one for over 4 years, though a friend uses (loves) Sublime Text and has high…
why did you not choose to use numba ? because the performance in a lot of cases is faster, and most importantly the IDE/debugging integration is seamless.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#36Earlier quoted context omitted.
I still find it terrifying monkey patching all the internal functions with Gevent; to be honest it caused a loads of weird bugs with celery that were impossible to debug. I’m pretty sure given my experience I’d choose to avoid it in future.
for standalone programs, there is simply no better multi-processing library than gevent. Forget the libev performance thing, the api is super pleasant... even more so than asyncio. When you use gevent to run explicit green threads, you do NOT monkey patch - you call the library functions. You only monkey patch if you want stuff happening for free. Which also works pretty well in production with gunicorn.. and yes wit…
Curious if you have tried or evaluated stackless in production. One doesnt hear a lot on stackless python these dsys. BTW this is for my education only, not advocating one over the other.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#37Earlier quoted context omitted.
for standalone programs, there is simply no better multi-processing library than gevent. Forget the libev performance thing, the api is super pleasant... even more so than asyncio. When you use gevent to run explicit green threads, you do NOT monkey patch - you call the library functions. You only monkey patch if you want stuff happening for free. Which also works pretty well in production with gunicorn.. and yes wit…
Curious if you have tried or evaluated stackless in production. One doesnt hear a lot on stackless python these dsys. BTW this is for my education only, not advocating one over the other.
Not really. I'm really going with the community here and the recent community is huge.
Gevent works in production with celery and even newrelic works with pretty OK (not perfect)
When we are talking about production systems, I need the whole ecosystem to work. So I'd rather use vanilla Python with few libraries like gevent.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#38Earlier quoted context omitted.
I still find it terrifying monkey patching all the internal functions with Gevent; to be honest it caused a loads of weird bugs with celery that were impossible to debug. I’m pretty sure given my experience I’d choose to avoid it in future.
I've used gevent monkey patching for years in real codebases with success although I have not used celery. Most problems I've seen with this come from not doing the monkey patching before importing anything else. Also, generally avoiding c-extensions for talking to databases unless they have explicit support for greenlets is advisable. My main webapp currently is Flask on gunicorn with async gevent workers. Mix in Fl…
Do you have a small sample of gunicorn + flask + websockets + sqlalchemu config ? I've been exploring a little here and looking for something that works well.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#39Earlier quoted context omitted.
I've used gevent monkey patching for years in real codebases with success although I have not used celery. Most problems I've seen with this come from not doing the monkey patching before importing anything else. Also, generally avoiding c-extensions for talking to databases unless they have explicit support for greenlets is advisable. My main webapp currently is Flask on gunicorn with async gevent workers. Mix in Fl…
Do you have a small sample of gunicorn + flask + websockets + sqlalchemu config ? I've been exploring a little here and looking for something that works well.
Sorry, I don't have something that is easily sharable.
Re: Python Practices for Efficient Code: Performance, Memory, and Usability
#40Earlier quoted context omitted.
On Python 2.7.10: In [2]: %timeit a+b+c+d The slowest run took 6.66 times longer than the fastest. This could mean that an intermediate result is being cached. 1000000 loops, best of 3: 247 ns per loop In [4]: %timeit "{}{}{}{}".format(a, b, c, d) The slowest run took 6.37 times longer than the fastest. This could mean that an intermediate result is being cached. 1000000 loops, best of 3: 709 ns per loop On Python 3.…
The reason join is fast for these cases is because join basically converts all iterables to a list first and figures out how much it has to join exactly.
It walks the iterables (no need to convert them to lists), but its main trick is actually knowing how much memory it'll need for the end result, so no re-allocations happen, simply a one pass (through multiple iterables) copy.