Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to choose the right Python concurrency API

superfastpython.com

Post reply on HN
Print view View original

41–50 of 64 posts

Re: How to choose the right Python concurrency API

#41

Post by capableweb »

jonstewart wrote: post #19

If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other newer programming language). Especially, don’t rely on Multiprocessing. I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the…

Hear hear!

If you already know Python, picking up the basics of Java/Rust/Go/Elixir is trivial. Picking up the basics of C++ would be slightly harder, but not impossible. Erlang even further away, at least syntax wise.

But, you'll be doing future you a huge service for whenever you need to reach for hard performance/latency and/or concurrency/parallelism again.

Re: How to choose the right Python concurrency API

#43

Post by gshulegaard »

I am not sure when this article was written, but I really wouldn't recommend threads for even IO-bound tasks. Since threads don't escape the GIL the added contention with OS thread scheduling (context switching) is a performance overhead that coroutines (asyncio) don't have. I haven't benchmarked threads vs. coroutines for IO-bound tasks...but my gut feeling is coroutines are going to generally be better because of the lack of thread switching overhead.

So for me, there really are only 2 concurrency choices: coroutines or multiprocessing. And generally, if I find myself reaching for multiprocessing I seriously evaluate if the logic shouldn't be ported to a different language.

Re: How to choose the right Python concurrency API

#44

Post by AceJohnny2 »

jonstewart wrote: post #19

If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other newer programming language). Especially, don’t rely on Multiprocessing. I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the…

I think I need Python concurrency because I'm using it to talk to a bunch of hardware in parallel (using pexpect, for testing), collecting global statistics. I'm not using concurrency for performance (well, not in the usual way I guess). Using a different language would put me at odds with the rest of the team, increasing the maintenance burden.

What solution would people recommend?

Re: How to choose the right Python concurrency API

#45

Post by vajenetehais »

jonstewart wrote: post #19

If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other newer programming language). Especially, don’t rely on Multiprocessing. I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the…

Genuinely asking, why don't you add dotnet ?

I'm currently in the situation you describe and I would like to try something new. Dotnet is among my candidates. (only asking for personal projects)

Re: How to choose the right Python concurrency API

#46

Post by lelandbatey »

gshulegaard wrote: post #43

I am not sure when this article was written, but I really wouldn't recommend threads for even IO-bound tasks. Since threads don't escape the GIL the added contention with OS thread scheduling (context switching) is a performance overhead that coroutines (asyncio) don't have. I haven't benchmarked threads vs. coroutines for IO-bound tasks...but my gut feeling is coroutines are going to generally be better because of t…

I wouldn't quite say that there's only 2 concurrency choices, but I agree that there's only 2 concurrency choices in Python if you are operating under heavy load.

The multi-threaded concurrency style can be a great choice if your performance requirements are low and you only want a small amount of concurrency. Then it can be very easy to emulate a Golang-esq model, by having just a few threads which are communicating by writing data via queues to one another.

Re: How to choose the right Python concurrency API

#47

Post by [deleted] »

jonstewart wrote: post #19

If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other newer programming language). Especially, don’t rely on Multiprocessing. I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the…

[deleted]

Re: How to choose the right Python concurrency API

#48

Post by henryfjordan »

AceJohnny2 wrote: post #44
jonstewart wrote: post #19

If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other newer programming language). Especially, don’t rely on Multiprocessing. I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the…

I think I need Python concurrency because I'm using it to talk to a bunch of hardware in parallel (using pexpect, for testing), collecting global statistics. I'm not using concurrency for performance (well, not in the usual way I guess). Using a different language would put me at odds with the rest of the team, increasing the maintenance burden. What solution would people recommend?

Write up the same simple example in Python and in a different language that is better suited to the task. If the other language is that much easier, it should be easy to convince people with the example.

Often times you think "since my team is using Python that'll be easiest" but if you have to rely on libraries (even parts of the standard lib) that your teammates don't often use then you might not be getting as much benefit as you'd think. A practical example might show everyone that switching languages to get access to a better interface for your problem might be easier for everyone. It might also show you that Python is actually the right choice, so it's a good exercise.

Re: How to choose the right Python concurrency API

#49

Post by vector57 »

I will briefly plug my library `unsync` (https://github.com/alex-sherman/unsync#quick-overview) which wraps all these methods (multiprocessing/threading/asyncio) into singular/simple-ish API.

It's a bit overly simple, but it's helped a few times writing code the makes use of several concurrency methods and combining them together etc.

Re: How to choose the right Python concurrency API

#50

Post by hbrn »

gshulegaard wrote: post #43

I am not sure when this article was written, but I really wouldn't recommend threads for even IO-bound tasks. Since threads don't escape the GIL the added contention with OS thread scheduling (context switching) is a performance overhead that coroutines (asyncio) don't have. I haven't benchmarked threads vs. coroutines for IO-bound tasks...but my gut feeling is coroutines are going to generally be better because of t…

But threads do release GIL on IO. There's a slight overhead of course, but if you're not running hundreds of servers and thousands of threads, the overhead can be negligible.

I've seen more than once people rewriting thread-based code to coroutines, despite not having any noticeable benefits other than being more sexy. In fact I was one of those people almost a decade ago, rewriting the whole networking portion of the app only to achieve ~5% performance improvement. One should always measure the overhead instead assuming how big it is.

Post reply on HN