Live data from Hacker News

How to choose the right Python concurrency API

superfastpython.com

61–64 of 64 posts

Re: How to choose the right Python concurrency API

#61
post #53

Earlier quoted context omitted.

It is a bit dated (he talks about GIL improvements in 3.2), but regardless I think it only proves my point. The examples David shows are threads with zero IO unable to properly cooperate. Having threads that are mostly doing IO would actually allow them to cooperate natively by releasing GIL. Another interesting example he talks about is IO-bound threads competing with CPU-bound threads, where CPU-threads are getting…

Right so the GIL improvements which (probably?) made it into 3.2 solves thrashing but introduces latency during "negotiation" for the GIL. > What David illustrated is that GIL makes it harder to mix CPU- and IO-bound tasks in the same process. But coroutines are not the solution here. Solution is not to mix them. Which I believe is my point and why I forwarded the personal notion that there really is only two accepta…

> threads are strictly inferior as they have additional overhead by nature

In terms of performance, sure. But performance is rarely your only priority. A lot of the times you already have existing code that can be switched to threads with 2 lines of code. Instead, folks choose to rewrite the whole app in asyncio/nodejs because threads aren't sexy. Hey, I've done it.

But today I'd rather have inferior working app in one day, than a completely rewritten app in a few weeks, with a mental model that half engineers don't understand and don't have experience with. Maybe in a few years you will be forced to rewrite it in asyncio. And that would be the right time to do it: when it's a business demand, not when few perfectionists are annoyed.

Re: How to choose the right Python concurrency API

#62
post #61

Earlier quoted context omitted.

Right so the GIL improvements which (probably?) made it into 3.2 solves thrashing but introduces latency during "negotiation" for the GIL. > What David illustrated is that GIL makes it harder to mix CPU- and IO-bound tasks in the same process. But coroutines are not the solution here. Solution is not to mix them. Which I believe is my point and why I forwarded the personal notion that there really is only two accepta…

> threads are strictly inferior as they have additional overhead by nature In terms of performance, sure. But performance is rarely your only priority. A lot of the times you already have existing code that can be switched to threads with 2 lines of code. Instead, folks choose to rewrite the whole app in asyncio/nodejs because threads aren't sexy. Hey, I've done it. But today I'd rather have inferior working app in o…

Very salient point.

Having used both threading and various coroutine libraries (gevent prior to asyncio and asyncio more recently), I find coroutines easier to work with in modern versions of Python. So I don't find them superior strictly on performance; I also consider them more productive due to ease of use.

But I wouldn't disagree with someone choosing to use threads because they find them easier to grok than asyncio.

Re: How to choose the right Python concurrency API

#63
post #38

Earlier quoted context omitted.

This is the correct answer. The love affair to do everything in python must end this year!

But it does almost everything! And it's especially painless if you're not building for scale, and just hobby projects. But yeah, stuff like concurrency and type annotations seem to go against some of Python's fundamental designs (GIL and syntax) so are kinda painful at times.

> But it does almost everything!

Ehhh... does it though? I've been doing shell scripts all my career and never stumbled upon a problem I couldn't solve by combining various tools.

I did a lot of homegrown data science that way.

With Python it seems it's mostly familiarity. People just love the idea of one universal tool, no matter how many times history proves there's no such thing.

Re: How to choose the right Python concurrency API

#64

"Firstly, there are three main Python concurrency APIs, they are:" asyncio, threading, multiprocessing, ... oh, and concurrent.futures All kidding aside, I used the multiprocessing module lately and it was a mess. Do I want 'map', 'starmap', 'imap', etc.? All I wanted was to run a function multiple times with different inputs (and multiple inputs per function call) and to fail when any launched process failed rather…

There's a lot of functionality in the multiprocessing library and it has its own problems, but I wouldn't call it a "mess" from your description. map, starmap, and imap are all useful for different applications depending on your usage and priorities. I'll agree that it can sometimes be difficult to understand the differences and which is best for your use case, but having used in different use cases I certainly appreciate the different functions available.
Post reply on HN