How to choose the right Python concurrency API • Hacker News Board

Re: How to choose the right Python concurrency API

#11

Post by mrweasel » Wed, Aug 10, 2022, 10:30 AM UTC

matsemann wrote: post #6

uniqueuid wrote: post #4

The article gives a good summary of the quite complex landscape of concurrency in python. There's more to it, for example gil-free c-extensions, subprocesses and cross-machine (plus IPC) communication. But I'm particularly bothered by the fact that many articles and tutorials look at concurrency as if it's only about factoring primes or writing a web server with many (perhaps even idempotent) parallel requests. In re…

It's actually frustratingly complex. Right now I'm working on a "bridge" that receives http requests, and then needs to send that on an existing websocket connection to another system then wait for some responses, send some more on the ws etc and keep monitoring the ws. So the idea is basically to have multiple permanent websocket workers (one for each machine we need to speak with), that get tasks sent to them. Some…

Then why not use Java? I love Python, it’s my language of choice for most things, but as you rightly point out, for some things it just invites complexity.

Re: How to choose the right Python concurrency API

#12

Post by RhysU » Wed, Aug 10, 2022, 10:33 AM UTC

The articles' discussion on multiprocessing approaches links to [1] on Pool vs ProcessPoolExecutor. I wish that rather recommendations for non-standard libraries appeared.

Several years ago, I spent an inordinate amount of time chasing down multiprocessing-usage edge cases. For example: How can I reliably learn that my subprocess died? How can I avoid a fork bomb if my work itself creates new work? Sane serialization semantics? Type hinting? Consistent behavior for spawn, fork, and forkserver? Register callbacks on Futures to run eagerly at completion? Etc.

I ultimately wrote a single-file, drop-it-anywhere “jobserver“ [2] to remove many footcanons. It aims to be the well-tested core inside any more ergonomic API enforcing any additional semantics one might want. No canceling, but that could probably be added.

My "jobserver" is loky-esque [3] as I believe they and I independently were fighting similar challenges but, of course, I prefer mine. Implementation is 560 lines including doc strings. Tests, in the same file, are about another 600 lines.

[1] https://superfastpython.com/multiprocessing-pool-vs-processp...

[2] https://github.com/RhysU/jobserver/blob/master/jobserver.py#...

[3] https://loky.readthedocs.io/en/stable/

Re: How to choose the right Python concurrency API

#13

Post by rmbyrro » Wed, Aug 10, 2022, 11:24 AM UTC

mrweasel wrote: post #11

matsemann wrote: post #6

Earlier quoted context omitted.

It's actually frustratingly complex. Right now I'm working on a "bridge" that receives http requests, and then needs to send that on an existing websocket connection to another system then wait for some responses, send some more on the ws etc and keep monitoring the ws. So the idea is basically to have multiple permanent websocket workers (one for each machine we need to speak with), that get tasks sent to them. Some…

Then why not use Java? I love Python, it’s my language of choice for most things, but as you rightly point out, for some things it just invites complexity.

Same I was thinking.

As it is today, Python just isn't a (relatively) good tool for concurrency.

Use Go, Java. Clojure has some of the most interesting abstractions in this area, but the lisp syntax scares people away.

Re: How to choose the right Python concurrency API

#14

Post by gspetr » Wed, Aug 10, 2022, 11:27 AM UTC

matsemann wrote: post #6

uniqueuid wrote: post #4

The article gives a good summary of the quite complex landscape of concurrency in python. There's more to it, for example gil-free c-extensions, subprocesses and cross-machine (plus IPC) communication. But I'm particularly bothered by the fact that many articles and tutorials look at concurrency as if it's only about factoring primes or writing a web server with many (perhaps even idempotent) parallel requests. In re…

It's actually frustratingly complex. Right now I'm working on a "bridge" that receives http requests, and then needs to send that on an existing websocket connection to another system then wait for some responses, send some more on the ws etc and keep monitoring the ws. So the idea is basically to have multiple permanent websocket workers (one for each machine we need to speak with), that get tasks sent to them. Some…

How do you test all of it?

Re: How to choose the right Python concurrency API

#15

Post by meowface » Wed, Aug 10, 2022, 12:08 PM UTC

XorNot wrote: post #8

I kind of wish we'd just add Go-style CSP to python and call it a day - it seems to be the best of both worlds when it comes "do two things at the same time".

I'm not sure how much it replicates the CSP model, but the closest thing I've found to Go-style concurrency in Python is gevent: https://github.com/gevent/gevent

I personally still prefer to use it in all my projects.

Re: How to choose the right Python concurrency API

#16

Post by uniqueuid » Wed, Aug 10, 2022, 12:09 PM UTC

matsemann wrote: post #6

uniqueuid wrote: post #4

The article gives a good summary of the quite complex landscape of concurrency in python. There's more to it, for example gil-free c-extensions, subprocesses and cross-machine (plus IPC) communication. But I'm particularly bothered by the fact that many articles and tutorials look at concurrency as if it's only about factoring primes or writing a web server with many (perhaps even idempotent) parallel requests. In re…

It's actually frustratingly complex. Right now I'm working on a "bridge" that receives http requests, and then needs to send that on an existing websocket connection to another system then wait for some responses, send some more on the ws etc and keep monitoring the ws. So the idea is basically to have multiple permanent websocket workers (one for each machine we need to speak with), that get tasks sent to them. Some…

Part of the problem - to me at least - is that in practice, concurrency with python is pretty much only heavy (slow) multiprocessing.queues. It's what multiprocessing.pool and the corresponding executors use.

At the same time, my experience (along with that of many others) has been that these queues are not exactly robust - there seem to be many horrible edge cases where queues and/or processes get stuck and it's really hard to have a robust all-python app that recovers from e.g. a dead child (!). Ask ML frameworks, for example, how they handle spawning while holding GPU contexts and the like. It's a nightmare.

So if we had a single robust, elegant and half-way fast abstraction for IPC, that would solve a lot of the existing issues. And it needs to be bi-directional of course.

Re: How to choose the right Python concurrency API

#17

Post by crabbygrabby » Wed, Aug 10, 2022, 12:23 PM UTC

Super good lay person summary. I like how this explains the underlying concepts more than the python specific stuff. Will recommend this to new comers.

Re: How to choose the right Python concurrency API

#18

Post by mountainriver » Wed, Aug 10, 2022, 12:45 PM UTC

uniqueuid wrote: post #4

The article gives a good summary of the quite complex landscape of concurrency in python. There's more to it, for example gil-free c-extensions, subprocesses and cross-machine (plus IPC) communication. But I'm particularly bothered by the fact that many articles and tutorials look at concurrency as if it's only about factoring primes or writing a web server with many (perhaps even idempotent) parallel requests. In re…

There is an effort to bring true parallelism to python, we’ll see how it goes, and yeah it’s desperately needed

Re: How to choose the right Python concurrency API

#19

Post by jonstewart » Wed, Aug 10, 2022, 1:33 PM UTC

If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other newer programming language). Especially, don’t rely on Multiprocessing.

I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the chance, but I’m slowly outsourcing critical pieces to C++ and Rust libraries. Multiprocessing has been a source of subtle portability errors as it has changed over different minor versions.

Re: How to choose the right Python concurrency API

#20

Post by jhokanson » Wed, Aug 10, 2022, 1:35 PM UTC

"Firstly, there are three main Python concurrency APIs, they are:" asyncio, threading, multiprocessing, ... oh, and concurrent.futures

All kidding aside, I used the multiprocessing module lately and it was a mess. Do I want 'map', 'starmap', 'imap', etc.? All I wanted was to run a function multiple times with different inputs (and multiple inputs per function call) and to fail when any launched process failed rather than waiting for every input variation to execute and then telling me about the error(which honestly I didn't think was asking for too much).