Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to choose the right Python concurrency API

superfastpython.com

Post reply on HN
Print view View original

21–30 of 64 posts

Re: How to choose the right Python concurrency API

#21

Post by O5vYtytb »

I will say that structured concurrency (coroutine based) via Trio/Anyio is, in my experience, so much better for most applications that can support it. Reasoning/testing threading code is so impossible once a project gets large enough. I'd highly recommend reading https://vorpus.org/blog/notes-on-structured-concurrency-or-g...

I work with trio/anyio on a daily basis for my job, and I'd always recommend people use it for their concurrency framework and then use await anyio.to_thread.run_sync() to spawn threads if needed.

I'll also add that if you need multi-processor execution, look at https://pypi.org/project/tractor/

Re: How to choose the right Python concurrency API

#22

Post by zach_garwood »

Every time I read about the difficulties in writing concurrent Python, it reminds me of the joke: Patient: "Doc, it hurts when I do this!" Doctor: "Well, stop doing that."

I love me some Python, but it's just not good at this. When your requirements dictate that you need concurrency, you should start evaluating a different language that is designed for it, in order to build that part of your system.

Re: How to choose the right Python concurrency API

#24

Post by js2 »

matsemann wrote: post #6
uniqueuid wrote: post #4

The article gives a good summary of the quite complex landscape of concurrency in python. There's more to it, for example gil-free c-extensions, subprocesses and cross-machine (plus IPC) communication. But I'm particularly bothered by the fact that many articles and tutorials look at concurrency as if it's only about factoring primes or writing a web server with many (perhaps even idempotent) parallel requests. In re…

It's actually frustratingly complex. Right now I'm working on a "bridge" that receives http requests, and then needs to send that on an existing websocket connection to another system then wait for some responses, send some more on the ws etc and keep monitoring the ws. So the idea is basically to have multiple permanent websocket workers (one for each machine we need to speak with), that get tasks sent to them. Some…

It sounds like your application is I/O bound. That's an ideal use case for threads because the GIL is not held during I/O, so threads allow you to multiplex across lots of connections.

I have a gunicorn server that's part of an internal processing pipeline. It receives an HTTP post from a Java client, grabs some files from S3, caches them to disk, does some processing on the POST and returns a reply. It spends most of its time blocked on I/O. I run it with 1000 threads per process no problem because that's what the Java side's thread-count is set to.

I also have a related server that receivs HTTP POSTs from millions of clients all over the Internet via an AWS ALB. For each POST, it validates the data, splits the POST into three files that are each written to S3 and adds an event to SQS. For this application, I used Falcon and gevent which has me I/O bound. I tested it with threads but that ended up being CPU bound. I also tested with pypy but again, that ended up CPU bound. Gevent got me the best concurrency.

Anyway, unless you've tested you can't just assume the GIL will be a problem. I've been writing Python for two decades using it in a variety of applications and I can count on one hand the number of times the GIL has been an issue.

Re: How to choose the right Python concurrency API

#25

Post by pdimitar »

Or here's a crazy idea, maybe Python is not a holy gospel and you shouldn't use it for things that it's obviously badly suited for. You have a GIL in there, what other subtle hints do you need that your scenario is not well-supported?

Erlang/Elixir and Rust will serve you perfectly for concurrent and parallel programming. OCaml 5.0, once stabilized, will do so as well.

This Python worshipping and trying to use it for everything just betrays amateur programmers underneath.

Whatever happened to "use the right tool for the job"?

Re: How to choose the right Python concurrency API

#26

Post by Yaser_Amiri »

XorNot wrote: post #8

I kind of wish we'd just add Go-style CSP to python and call it a day - it seems to be the best of both worlds when it comes "do two things at the same time".

I actually tried to build a CSP interface on top of AsyncIO, It's still experimental an I haven't developed it for sometime but you know, It works.

https://github.com/Yaser-Amiri/one-ring/blob/main/docs/sampl...

As long as GIL exists with current conditions, we will not have a Go-like CSP, but that's about scheduling, not the CSP itself.

Re: How to choose the right Python concurrency API

#28

Post by wenc »

jonstewart wrote: post #19

If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other newer programming language). Especially, don’t rely on Multiprocessing. I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the…

If you need to stick with Python, Joblib is more reliable and works better than Multiprocessing in my experience.

I ran into so many subtle quirks that needed workarounds in Multiprocessing. Joblib.parallel with the Loky backend just seems to be Multiprocessing done right.

Re: How to choose the right Python concurrency API

#29

Post by infinite8s »

O5vYtytb wrote: post #21

I will say that structured concurrency (coroutine based) via Trio/Anyio is, in my experience, so much better for most applications that can support it. Reasoning/testing threading code is so impossible once a project gets large enough. I'd highly recommend reading https://vorpus.org/blog/notes-on-structured-concurrency-or-g... I work with trio/anyio on a daily basis for my job, and I'd always recommend people use it…

+1 for trio. I used it in a past job to proxy a socket-based API to websockets and it was easy to write and debug.

Tractor is also architected similar to Erlangs OTP system, which trio with it's structured concurrency makes easy to reason about.

Re: How to choose the right Python concurrency API

#30

Post by wpietri »

uniqueuid wrote: post #4

The article gives a good summary of the quite complex landscape of concurrency in python. There's more to it, for example gil-free c-extensions, subprocesses and cross-machine (plus IPC) communication. But I'm particularly bothered by the fact that many articles and tutorials look at concurrency as if it's only about factoring primes or writing a web server with many (perhaps even idempotent) parallel requests. In re…

Yes, I was surprised how good it is! I especially liked how it started out with CPU-bound vs IO-bound. It's such a key distinction, but a lot of people just getting into concurrency will not have thought much about that.

(And agreed on an actor system. It's definitely the approach that best fits my brain.)

Post reply on HN