Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to choose the right Python concurrency API

superfastpython.com

Post reply on HN
Print view View original

31–40 of 64 posts

Re: How to choose the right Python concurrency API

#31

Post by XorNot »

Yaser_Amiri wrote: post #26
XorNot wrote: post #8

I kind of wish we'd just add Go-style CSP to python and call it a day - it seems to be the best of both worlds when it comes "do two things at the same time".

I actually tried to build a CSP interface on top of AsyncIO, It's still experimental an I haven't developed it for sometime but you know, It works. https://github.com/Yaser-Amiri/one-ring/blob/main/docs/sampl... As long as GIL exists with current conditions, we will not have a Go-like CSP, but that's about scheduling, not the CSP itself.

Well that's the thought: GIL-less Python is coming, at which point we can actually have something like this and it'll just work.

Re: How to choose the right Python concurrency API

#32

Post by malkia »

Any resolution to this (python's multiprocessing on Windows and WaitForMultipleObjects failing when the number of objects (threads) to wait > 60)

https://stackoverflow.com/questions/65252807/multiprocessing...

Just curious, as yesterday had to deal with the `black` formatter, and it had this hack into it to prevent no more than 60 being created.

Re: How to choose the right Python concurrency API

#33

Post by girfan »

This is a great analysis; thanks for writing.

I have also been working on running multiple Python interpreters in the same process by isolating them in different namespaces using `dlmopen` [1]. The objective on a high level is to receive requests for some compute intensive operations from a TCP/HTTP server and dispatch them on to different workers. In this case, a thin C++ shim receives the requests and dispatches them on to one of the Python interpreters in a namespace. This eliminates contention for the GIL amongst the interpreters and can exploit parallelism by running each interpreter on a different set of cores. The data obtained from the request does not need to be copied into the interpreter because everything is in the same address space; similarly the output produced by the Python interpreter is also just passed back without any copies to the server.

[1] https://www.man7.org/linux/man-pages/man3/dlmopen.3.html

Re: How to choose the right Python concurrency API

#35

Post by liveoneggs »

matsemann wrote: post #6
uniqueuid wrote: post #4

The article gives a good summary of the quite complex landscape of concurrency in python. There's more to it, for example gil-free c-extensions, subprocesses and cross-machine (plus IPC) communication. But I'm particularly bothered by the fact that many articles and tutorials look at concurrency as if it's only about factoring primes or writing a web server with many (perhaps even idempotent) parallel requests. In re…

It's actually frustratingly complex. Right now I'm working on a "bridge" that receives http requests, and then needs to send that on an existing websocket connection to another system then wait for some responses, send some more on the ws etc and keep monitoring the ws. So the idea is basically to have multiple permanent websocket workers (one for each machine we need to speak with), that get tasks sent to them. Some…

> It's actually frustratingly complex.

Kind of like every other part of the python ecosystem

Re: How to choose the right Python concurrency API

#36

Post by jeremycarter »

pdimitar wrote: post #25

Or here's a crazy idea, maybe Python is not a holy gospel and you shouldn't use it for things that it's obviously badly suited for. You have a GIL in there, what other subtle hints do you need that your scenario is not well-supported? Erlang/Elixir and Rust will serve you perfectly for concurrent and parallel programming. OCaml 5.0, once stabilized, will do so as well. This Python worshipping and trying to use it for…

This is the correct answer. The love affair to do everything in python must end this year!

Re: How to choose the right Python concurrency API

#37

Post by jeremycarter »

jonstewart wrote: post #19

If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other newer programming language). Especially, don’t rely on Multiprocessing. I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the…

I have a similar project and now have to split up the single application to actually get the muli core processing I demand. I regret not using Rust/Go... or even just .NET.

Re: How to choose the right Python concurrency API

#38

Post by Frotag »

jeremycarter wrote: post #36
pdimitar wrote: post #25

Or here's a crazy idea, maybe Python is not a holy gospel and you shouldn't use it for things that it's obviously badly suited for. You have a GIL in there, what other subtle hints do you need that your scenario is not well-supported? Erlang/Elixir and Rust will serve you perfectly for concurrent and parallel programming. OCaml 5.0, once stabilized, will do so as well. This Python worshipping and trying to use it for…

This is the correct answer. The love affair to do everything in python must end this year!

But it does almost everything! And it's especially painless if you're not building for scale, and just hobby projects.

But yeah, stuff like concurrency and type annotations seem to go against some of Python's fundamental designs (GIL and syntax) so are kinda painful at times.

Re: How to choose the right Python concurrency API

#40

Post by ansre »

jonstewart wrote: post #19

If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other newer programming language). Especially, don’t rely on Multiprocessing. I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the…

[dead]
Post reply on HN