Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An easy way to concurrency and parallelism with Python stdlib

bitecode.dev

Post reply on HN
Print view View original

21–30 of 70 posts

Re: An easy way to concurrency and parallelism with Python stdlib

#21

Post by p5a0u9l »

cle wrote: post #17

I recently have been doing--what should be--straightforward subprocess work in Python, and the experience is infuriatingly bad. There are so many options for launching subprocesses and communicating with them, and each one has different caveats and undocumented limitations, especially around edge cases like processes crashing, timing out, killing them, if they are stuck in native code outside of the VM, etc. For exam…

The mess more reflects supporting a programmatic interface to processes in a cross platform manner, coupled with the actual complexity of parallel processing.

You didn’t mention the recommended high level option for subprocess, ‘subprocess. run’.

Re: An easy way to concurrency and parallelism with Python stdlib

#22

Post by gpderetta »

smallerfish wrote: post #20

Maybe I missed it, but how do the threads circumvent the GIL? > When a request is waiting on the network, another thread is executing. I'm guessing this is the meat, but what controls that? What other operations allow the GIL to switch to another thread?

My understanding is that the GIL is typically released around blocking operations. Aside for allowing actual concurrency for I/O heavy programs, it would be a trivial way to deadlock if it wasn't.

Re: An easy way to concurrency and parallelism with Python stdlib

#23

Post by ameliaquining »

smallerfish wrote: post #20

Maybe I missed it, but how do the threads circumvent the GIL? > When a request is waiting on the network, another thread is executing. I'm guessing this is the meat, but what controls that? What other operations allow the GIL to switch to another thread?

Python functions implemented in C can release the GIL when they're doing something that doesn't directly involve manipulating Python objects, and then re-acquire it when they're done: https://docs.python.org/3/c-api/init.html#thread-state-and-t...

All I/O functions in the standard library do this when blocked.

Re: An easy way to concurrency and parallelism with Python stdlib

#24

Post by oefrha »

pdimitar wrote: post #4
simonw wrote: post #3

Earlier quoted context omitted.

If you already know Python, the advice in this article is certainly a lot easier and more actionable than "just learn Go or Rust or Zig instead".

Certainly. My point is that if you need to write that much code and/or do that much research, at one point the effort of doing it in another language will be less than to keep insisting on using a tool that's not designed for it. It happened with me and many other former colleagues. Though obviously, everyone decides for themselves when does that point come -- or if it comes at all.

If this is too much research for you, wait until you have to deal with the many problems of Go channels in the real world. (Reasonably well-known though controversial article: [1]) Don't even get me started on Rust. Concurrency and parallelism is hard.

Yes, I've written a shit ton of code in all aforementioned languages.

[1] https://www.jtolio.com/2016/03/go-channels-are-bad-and-you-s...

Re: An easy way to concurrency and parallelism with Python stdlib

#28

Post by cle »

p5a0u9l wrote: post #21
cle wrote: post #17

I recently have been doing--what should be--straightforward subprocess work in Python, and the experience is infuriatingly bad. There are so many options for launching subprocesses and communicating with them, and each one has different caveats and undocumented limitations, especially around edge cases like processes crashing, timing out, killing them, if they are stuck in native code outside of the VM, etc. For exam…

The mess more reflects supporting a programmatic interface to processes in a cross platform manner, coupled with the actual complexity of parallel processing. You didn’t mention the recommended high level option for subprocess, ‘subprocess. run’.

Sure that exists too, but it blocks on process exit. I suppose I can run that in a separate thread but now I've got another dimension of complexity to deal with, and it's unclear if I can stream output from the subprocess?

There are other things I didn't mention that get thrown around too such as os.system() and os.fork().

Re: An easy way to concurrency and parallelism with Python stdlib

#29

Post by p5a0u9l »

cle wrote: post #28
p5a0u9l wrote: post #21

Earlier quoted context omitted.

The mess more reflects supporting a programmatic interface to processes in a cross platform manner, coupled with the actual complexity of parallel processing. You didn’t mention the recommended high level option for subprocess, ‘subprocess. run’.

Sure that exists too, but it blocks on process exit. I suppose I can run that in a separate thread but now I've got another dimension of complexity to deal with, and it's unclear if I can stream output from the subprocess? There are other things I didn't mention that get thrown around too such as os.system() and os.fork().

For my use cases the asyncio wrapper makes it really easy to stack up a bunch of tasks, let the OS it’s thing, and then collect the results when they’re ready.

Re: An easy way to concurrency and parallelism with Python stdlib

#30

Post by nerdponx »

cle wrote: post #17

I recently have been doing--what should be--straightforward subprocess work in Python, and the experience is infuriatingly bad. There are so many options for launching subprocesses and communicating with them, and each one has different caveats and undocumented limitations, especially around edge cases like processes crashing, timing out, killing them, if they are stuck in native code outside of the VM, etc. For exam…

To be clear, Popen is very different from all the other options. That's for running other programs.

Process is low-level and is almost never what you want. Pool is "mid-level", and usually isn't what you want. ProcessPoolExecutor is usually what you want, it is the "one obvious way to do it". That's not at all clear from the docs though.

The one obvious way to do it, in general, is: subprocess.run for running external processes, subprocess.Popen for async interaction with external processes, and concurrent.futures.ProcessPoolExecutor for Python multiprocessing.

Your other complaints about actually using the multiprocessing stuff are 100% valid. Error handling, cancellation, etc. is all very difficult. Passing data back and forth between the main process and subprocesses is not trivial.

But I do want to emphasize that there is a somewhat-well-defined gradient of lower- and higher-level tools in the standard library, and your "obvious way to do it" should usually start at the higher end of that gradient.

You might also want to look into the third-party Joblib library, which makes process parallelism a lot less painful for the straightforward use case of "run a function on a large amount of data, using multiple OS processes."

Post reply on HN