Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An easy way to concurrency and parallelism with Python stdlib

bitecode.dev

Post reply on HN
Print view View original

51–60 of 70 posts

Re: An easy way to concurrency and parallelism with Python stdlib

#51

Post by hot_gril »

eachro wrote: post #47

So what is the consensus view on how to do parallelism in python if you just have something that is embarassingly parallel with no communication between processes necessary?

People here mention Pool, and I've seen it many times. It's this: https://docs.python.org/3/library/multiprocessing.html#intro...

  from multiprocessing import Pool

  def f(x):
      return x*x

  if __name__ == '__main__':
      with Pool(5) as p:
          print(p.map(f, [1, 2, 3]))
This forks out up to 5 processes. f(x) runs fully in parallel for each input. The inputs and outputs sent between processes via pickling.

Re: An easy way to concurrency and parallelism with Python stdlib

#52

Post by hot_gril »

p5a0u9l wrote: post #21
cle wrote: post #17

I recently have been doing--what should be--straightforward subprocess work in Python, and the experience is infuriatingly bad. There are so many options for launching subprocesses and communicating with them, and each one has different caveats and undocumented limitations, especially around edge cases like processes crashing, timing out, killing them, if they are stuck in native code outside of the VM, etc. For exam…

The mess more reflects supporting a programmatic interface to processes in a cross platform manner, coupled with the actual complexity of parallel processing. You didn’t mention the recommended high level option for subprocess, ‘subprocess. run’.

Other high-level languages do a better job with this.

Re: An easy way to concurrency and parallelism with Python stdlib

#53

Post by hot_gril »

mixmastamyk wrote: post #49
cle wrote: post #17

I recently have been doing--what should be--straightforward subprocess work in Python, and the experience is infuriatingly bad. There are so many options for launching subprocesses and communicating with them, and each one has different caveats and undocumented limitations, especially around edge cases like processes crashing, timing out, killing them, if they are stuck in native code outside of the VM, etc. For exam…

This off topic rant is the top comment? Really?

Title: "An easy way to concurrency and parallelism with Python"

Content: basically how to use ThreadPoolExecutor

Comment: Concurrency and parallelism aren't easy in Python.

How is this off-topic?

Re: An easy way to concurrency and parallelism with Python stdlib

#55

Post by hot_gril »

paulddraper wrote: post #44
crabbone wrote: post #40

> For those, Python actually comes with pretty decent tools: the pool executors. Delusion level: max. You have to be in a very, very bad place when this marginal improvement over absolute horror-show that bare Process offers seemed "pretty decent". Python doesn't have good tools for parallelism / concurrency. It doesn't have average tools. It doesn't have even bad tools. It has the worst. Though, unfortunately, it's…

> It doesn't have even bad tools. It has the worst. > It's not the only language in this category Soo....not the worst? :) Or tied for it? What do you find difficult/wrong with pool executors? Also, you reference "Process", but FYI the article talks about multiple threads, not multiple processes.

Pool executors only solve one kind of use case. They aren't a general solution to concurrency+parallelism.

And they're still the worst version of this pattern, because despite using multiple OS-level threads with all the associated overhead, the GIL prevents most of the real parallelism from happening. And if you want full parallelism, you have to use multiprocessing.Pool, which adds pickling overhead and incompatibility.

Re: An easy way to concurrency and parallelism with Python stdlib

#56

Post by mixmastamyk »

hot_gril wrote: post #53
mixmastamyk wrote: post #49

Earlier quoted context omitted.

This off topic rant is the top comment? Really?

Title: "An easy way to concurrency and parallelism with Python" Content: basically how to use ThreadPoolExecutor Comment: Concurrency and parallelism aren't easy in Python. How is this off-topic?

It's mostly about communicating with subprocess and Popen, which has little to do with this article, other than being Python modules you can use with concurrent futures. It's also long-winded and beside the point. Shouldn't be the top comment.

Re: An easy way to concurrency and parallelism with Python stdlib

#57

Post by hot_gril »

Demiurge wrote: post #45
crabbone wrote: post #40

> For those, Python actually comes with pretty decent tools: the pool executors. Delusion level: max. You have to be in a very, very bad place when this marginal improvement over absolute horror-show that bare Process offers seemed "pretty decent". Python doesn't have good tools for parallelism / concurrency. It doesn't have average tools. It doesn't have even bad tools. It has the worst. Though, unfortunately, it's…

If it's the worst, how is it not the only language in this category? How do you rank C, Perl, JavaScript, PHP, ... parallelism compared to execution pool + futures here? The absolute MAX WORST?

Well execution pool doesn't even do parallelism really, just concurrency for the most part (thanks GIL). And JavaScript handles concurrency far better than Python; its event loop is designed for just that. JS and Py can also use subprocesses for true parallelism.

C and Java threads are better than Python because, uh, they can actually run in parallel. Rust adds convenience and safety on top, plus its own event loops. Golang has Goroutines. Erlang has some very powerful solution that I don't remember.

IDK about PHP and Perl, barely touched them. Maybe they're worse than Python for this. Everything else isn't. Python was not originally built with these use cases in mind, which is totally fine, but I'm not going to pick Python if I'm doing complex concurrency/parallelism. For simple process pools, Python is good enough.

Re: An easy way to concurrency and parallelism with Python stdlib

#58

Post by cpach »

potta_coffee wrote: post #39

If I need concurrency these days, I just write it in Golang. My primary use for Python was one off scripts for cloud management / automation tasks. Today I write maybe 70% Golang and 30% Python.

I agree. The team behind Go has thought a lot about concurrency right from the start, and it really shows.

Re: An easy way to concurrency and parallelism with Python stdlib

#59

Post by paulddraper »

seabrookmx wrote: post #38
hot_gril wrote: post #36

Earlier quoted context omitted.

Python manages to combine the worst parts of high-level and low-level programming when it comes to multithreading. Like it's using multiple OS-level threads with the associated overhead (not greenthreading like in JS), except it's locking to negate actual multiprocessing, but you still have to use mutexes about as much as in C (no event loop like JS), and the whole API feels low-level and convoluted. It's like they t…

JS doesn't have green threads, just a single threaded event loop and context switching via promises or async/await. Green threads implies parallelism implemented in user space (ala. GoLang goroutines or JVM virtual threads).. JS is not parallel only concurrent.

JS has green threads.

Green threads imply only concurrency, not parallelism.

(JS also has parallelism, via worker threads, FYI)

Re: An easy way to concurrency and parallelism with Python stdlib

#60

Post by potta_coffee »

cpach wrote: post #58
potta_coffee wrote: post #39

If I need concurrency these days, I just write it in Golang. My primary use for Python was one off scripts for cloud management / automation tasks. Today I write maybe 70% Golang and 30% Python.

I agree. The team behind Go has thought a lot about concurrency right from the start, and it really shows.

Concurrency in Go is just so easy and powerful.
Post reply on HN