Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An easy way to concurrency and parallelism with Python stdlib

bitecode.dev

Post reply on HN
Print view View original

41–50 of 70 posts

Re: An easy way to concurrency and parallelism with Python stdlib

#41

Post by hot_gril »

seabrookmx wrote: post #38
hot_gril wrote: post #36

Earlier quoted context omitted.

Python manages to combine the worst parts of high-level and low-level programming when it comes to multithreading. Like it's using multiple OS-level threads with the associated overhead (not greenthreading like in JS), except it's locking to negate actual multiprocessing, but you still have to use mutexes about as much as in C (no event loop like JS), and the whole API feels low-level and convoluted. It's like they t…

JS doesn't have green threads, just a single threaded event loop and context switching via promises or async/await. Green threads implies parallelism implemented in user space (ala. GoLang goroutines or JVM virtual threads).. JS is not parallel only concurrent.

Greenthreading implies concurrency, not parallelism, implemented in userspace rather than OS. Two Java/whatever greenthreads atop a single OS thread cannot run in parallel. It's switching contexts (as managed in userspace) during I/O waits, just like the JS event loop. You call Goroutines threenthreading, and some Golang users would disagree, but it is too.

Some environments support "M:N" greenthreading, mapping multiple userspace threads to multiple (but fewer) OS threads that are running in parallel, but that's not a required feature of greenthreading. In this case, the OS is still doing the parallelism.

And Python is not greenthreading because the concurrency comes from the OS, since each Py thread maps 1:1 to an OS thread.

Re: An easy way to concurrency and parallelism with Python stdlib

#44

Post by paulddraper »

crabbone wrote: post #40

> For those, Python actually comes with pretty decent tools: the pool executors. Delusion level: max. You have to be in a very, very bad place when this marginal improvement over absolute horror-show that bare Process offers seemed "pretty decent". Python doesn't have good tools for parallelism / concurrency. It doesn't have average tools. It doesn't have even bad tools. It has the worst. Though, unfortunately, it's…

> It doesn't have even bad tools. It has the worst.

> It's not the only language in this category

Soo....not the worst? :) Or tied for it?

What do you find difficult/wrong with pool executors?

Also, you reference "Process", but FYI the article talks about multiple threads, not multiple processes.

Re: An easy way to concurrency and parallelism with Python stdlib

#45

Post by Demiurge »

crabbone wrote: post #40

> For those, Python actually comes with pretty decent tools: the pool executors. Delusion level: max. You have to be in a very, very bad place when this marginal improvement over absolute horror-show that bare Process offers seemed "pretty decent". Python doesn't have good tools for parallelism / concurrency. It doesn't have average tools. It doesn't have even bad tools. It has the worst. Though, unfortunately, it's…

If it's the worst, how is it not the only language in this category?

How do you rank C, Perl, JavaScript, PHP, ... parallelism compared to execution pool + futures here? The absolute MAX WORST?

Re: An easy way to concurrency and parallelism with Python stdlib

#46

Post by paulddraper »

Demiurge wrote: post #45
crabbone wrote: post #40

> For those, Python actually comes with pretty decent tools: the pool executors. Delusion level: max. You have to be in a very, very bad place when this marginal improvement over absolute horror-show that bare Process offers seemed "pretty decent". Python doesn't have good tools for parallelism / concurrency. It doesn't have average tools. It doesn't have even bad tools. It has the worst. Though, unfortunately, it's…

If it's the worst, how is it not the only language in this category? How do you rank C, Perl, JavaScript, PHP, ... parallelism compared to execution pool + futures here? The absolute MAX WORST?

While I'm painting with broad brush, I'll guess that the parent divides languages into two categories "Go" and "the worst".

Re: An easy way to concurrency and parallelism with Python stdlib

#48

Post by WinLychee »

eachro wrote: post #47

So what is the consensus view on how to do parallelism in python if you just have something that is embarassingly parallel with no communication between processes necessary?

if you have a task that is easy to split, make a python script that runs on a subset of the task, split into N subsets, and write one output per process? Once they all complete, join together the outputs. Maybe https://docs.dask.org/en/stable/ is a good start if you want a framework. I don't think there's a consensus, it depends on the problem.

Re: An easy way to concurrency and parallelism with Python stdlib

#49

Post by mixmastamyk »

cle wrote: post #17

I recently have been doing--what should be--straightforward subprocess work in Python, and the experience is infuriatingly bad. There are so many options for launching subprocesses and communicating with them, and each one has different caveats and undocumented limitations, especially around edge cases like processes crashing, timing out, killing them, if they are stuck in native code outside of the VM, etc. For exam…

This off topic rant is the top comment? Really?
Post reply on HN