Linear Exeuction, Multiprocessing, and Multithreading IO-Bound Tasks in Python
1–9 of 9 posts
Re: Linear Exeuction, Multiprocessing, and Multithreading IO-Bound Tasks in Python
#2>First, terms. Most programs work from top to bottom. The next line runs after the last one finishes. We call these linear
Sequential is the term you're looking for.
Re: Linear Exeuction, Multiprocessing, and Multithreading IO-Bound Tasks in Python
#3I'm 99% sure threads will run on all cores. I think the author has confused multithreading with hyperthreading.
Re: Linear Exeuction, Multiprocessing, and Multithreading IO-Bound Tasks in Python
#4Whenever I see a blog post on Python concurrency that recommends something other than concurrent.futures ThreadPoolExecutor or ProcessPoolExecutor, I shake my head in disappointment at how far we've strayed from "one obviously right way". Why in 2020 am I seeing someone manually close and join a worker pool (blocking!) to return all results all at once because they rewrote executor.map using multiprocessing? Why do they del pool immediately before returning instead of just letting it run out of scope? Why are they using map, which orders the result, if they just want to call all? If performance matters, you want to read the results in the order of completion not the order of submission, and you don't want to wait for all of them to complete if any of them fail. Where did Python's "right way" go sideways?
Re: Linear Exeuction, Multiprocessing, and Multithreading IO-Bound Tasks in Python
#5I'm 99% sure threads will run on all cores. I think the author has confused multithreading with hyperthreading.
I was never sure about that, thinking that a single interpreter process runs on one core and the threads must share that process space. But you're right, and I found good confirmation in this visualization (see the "code" link also): http://www.dabeaz.com/GIL/gilvis/fourthread.html
See also http://www.dabeaz.com/python/UnderstandingGIL.pdf, which explains that
• Python threads are real system threads
• POSIX threads (pthreads)
• Windows threads
• Fully managed by the host operating system
• Represent threaded execution of the Python interpreter process (written in C)Re: Linear Exeuction, Multiprocessing, and Multithreading IO-Bound Tasks in Python
#6I'm 99% sure threads will run on all cores. I think the author has confused multithreading with hyperthreading.
The article is full of inaccuracies and errors. Confusing threads with eventloops is just one of them.
I can't imagine it's being upvoted.
Re: Linear Exeuction, Multiprocessing, and Multithreading IO-Bound Tasks in Python
#7>First, terms. Most programs work from top to bottom. The next line runs after the last one finishes. We call these linear Sequential is the term you're looking for.
Linear felt right when I was writing it, for some reason, but you're right. Fixed.
Re: Linear Exeuction, Multiprocessing, and Multithreading IO-Bound Tasks in Python
#8Whenever I see a blog post on Python concurrency that recommends something other than concurrent.futures ThreadPoolExecutor or ProcessPoolExecutor, I shake my head in disappointment at how far we've strayed from "one obviously right way". Why in 2020 am I seeing someone manually close and join a worker pool (blocking!) to return all results all at once because they rewrote executor.map using multiprocessing? Why do t…
If I google for python concurrency the second link is to the docs:
https://docs.python.org/3/library/concurrency.html
I don’t see where in the docs it says, it’s 2020 so do this from now on. Maybe that’s the problem?
Re: Linear Exeuction, Multiprocessing, and Multithreading IO-Bound Tasks in Python
#9Whenever I see a blog post on Python concurrency that recommends something other than concurrent.futures ThreadPoolExecutor or ProcessPoolExecutor, I shake my head in disappointment at how far we've strayed from "one obviously right way". Why in 2020 am I seeing someone manually close and join a worker pool (blocking!) to return all results all at once because they rewrote executor.map using multiprocessing? Why do t…
If I google for python concurrency the second link is to the docs: https://docs.python.org/3/library/concurrency.html I don’t see where in the docs it says, it’s 2020 so do this from now on. Maybe that’s the problem?
> I don’t see where in the docs it says, it’s 2020 so do this from now on. Maybe that’s the problem?
I guess lack of clear direction is exactly the problem. For all of PEP20's supposed importance, it's notoriously difficult to discover which parts of the standard library to use and which ones to ignore unless you like reinventing wheels.