Parallel Programming with Python
chryswoods.com
Parallel Programming with Python
1–10 of 147 posts
Re: Parallel Programming with Python
#2Sweet, a guide! I always end up rolling my own thread pool / manager. I wish something like the parallel gem for Ruby existed in pyland...
Re: Parallel Programming with Python
#3> (note that you must be using Python 2 for this workshop and not using Python 3. Complete this workshop using Python 2, then read about the small changes if you are interested in using Python 3)
Why using legacy Python for this?
Re: Parallel Programming with Python
#4Did they ever fix the global interpreter lock? Sort of a show stopper with doing stuff concurrently in python. I've done a bit of batch processing using the multi process module; which uses processes instead of threads. This works but it is a bit of a kludge if you are used to languages that support concurrency properly.
Re: Parallel Programming with Python
#5"...take advantage of the processing power of multicore processors"
Step 1: stop using Python.
"You can have a second core when you know how to use one"
Now don't get me wrong, Python is a perfectly fine language for lots of things, but not for taking optimal advantage of the CPU.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Relative performance compared to C is somewhere between an order of magnitude or two slower. Considering how much harder and more error-prone multi-core is, maybe first try a fast sequential solution.
Re: Parallel Programming with Python
#6Did they ever fix the global interpreter lock? Sort of a show stopper with doing stuff concurrently in python. I've done a bit of batch processing using the multi process module; which uses processes instead of threads. This works but it is a bit of a kludge if you are used to languages that support concurrency properly.
They did not, which is why this "course" illustrates taking advantage of multiple cores via multiprocessing without mentioning the GIL at all. Which is a little misleading if you think about it.
Also, by having the introductory chapter be about "functional programming" (which incidentally Python does not do well), he completely bypasses the serious issue of shared state.
Which goes to show that parallelism in Python is more like a gimmick than a real-world solution since it doesn't let you do in-process shared-memory processing via threads in parallel which is so important for many applications. In my case, the vast majority of the time I do not want to farm workers out to different operating system processes and deal with serialization and communication, but this is the only way for Python code to take advantage of multiple cores [1].
[1] Another way is to write a module in C and have Python code call into it on a new thread and release the GIL while doing so, but of course this is even worse pain-wise than doing it with multiprocessing and you end up writing/compiling C.
Re: Parallel Programming with Python
#7> (note that you must be using Python 2 for this workshop and not using Python 3. Complete this workshop using Python 2, then read about the small changes if you are interested in using Python 3) Why using legacy Python for this?
why not re-write the workshop for python3 and require python2 users to wear the pain downgrade brings?
Re: Parallel Programming with Python
#8Did they ever fix the global interpreter lock? Sort of a show stopper with doing stuff concurrently in python. I've done a bit of batch processing using the multi process module; which uses processes instead of threads. This works but it is a bit of a kludge if you are used to languages that support concurrency properly.
> Did they ever fix the global interpreter lock? Sort of a show stopper with doing stuff concurrently in python.
It means threas-based parallelism of pure-python code is unavailable; concurrency is just fine on Python.
Re: Parallel Programming with Python
#9i think use python3 multiprocess and async is better for product.
Re: Parallel Programming with Python
#10Did they ever fix the global interpreter lock? Sort of a show stopper with doing stuff concurrently in python. I've done a bit of batch processing using the multi process module; which uses processes instead of threads. This works but it is a bit of a kludge if you are used to languages that support concurrency properly.
> Did they ever fix the global interpreter lock? Sort of a show stopper with doing stuff concurrently in python. It means threas-based parallelism of pure-python code is unavailable; concurrency is just fine on Python.
I have to work with Python on Windows and believe me, concurrency is not just fine in Python when you can't use fork().