Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel Programming with Python

chryswoods.com

Post reply on HN
Print view View original

31–40 of 147 posts

Re: Parallel Programming with Python

#31

Post by TickleSteve »

devxpy wrote: post #13
armitron wrote: post #6

Earlier quoted context omitted.

They did not, which is why this "course" illustrates taking advantage of multiple cores via multiprocessing without mentioning the GIL at all. Which is a little misleading if you think about it. Also, by having the introductory chapter be about "functional programming" (which incidentally Python does not do well), he completely bypasses the serious issue of shared state. Which goes to show that parallelism in Python…

> deal with serialization and communication I thought a lot about this problem, for over 2 years, and came up with zproc https://github.com/pycampers/zproc Basically, > It lets you do message passing parallelism without the effort of tedious wiring. You'll be doing message passing without ever dealing with sockets! Also, Shared memory parallelism is hard to get right irregardless of which language you use. I would re…

you claim "To make utterly perfect MT programs (and I mean that literally)".

you've rediscovered message-passing... please take an elementary CS course on parallel systems.

That claim is naive in the extreme.

Re: Parallel Programming with Python

#32

Post by TickleSteve »

devxpy wrote: post #23
armitron wrote: post #17

Earlier quoted context omitted.

The mantra that shared memory parallelism is hard to get right to the point where such platitudes as "unless you're writing some really really really niche thing" are uttered is entirely erroneous I find, through my own experience. There are idiot-proof thread-safe datastructures and producer/consumer APIs that map extremely well to most problems that come up in practice in the domain, that one should confidently use…

You must be some sort of programming GOD, I guess. The problem is that its _hard_ to get right. For example - It's not trivial to use locks when you're working at an abstraction level higher than operating systems. Most people don't even realise there is a race in their application, because locks are inherently non-enforcing. Code written in locks is also really hard to read and reason by. Message passing just makes…

busy-waiting is a valid technique for some use-cases (and gives better performance in those situations) than other techniques.

Please research your topic.

Re: Parallel Programming with Python

#33

Post by brennebeck »

ggm wrote: post #7
elcombato wrote: post #3

> (note that you must be using Python 2 for this workshop and not using Python 3. Complete this workshop using Python 2, then read about the small changes if you are interested in using Python 3) Why using legacy Python for this?

why not re-write the workshop for python3 and require python2 users to wear the pain downgrade brings?

Because python2 is a deprecated language that will EOL?

Re: Parallel Programming with Python

#34

Post by ZeroCool2u »

ilovetux wrote: post #28

I find it strange that nobody ever seems to mention python's concurrent.futures module [0] which is new in Python 3.2. I think asyncio got a lot of attention when it came out in Python 3.4 and concurrent.futures took a back seat. This article also doesn't mention the module in it's Python 2 and 3 differences link. asyncio is a good library for asyncronous I/O but concurrent.futures gives us some pretty nifty tooling…

ThreadPoolExecutor and ProcessPoolExecutor were exactly what I was waiting for someone to mention. I was doing some Python as a systems architect at my previous position and now as a full time data scientist, my life has pretty much been consumed by Python. Unsurprisingly, a lot of my initial work is retrieving and cleaning very large volumes of data, the later usually being I/O bound and the former being CPU bound and frankly myself and a lot of my team immediately default to using both ThreadPoolExecutor and ProcessPoolExecutor respectively, because of how simple and performant they are. Perhaps asyncio is more familiar terminology to people coming from Web Dev, so that's why they're gravitating towards it, but there are few times when I find myself needing that particular tooling outside of Web Dev anyways.

Re: Parallel Programming with Python

#35

Post by quietbritishjim »

charlescearl wrote: post #29
quietbritishjim wrote: post #19

In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global interpreter lock): This is just not true, because C extension modules (i.e. libraries written to be used from Python but whose implementations are written in C) can release the global interpreter lock while inside a function call. Examples of these include nump…

Also a nice short talk by Caleb Hattingh https://www.youtube.com/watch?v=NfnMJMkhDoQ

> talk [about Cython]

That was interesting, thanks!

I really wish he had shown his numpy code. He said at 13:46 "Numpy actually doesn't help you at all because the calculation is still getting done at the Python level". But his function could be vectorised with numpy using functions like numpy.maximum or numpy.where, in which case the main loop will be in C not Python. I can't figure out from what he said whether his numpy code did that or not.

But either way, it's interesting that in this case the numpy version is arguably harder to write than the Cython version: rather than just adding a few bits of metadata (the types), you have to permute the whole control flow. If there's only a small amount of code you want to convert, I would still say it's better to use numpy though (if it actually is fast enough), because getting the build tools onto your computer for Cython can be a pain. And for some matrix computation there are speed inprovements above the fact that it's implemented in C e.g. matrix multiplication is faster than the naive O(n^3) version.

Re: Parallel Programming with Python

#36

Post by TickleSteve »

heavenlyblue wrote: post #25
devxpy wrote: post #15

Earlier quoted context omitted.

I believe that since the Advent of zeromq, parallelism is possible in almost any language, including python. My library lets you do parallelism in a unique way, where you do message passing parallelism without being explicit about it. https://github.com/pycampers/zproc/

>> Zproc uses a Server, which is responsible for storing and communicating the state. >> >> This isolates our resource (state), eliminating the need for locks. So you've just invented a new name for a coordinator process and called it a new fashion in computation?

No, he's reinvented multiprocessing... pickling data structures across multiple processes.

Just without the 'niceties'.

Re: Parallel Programming with Python

#37

Post by mpweiher »

another-cuppa wrote: post #12
jillesvangurp wrote: post #4

Did they ever fix the global interpreter lock? Sort of a show stopper with doing stuff concurrently in python. I've done a bit of batch processing using the multi process module; which uses processes instead of threads. This works but it is a bit of a kludge if you are used to languages that support concurrency properly.

Concurrency and parallelism are two different things. Python is fine for concurrency.

And the article is about "Parallel Programming with Python", in order to "...take advantage of the processing power of multicore processors".

Re: Parallel Programming with Python

#38

Post by chrisseaton »

quietbritishjim wrote: post #19

In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global interpreter lock): This is just not true, because C extension modules (i.e. libraries written to be used from Python but whose implementations are written in C) can release the global interpreter lock while inside a function call. Examples of these include nump…

> complaining that multithreading is impossible in Python without using multiple processes, because of the GIL ... this is not true

I think some people's opinions is that if you're writing in C then you're not really writing a Python program, so they think it is impossible in Python. Which seems a reasonable point to make to me.

Your argument is that Python is fine for multithreading... as long as you actually write C instead of Python.

Re: Parallel Programming with Python

#39

Post by TickleSteve »

mpweiher wrote: post #5

"...take advantage of the processing power of multicore processors" Step 1: stop using Python. "You can have a second core when you know how to use one" Now don't get me wrong, Python is a perfectly fine language for lots of things, but not for taking optimal advantage of the CPU. https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Relative performance compared to C is somewhere between an order of magnit…

You're absolutely right (but you're probably gonna get some downvotes for saying that).

The ratio between the most-performant parallel framework and the least on Python will be a factor of (guessing) 1.5.

The ratio between a CPU-bound algorithm written in C and one in Python will be of the order of 10000 (again guessing as it's application-dependent).

Where is your time most profitably spent?

Re: Parallel Programming with Python

#40

Post by zbentley »

TickleSteve wrote: post #30
devxpy wrote: post #15

Earlier quoted context omitted.

I believe that since the Advent of zeromq, parallelism is possible in almost any language, including python. My library lets you do parallelism in a unique way, where you do message passing parallelism without being explicit about it. https://github.com/pycampers/zproc/

You make some extremely large claims about ZProc, what advantages does it have over every other message-passing library for every other language ever built? (including the other zeromq bindings?) TBH, you're claims sound like you've just "discovered" message-passing, of which many, many languages, runtimes and operating systems have been using for many years/decades. ( https://en.wikipedia.org/wiki/Message_passing )…

> What you've created here is pretty much what multiprocessing gives you already in a more performant solution (i.e. no zeromq involved)

Minor point of pedantry which I'll state because it's an often-overlooked timesaver for folks developing on multiprocessing: not only is MP potentially faster for transferring data between processes compared to this solution, but it can also be way, way faster in situations where you have all your data before creating your processes/pool and just want to farm it out to your MP processes without waiting for it all to be chunked/pickled/unpickled.

Because of copy-on-write fork magic, many multiprocessing configurations (including the default) can "send" that data to child processes in constant* time, if the data's already present in e.g. a global when children are created.

This pattern can be used to totally bypass all considerations of performance/CPU/etc. for pickling/unpickling data and lends a massive speed boost in certain situations--e.g. a massive dataset is read into memory at startup, and then ranges of that dataset are processed in parallel by a pool of MP processes, each of which will return a relatively small result-set back to the parent, or each of which will write its processed (think: data scrubbing) range to a separate file which could be `cat`ed together, or written in parallel with careful `seek` bookkeeping.

Unix-ish OSes only, though (unless the fork() emulation in WSL works for this--I have not tested that).

* Technically it's O(N) for the size of data you have in memory at process pool start, because fork() can take time, but the multiplier is small enough in practice compared to sending data to/from MP processes via queues or whatever that it might as well be constant.

Post reply on HN