Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel Programming with Python

chryswoods.com

Post reply on HN
Print view View original

141–147 of 147 posts

Re: Parallel Programming with Python

#141

Post by gpm »

quietbritishjim wrote: post #90
gpm wrote: post #87

Earlier quoted context omitted.

I've done it once, converting about 15 lines of python to rust. It was completely painless and resulted in a large speedup (changed a hotspot that was taking approximately 90% of execution time in a scientific simulation to approximately 0%). Type system and expressive macros seems like a big win over c to me.

Care to share a bit more detail on how you did this? Was there some interfacing library that you used analogous to Cython/SWIG/etc.? Presumably you didn't code directly against the C API (in python.h)?

The rust library interfacing with python is https://github.com/dgrunwald/rust-cpython This library understand things like python arrays, objects, etc. and provides nice rust interfaces to them. Basically I just have to write a macro that specifies what functions I'm exposing to python, and other then that I'm writing normal rust. On the python side I'm importing them and calling them like any other python library.

The build system is https://github.com/PyO3/setuptools-rust (which is linked at the bottom of the above readme).

Re: Parallel Programming with Python

#142

Post by goerz »

jashmatthews wrote: post #69
goerz wrote: post #60

The GIL has considerable benefits: I don’t have to worry about whether Python functions are thread-safe. Thread-based parallelism is hard to get right, and given the number of workarounds, Python’s GIL is a total non-issue.

> The GIL has considerable benefits: I don’t have to worry about whether Python functions are thread-safe. Hold on, the GIL doesn't make Python automatically thread-safe! You can still have classic data races as the VM can pause and resume two threads writing to the same variable.

Can you elaborate on that? Is there a blog post somewhere that illustrates the problem you're talking about? I was under the assumption that Python interpreters run single-threaded.

Re: Parallel Programming with Python

#143

Post by srean »

zbentley wrote: post #134
srean wrote: post #103

Earlier quoted context omitted.

> Because of copy-on-write fork magic, many multiprocessing configurations (including the default) can "send" that data to child processes in constant* time, if the data's already present in e.g. a global when children are created. Have you tried this or got it working ? The fly in the ointment is the reference count . Add a reference and BOOM you suddenly have a huge copy. It can be made to work efficiently in certa…

In practice, I find reference-count related issues with this pattern to be minor. Most of the situations where I care enough about memory and/or pickling overhead fall into the "take a giant block of binary/string data and process ranges of it in parallel" family, in which case there aren't too many references until the subprocesses get to work. If I had more complex structures of data I'd probably get a little less…

> in which case there aren't too many references until the subprocesses get to work.

In my workload that's exactly when it hits.

We ran into this when sharing different parts of a huge matrix with different workers. We had to be extra careful that we did not create new references in the subprocesses. We were operating at scale where if we got it wrong OOM will kill us.

Working with memory mapped arrays are more forgiving.

Re: Parallel Programming with Python

#144

Post by devxpy »

zbentley wrote: post #138
devxpy wrote: post #74

Earlier quoted context omitted.

Performance doesn't equal Better software. In fact, I think Performance centric development is a lesser known evil. > have all your data before creating your processes/pool Zproc exposes the required API for this (Nothing new, just the python API) :) https://zproc.readthedocs.io/en/latest/api.html#zproc.Proces... ( args and kwargs) > a massive dataset Wouldn't you be better off using a Database for that kind of work?…

> Any resources on how to implement that? big_data = read_huge_binary_or_string() def process_range(rng): start, end = rng do_something(big_data[start:end]) pool = multiprocesing.Pool(2) pool.map(process_range, [ (0, 10000), (10001, len(big_data), ])

Also, after doing some research:

The `multiprocessing.Pool` uses a `multiprocessing.Queue` in the background to retrieve the results after completion.

The `multiprocessing.Queue` in turn uses `multiprocessing.connection.Pipe` and sends the pickled objects over to the wire.

So I don't see how this is any better than ZMQ.

Just because stuff has an API that doesn't look like message passing doesn't mean it can't be doing that in the background. Which is funny, because that's the whole point of ZProc.

I realize the subtle difference that Cpython uses pipes, not sockets, unlike ZMQ. But that doesn't really make a difference now, does it?

Proof:

Process Pool worker, returning the result by using `outqueue.put()`

https://github.com/python/cpython/blob/86b89916d1b0a26c1e77f...

multiprocessing Queue, initializing a Pipe

https://github.com/python/cpython/blob/86b89916d1b0a26c1e77f...

multiprocessing Queue serializing data to send it using that Pipe

https://github.com/python/cpython/blob/86b89916d1b0a26c1e77f...

Re: Parallel Programming with Python

#145

Post by auggierose »

stevesimmons wrote: post #48
auggierose wrote: post #11

Earlier quoted context omitted.

Yeah. Recently switched some Blender Python algorithms I wrote to Swift/Metal, and the speedup was somewhere between 1000 and 1000000 depending on the algorithm.

Speedups of that magnitude suggest the original Python approach was particularly inefficient...

Not going to dispute that. If I spend time optimising code, I might do it as well in an environment like Swift/Metal instead of Python.

Re: Parallel Programming with Python

#146

Post by auggierose »

kilon wrote: post #65
auggierose wrote: post #11

Earlier quoted context omitted.

Yeah. Recently switched some Blender Python algorithms I wrote to Swift/Metal, and the speedup was somewhere between 1000 and 1000000 depending on the algorithm.

Who would have guessed that compiled, static, non-dynamic, hardware accelerated code would be a ton more performant than runtime, highly dynamic, garbage collected and very powerful code that is not hardware accelerated.

not sure though what you mean by "very powerful code" in that context ;-)

Re: Parallel Programming with Python

#147

Post by zbentley »

devxpy wrote: post #144
zbentley wrote: post #138

Earlier quoted context omitted.

> Any resources on how to implement that? big_data = read_huge_binary_or_string() def process_range(rng): start, end = rng do_something(big_data[start:end]) pool = multiprocesing.Pool(2) pool.map(process_range, [ (0, 10000), (10001, len(big_data), ])

Also, after doing some research: The `multiprocessing.Pool` uses a `multiprocessing.Queue` in the background to retrieve the results after completion. The `multiprocessing.Queue` in turn uses `multiprocessing.connection.Pipe` and sends the pickled objects over to the wire. So I don't see how this is any better than ZMQ. Just because stuff has an API that doesn't look like message passing doesn't mean it can't be doin…

No pipes or queues are used as part of the example code above. It transfers the large piece of data without serialization.

The point of the original post is that MP lets you do more than just serialize/ship data around after pool start time; there are substantial optimizations you can do if you know lots of the data you need to process early on.

Post reply on HN