Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel Programming with Python

chryswoods.com

Post reply on HN
Print view View original

41–50 of 147 posts

Re: Parallel Programming with Python

#41

Post by zbentley »

TickleSteve wrote: post #36
heavenlyblue wrote: post #25

Earlier quoted context omitted.

>> Zproc uses a Server, which is responsible for storing and communicating the state. >> >> This isolates our resource (state), eliminating the need for locks. So you've just invented a new name for a coordinator process and called it a new fashion in computation?

No, he's reinvented multiprocessing... pickling data structures across multiple processes. Just without the 'niceties'.

You're probably right, but see my comment above: not only is MP possibly superior at being a picking/arbitrating server, but it also supports taking advantage of copy-on-write semantics on Unix-ish systems to transfer memory to children at startup in constant time with no pickling/unpickling necessary.

Re: Parallel Programming with Python

#42

Post by zbentley »

skrause wrote: post #10
dragonwriter wrote: post #8

Earlier quoted context omitted.

> Did they ever fix the global interpreter lock? Sort of a show stopper with doing stuff concurrently in python. It means threas-based parallelism of pure-python code is unavailable; concurrency is just fine on Python.

I have to work with Python on Windows and believe me, concurrency is not just fine in Python when you can't use fork().

Obligatory "concurrency != parallelism" statement; concurrency is fine on both platforms with Python threading in a single process with a GIL; parallelism is less of a done deal.

While it's a very big hammer, consider experimenting with Celery for your parallelism needs on Windows. I've had good results using per-script Celery "clusters" with either a filesystem (on a ramdisk for extra speed) or an embedded Redis backend to accomplish pretty nice bidirectional RPC-ish parallelism. The initial setup is much more complicated than something like goroutines, but once you get it working you can boilerplate it onto other tasks without much trouble.

It still won't save you from memory constraints imposed by the lack of good fork() emulation, though. Hopefully the WSL stuff will either bring better fork() emulation, or allow support for shared memory objects (e.g. multiprocessing.Value) in order to ease some of that pain.

Re: Parallel Programming with Python

#43

Post by ubernostrum »

chrisseaton wrote: post #38
quietbritishjim wrote: post #19

In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global interpreter lock): This is just not true, because C extension modules (i.e. libraries written to be used from Python but whose implementations are written in C) can release the global interpreter lock while inside a function call. Examples of these include nump…

> complaining that multithreading is impossible in Python without using multiple processes, because of the GIL ... this is not true I think some people's opinions is that if you're writing in C then you're not really writing a Python program, so they think it is impossible in Python. Which seems a reasonable point to make to me. Your argument is that Python is fine for multithreading... as long as you actually write…

A whole lot depends on what exactly it is that someone wants to get out of using threading.

The GIL means that a single Python interpreter process can execute at most one Python thread at a time, regardless of the number of CPUs or CPU cores available on the host machine. The GIL also introduces overhead which affects the performance of code using Python threads; how much you're affected by it will vary depending on what your code is doing. I/O-bound code tends to be much less affected, while CPU-bound code is much more affected.

All of this dates back to design decisions made in the 1990s which presumably seemed reasonable for the time: most people using Python were running it on machines with one CPU which had one core, so being able to take advantage of multiple CPUs/cores to schedule multiple threads to execute simultaneously was not necessarily a high priority. And most people who wanted threading wanted it to use in things like network daemons, which are primarily I/O-bound. Hence, the GIL and the set of tradeoffs it makes. Now, of course, we carry multi-core computers in our pockets and people routinely use Python for CPU-bound data science tasks. Hindsight is great at spotting that, but hindsight doesn't give us a time machine to go back and change the decisions.

Anyway. This is not the same thing as "multithreading is impossible". This is the same thing as "multithreading has some limitations, and for some cases the easiest way to work around them will be to use Python's C extension API". Which is what the parent comment seemed to be saying.

Re: Parallel Programming with Python

#44

Post by TheCondor »

quietbritishjim wrote: post #19

In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global interpreter lock): This is just not true, because C extension modules (i.e. libraries written to be used from Python but whose implementations are written in C) can release the global interpreter lock while inside a function call. Examples of these include nump…

The thread workers pool circumvent the GIL if you carefully follow some rules. I think the arguments and results need to be pickleable.

Re: Parallel Programming with Python

#45

Post by [deleted] »

ubernostrum wrote: post #43
chrisseaton wrote: post #38

Earlier quoted context omitted.

> complaining that multithreading is impossible in Python without using multiple processes, because of the GIL ... this is not true I think some people's opinions is that if you're writing in C then you're not really writing a Python program, so they think it is impossible in Python. Which seems a reasonable point to make to me. Your argument is that Python is fine for multithreading... as long as you actually write…

A whole lot depends on what exactly it is that someone wants to get out of using threading. The GIL means that a single Python interpreter process can execute at most one Python thread at a time, regardless of the number of CPUs or CPU cores available on the host machine. The GIL also introduces overhead which affects the performance of code using Python threads; how much you're affected by it will vary depending on…

[deleted]

Re: Parallel Programming with Python

#46

Post by quietbritishjim »

chrisseaton wrote: post #38
quietbritishjim wrote: post #19

In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global interpreter lock): This is just not true, because C extension modules (i.e. libraries written to be used from Python but whose implementations are written in C) can release the global interpreter lock while inside a function call. Examples of these include nump…

> complaining that multithreading is impossible in Python without using multiple processes, because of the GIL ... this is not true I think some people's opinions is that if you're writing in C then you're not really writing a Python program, so they think it is impossible in Python. Which seems a reasonable point to make to me. Your argument is that Python is fine for multithreading... as long as you actually write…

Let's say I write this:

    def add_and_mult(a, b, c):
        return a + b @ c
If a, b and c are numpy arrays then this function releases the GIL and so will run in multiple threads with no further work and with little overhead (if a, b and c are large). I would describe this as a function "written in Python", even though numpy uses C under the hood. It seems you describe this snippet as being "written in C instead of Python"; I find that odd, but OK.

But, if I understand you right, you are also suggesting that the other commenters here that talk about the GIL would also describe this as "written in C". They realise that this releases the GIL and will run on multiple threads, but the point of their comments is that proper pure Python function wouldn't. I disagree. I think that most others would describe this function as "written in Python", and when they say that functions written in Python can't be parallelised they do so because they don't realise that functions like this can be.

Re: Parallel Programming with Python

#47

Post by chrisseaton »

ubernostrum wrote: post #43
chrisseaton wrote: post #38

Earlier quoted context omitted.

> complaining that multithreading is impossible in Python without using multiple processes, because of the GIL ... this is not true I think some people's opinions is that if you're writing in C then you're not really writing a Python program, so they think it is impossible in Python. Which seems a reasonable point to make to me. Your argument is that Python is fine for multithreading... as long as you actually write…

A whole lot depends on what exactly it is that someone wants to get out of using threading. The GIL means that a single Python interpreter process can execute at most one Python thread at a time, regardless of the number of CPUs or CPU cores available on the host machine. The GIL also introduces overhead which affects the performance of code using Python threads; how much you're affected by it will vary depending on…

Right I should have said parallel threads, not multithreading.

Re: Parallel Programming with Python

#48

Post by stevesimmons »

auggierose wrote: post #11
mpweiher wrote: post #5

"...take advantage of the processing power of multicore processors" Step 1: stop using Python. "You can have a second core when you know how to use one" Now don't get me wrong, Python is a perfectly fine language for lots of things, but not for taking optimal advantage of the CPU. https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Relative performance compared to C is somewhere between an order of magnit…

Yeah. Recently switched some Blender Python algorithms I wrote to Swift/Metal, and the speedup was somewhere between 1000 and 1000000 depending on the algorithm.

Speedups of that magnitude suggest the original Python approach was particularly inefficient...

Re: Parallel Programming with Python

#49

Post by blattimwind »

zbentley wrote: post #40
TickleSteve wrote: post #30

Earlier quoted context omitted.

You make some extremely large claims about ZProc, what advantages does it have over every other message-passing library for every other language ever built? (including the other zeromq bindings?) TBH, you're claims sound like you've just "discovered" message-passing, of which many, many languages, runtimes and operating systems have been using for many years/decades. ( https://en.wikipedia.org/wiki/Message_passing )…

> What you've created here is pretty much what multiprocessing gives you already in a more performant solution (i.e. no zeromq involved) Minor point of pedantry which I'll state because it's an often-overlooked timesaver for folks developing on multiprocessing: not only is MP potentially faster for transferring data between processes compared to this solution, but it can also be way, way faster in situations where yo…

> Because of copy-on-write fork magic

Note that this works for big objects, but not for small objects. E.g. if you fork-share a large list of integers or dicts or something like that, then you don't get any memory usage benefits, because every access will cause a refcount-write and that will copy the whole page containing the object.

> * Technically it's O(N) for the size of data you have in memory at process pool start

It's not quite that simple; sharing n pages can take very little time or a bit more time; it depends on how the pages are mapped; sharing a large mapping doesn't take longer than a small mapping.

Re: Parallel Programming with Python

#50

Post by quietbritishjim »

TheCondor wrote: post #44
quietbritishjim wrote: post #19

In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global interpreter lock): This is just not true, because C extension modules (i.e. libraries written to be used from Python but whose implementations are written in C) can release the global interpreter lock while inside a function call. Examples of these include nump…

The thread workers pool circumvent the GIL if you carefully follow some rules. I think the arguments and results need to be pickleable.

I think you're thinking of the multiprocessing module, which uses separate processes to bypass the GIL. That's why the arguments and results need to be pickleable: pickle is a serialisation procotol, so it allows you to communicate the contents of objects between different processes. If you use threads within a single process, you don't need to pickle the objects; you just pass the object directly.
Post reply on HN