Live data from Hacker News

Parallel Programming with Python

chryswoods.com

51–60 of 147 posts

Re: Parallel Programming with Python

#51

Earlier quoted context omitted.

> complaining that multithreading is impossible in Python without using multiple processes, because of the GIL ... this is not true I think some people's opinions is that if you're writing in C then you're not really writing a Python program, so they think it is impossible in Python. Which seems a reasonable point to make to me. Your argument is that Python is fine for multithreading... as long as you actually write…

Let's say I write this: def add_and_mult(a, b, c): return a + b @ c If a, b and c are numpy arrays then this function releases the GIL and so will run in multiple threads with no further work and with little overhead (if a, b and c are large). I would describe this as a function "written in Python", even though numpy uses C under the hood. It seems you describe this snippet as being "written in C instead of Python";…

This only gives you parallelism in one specific situation though - where operations like '+' and '@' take a long time. If they were fine-grained operations, then this doesn't help you.

If instead of operating on a numerical matrix, you were instead operating on something like a graph of Python objects, something like a graph traversal would be hard to parallelise as you could not stay out of the GIL long enough to get anything done.

Re: Parallel Programming with Python

#52
post #16

I love python. But its seriously, incapable for doing non trivial concurrent tasks. Multiprocessing module doesnt count. I hope the python core-devs take some inspiration from golang for developing the right abstractions for concurrency.

While I agree that python isn't ideal beyond a certain scope, I think you're overstating how bad it is. My team and I have built a number of non-trivial machine learning products with pipelines that use both the ThreadPool and ProcessPool components successfully. The headaches we have are related more to the fact that Python is dynamic than its concurrency story.

Re: Parallel Programming with Python

#53

Earlier quoted context omitted.

> complaining that multithreading is impossible in Python without using multiple processes, because of the GIL ... this is not true I think some people's opinions is that if you're writing in C then you're not really writing a Python program, so they think it is impossible in Python. Which seems a reasonable point to make to me. Your argument is that Python is fine for multithreading... as long as you actually write…

Let's say I write this: def add_and_mult(a, b, c): return a + b @ c If a, b and c are numpy arrays then this function releases the GIL and so will run in multiple threads with no further work and with little overhead (if a, b and c are large). I would describe this as a function "written in Python", even though numpy uses C under the hood. It seems you describe this snippet as being "written in C instead of Python";…

[deleted]

Re: Parallel Programming with Python

#54

Earlier quoted context omitted.

Let's say I write this: def add_and_mult(a, b, c): return a + b @ c If a, b and c are numpy arrays then this function releases the GIL and so will run in multiple threads with no further work and with little overhead (if a, b and c are large). I would describe this as a function "written in Python", even though numpy uses C under the hood. It seems you describe this snippet as being "written in C instead of Python";…

This only gives you parallelism in one specific situation though - where operations like '+' and '@' take a long time. If they were fine-grained operations, then this doesn't help you. If instead of operating on a numerical matrix, you were instead operating on something like a graph of Python objects, something like a graph traversal would be hard to parallelise as you could not stay out of the GIL long enough to ge…

This is a good point. Fine grained parallelism works well with c extensions, but coarse grained does not. Which is why data science and MATLAB like tasks are so often done in python without worrying too much about the python penalty. But if you have a lot of small dense matrix operations, even VBA in excel will be faster by more than 10x because you keep popping back into python.

Re: Parallel Programming with Python

#55

Earlier quoted context omitted.

No, he's reinvented multiprocessing... pickling data structures across multiple processes. Just without the 'niceties'.

You're probably right, but see my comment above: not only is MP possibly superior at being a picking/arbitrating server, but it also supports taking advantage of copy-on-write semantics on Unix-ish systems to transfer memory to children at startup in constant time with no pickling/unpickling necessary.

I agree, multiprocessing will be more performant than ZProc, much more thought has gone into it than the simple 0mq wrapper that is ZProc.

Re: Parallel Programming with Python

#56

In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global interpreter lock): This is just not true, because C extension modules (i.e. libraries written to be used from Python but whose implementations are written in C) can release the global interpreter lock while inside a function call. Examples of these include nump…

that's not writing Python though

Re: Parallel Programming with Python

#58

I think a lot of this complexity can be avoided by just writing single threaded python and using GNU parallel for running it on multiple cores. You can even trivially distribute the work across a cluster that way.

This is the approach I've taken, albeit at the "top level" of the program. Since I know I don't have to deal with Windows I much prefer simply piping to parallel instead of xargs, or calling make -j8, or similarly letting some shell wrapper handle it over dealing with the overhead inside of python, especially multiprocessing.

However, where I think having this stuff available inside of python is useful is that it's cross platform and consumable from "higher levels" of python. A library can do some mucky stuff internally to speed computation but still present a simple sync interface, all without external dependencies.

Re: Parallel Programming with Python

#59

Earlier quoted context omitted.

This only gives you parallelism in one specific situation though - where operations like '+' and '@' take a long time. If they were fine-grained operations, then this doesn't help you. If instead of operating on a numerical matrix, you were instead operating on something like a graph of Python objects, something like a graph traversal would be hard to parallelise as you could not stay out of the GIL long enough to ge…

This is a good point. Fine grained parallelism works well with c extensions, but coarse grained does not. Which is why data science and MATLAB like tasks are so often done in python without worrying too much about the python penalty. But if you have a lot of small dense matrix operations, even VBA in excel will be faster by more than 10x because you keep popping back into python.

I agree with the idea, but you have fine grained and course grained switched.

Re: Parallel Programming with Python

#60
The GIL has considerable benefits: I don’t have to worry about whether Python functions are thread-safe. Thread-based parallelism is hard to get right, and given the number of workarounds, Python’s GIL is a total non-issue.
Post reply on HN