Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel Programming with Python

chryswoods.com

Post reply on HN
Print view View original

81–90 of 147 posts

Re: Parallel Programming with Python

#81

Post by devxpy »

azag0 wrote: post #20
ram_rar wrote: post #16

I love python. But its seriously, incapable for doing non trivial concurrent tasks. Multiprocessing module doesnt count. I hope the python core-devs take some inspiration from golang for developing the right abstractions for concurrency.

Concurrent or parallel? For concurrency, python has asyncio, which many people consider a success. For parallel execution, there's the GIL, but in practice it rarely matters, because once you want to do parallel execution, you have most likely a computationally intensive task to do, at which point you call down to C or something, and then GIL doesn't matter.

> most likely a computationally intensive task to do

Eh, let me stop you there. Everything isn't about performance.

Hardware and UI based things really benefit from parallelism.

Re: Parallel Programming with Python

#82

Post by devxpy »

Rotareti wrote: post #27
ram_rar wrote: post #16

I love python. But its seriously, incapable for doing non trivial concurrent tasks. Multiprocessing module doesnt count. I hope the python core-devs take some inspiration from golang for developing the right abstractions for concurrency.

I hope some of the (new?) concepts from trio find their way into the standard lib. trio: https://github.com/python-trio/trio trio compared to asyncio, goroutines, etc.: https://stackoverflow.com/a/49485603/1612318 "Notes on structured concurrency, or: Go statement considered harmful": https://vorpus.org/blog/notes-on-structured-concurrency-or-g...

> https://vorpus.org/blog/notes-on-structured-concurrency-or-g...

Was a damn good read, Thanks!

Re: Parallel Programming with Python

#83

Post by TickleSteve »

devxpy wrote: post #76
TickleSteve wrote: post #39

Earlier quoted context omitted.

You're absolutely right (but you're probably gonna get some downvotes for saying that). The ratio between the most-performant parallel framework and the least on Python will be a factor of (guessing) 1.5. The ratio between a CPU-bound algorithm written in C and one in Python will be of the order of 10000 (again guessing as it's application-dependent). Where is your time most profitably spent?

Wild Guess: You're not really a python programmer, are you? Just curious...

Yes, I am... ( and C/C++ )

But I use the right tool for the job. Python is a great tool, but not for performance (Applies to all dynamic, interpreted languages TBH).

Re: Parallel Programming with Python

#84

Post by bogomipz »

quietbritishjim wrote: post #46
chrisseaton wrote: post #38

Earlier quoted context omitted.

> complaining that multithreading is impossible in Python without using multiple processes, because of the GIL ... this is not true I think some people's opinions is that if you're writing in C then you're not really writing a Python program, so they think it is impossible in Python. Which seems a reasonable point to make to me. Your argument is that Python is fine for multithreading... as long as you actually write…

Let's say I write this: def add_and_mult(a, b, c): return a + b @ c If a, b and c are numpy arrays then this function releases the GIL and so will run in multiple threads with no further work and with little overhead (if a, b and c are large). I would describe this as a function "written in Python", even though numpy uses C under the hood. It seems you describe this snippet as being "written in C instead of Python";…

Could you explain the return statement in your example? I only know the '@' as a decorator in Python. This looks like invalid syntax to me what am I missing?

Re: Parallel Programming with Python

#85

Post by devxpy »

TickleSteve wrote: post #79
devxpy wrote: post #66

Earlier quoted context omitted.

> "high performance" I never claimed it to be performant! "Above all, ZProc is written for safety and the ease of use." (Read here - https://github.com/pycampers/zproc?files=1#faq ) > It's not a revolution I totally agree. It's just a better way of doing things zmq already perfected. Like, tell me if you've ever seen a python object that has a `dict` API, but does message passing in the background. > central (pubsub?…

I would suggest you don't make dramatic claims for a subject that has decades of theory behind it with a huge amount of nuance depending on the exact workload and characteristics of the machines in question. Don't get me wrong, message-passing has some advantages, but they certainly aren't that it 'solves' parallelism. If you wish to know more, investigate: - Smalltalk and Erlang (for message passing languages). - QN…

> I would suggest you don't make dramatic claims

If you could point out some stuff from ZProc's page, that would be nice!

> mpi is the grandfather of message passing libraries

Never heard of it before, but just a simple google search reveals that it _might_ be more performant than zmq, but not as fault-tolerant and flexible. It really looks like a niche thing, from this comment by peter hintjens

> Why smart cloud builders are betting everything on 0MQ. In detail, compare to the alternatives. Hand-rolling your own TCP stack is insane. Using any broker-based product won't scale. Buying licenses from IBM or TIBCO would eat up your capital. Supercomputing products like MPI aren't designed for this scale. There is literally no alternative.

(http://zeromq.org/docs:the-ten-minute-talk)

> Don't get me wrong, message-passing has some advantages, but they certainly aren't that it 'solves' parallelism.

Doesn't it? (For most people)

---

I can't believe I'm hearing words against zmq on HN, its wierd.

Even the guys over at Dask settled on ZMQ over anything - https://github.com/dask/distributed/issues/776

P.S. Seems like you know quite a lot about this topic. Do you have any projects of your own that I can see?

Bottom line, I think most people would be happy doing message passing parallelism in the real world. Sure, it doesn't look that good in theory but works damn good in practicality.

Re: Parallel Programming with Python

#86

Post by devxpy »

TickleSteve wrote: post #83
devxpy wrote: post #76

Earlier quoted context omitted.

Wild Guess: You're not really a python programmer, are you? Just curious...

Yes, I am... ( and C/C++ ) But I use the right tool for the job. Python is a great tool, but not for performance (Applies to all dynamic, interpreted languages TBH).

> C/C++

Got it!

Re: Parallel Programming with Python

#87

Post by gpm »

Rotareti wrote: post #24
quietbritishjim wrote: post #19

In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global interpreter lock): This is just not true, because C extension modules (i.e. libraries written to be used from Python but whose implementations are written in C) can release the global interpreter lock while inside a function call. Examples of these include nump…

Does anyone know how well Python and Rust team up compared to Python and C in practice?

I've done it once, converting about 15 lines of python to rust. It was completely painless and resulted in a large speedup (changed a hotspot that was taking approximately 90% of execution time in a scientific simulation to approximately 0%).

Type system and expressive macros seems like a big win over c to me.

Re: Parallel Programming with Python

#88

Post by augusto2112 »

bogomipz wrote: post #84
quietbritishjim wrote: post #46

Earlier quoted context omitted.

Let's say I write this: def add_and_mult(a, b, c): return a + b @ c If a, b and c are numpy arrays then this function releases the GIL and so will run in multiple threads with no further work and with little overhead (if a, b and c are large). I would describe this as a function "written in Python", even though numpy uses C under the hood. It seems you describe this snippet as being "written in C instead of Python";…

Could you explain the return statement in your example? I only know the '@' as a decorator in Python. This looks like invalid syntax to me what am I missing?

It's matrix multiplication.

Has been in Python since version 3.5

https://www.python.org/dev/peps/pep-0465/

Re: Parallel Programming with Python

#89

Post by TickleSteve »

devxpy wrote: post #70
TickleSteve wrote: post #31

Earlier quoted context omitted.

you claim "To make utterly perfect MT programs (and I mean that literally)". you've rediscovered message-passing... please take an elementary CS course on parallel systems. That claim is naive in the extreme.

That's not my claim man, its written in the zguide http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ Maybe I should've just linked it there,sorry! Okay, I will take that course and get back, thanks for the suggestion. P.S. You just implied Pieter Hintjens is naive. You have to live with that now :(

I think you took that claim out of context:

"By "perfect MT programs", I mean code that's easy to write and understand, that works with the same design approach in any programming language, and on any operating system, and that scales across any number of CPUs with zero wait states and no point of diminishing returns."

That doesn't mean to say its "perfect" or "solves" multithreading, just that its easy to write and understand and portable across architectures. That says nothing of how optimal it is for concurrency or parallelism ease-of-use wise or performance-wise, just that its 'easy'.

Re: Parallel Programming with Python

#90

Post by quietbritishjim »

gpm wrote: post #87
Rotareti wrote: post #24

Earlier quoted context omitted.

Does anyone know how well Python and Rust team up compared to Python and C in practice?

I've done it once, converting about 15 lines of python to rust. It was completely painless and resulted in a large speedup (changed a hotspot that was taking approximately 90% of execution time in a scientific simulation to approximately 0%). Type system and expressive macros seems like a big win over c to me.

Care to share a bit more detail on how you did this? Was there some interfacing library that you used analogous to Cython/SWIG/etc.? Presumably you didn't code directly against the C API (in python.h)?
Post reply on HN