Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel Programming with Python

chryswoods.com

Post reply on HN
Print view View original

101–110 of 147 posts

Re: Parallel Programming with Python

#101

Post by adw »

devxpy wrote: post #15
another-cuppa wrote: post #12

Earlier quoted context omitted.

Concurrency and parallelism are two different things. Python is fine for concurrency.

I believe that since the Advent of zeromq, parallelism is possible in almost any language, including python. My library lets you do parallelism in a unique way, where you do message passing parallelism without being explicit about it. https://github.com/pycampers/zproc/

> My library lets you do parallelism in a unique way

That's a big claim which you don't really back up as much as you need to. Unique is an extremely high bar in this very busy field.

There are several other similar red flags on the linked GitHub; I think your enthusiasm is running away from you a little. You might want to dial the ten-dollar language back a bit – it made me immediately suspicious ("utterly perfect", for example is another danger phrase).

It's the combination of grandiose language + solution-in-search-of-a-problem which leads to that.

If you're going to sell hard, what I would want to see is a large, complex, high-traffic system which makes extensive use of this; if you compare and contrast with Ray, which I've also only just encountered in this thread, there's a real problem (distributed hyperparameter optimization) which they've built a solution for with the library, and that immediately lends it credibility; I know the system can be used for something because it has been.

Re: Parallel Programming with Python

#102

Post by TickleSteve »

devxpy wrote: post #85
TickleSteve wrote: post #79

Earlier quoted context omitted.

I would suggest you don't make dramatic claims for a subject that has decades of theory behind it with a huge amount of nuance depending on the exact workload and characteristics of the machines in question. Don't get me wrong, message-passing has some advantages, but they certainly aren't that it 'solves' parallelism. If you wish to know more, investigate: - Smalltalk and Erlang (for message passing languages). - QN…

> I would suggest you don't make dramatic claims If you could point out some stuff from ZProc's page, that would be nice! > mpi is the grandfather of message passing libraries Never heard of it before, but just a simple google search reveals that it _might_ be more performant than zmq, but not as fault-tolerant and flexible. It really looks like a niche thing, from this comment by peter hintjens > Why smart cloud bui…

Nothing against zeromq, its good s/w, but like all tools it must be used appropriately.

...also, nanomsg is the 'improved' successor.

Also, MPI isn't a 'niche' thing, its the way that a large proportion of high-performance applications have been implemented for a few decades (think Crays & weather prediction). Zeromq has a few simple web-apps using it (I exagerate slightly).

Re: Parallel Programming with Python

#103

Post by srean »

zbentley wrote: post #40
TickleSteve wrote: post #30

Earlier quoted context omitted.

You make some extremely large claims about ZProc, what advantages does it have over every other message-passing library for every other language ever built? (including the other zeromq bindings?) TBH, you're claims sound like you've just "discovered" message-passing, of which many, many languages, runtimes and operating systems have been using for many years/decades. ( https://en.wikipedia.org/wiki/Message_passing )…

> What you've created here is pretty much what multiprocessing gives you already in a more performant solution (i.e. no zeromq involved) Minor point of pedantry which I'll state because it's an often-overlooked timesaver for folks developing on multiprocessing: not only is MP potentially faster for transferring data between processes compared to this solution, but it can also be way, way faster in situations where yo…

> Because of copy-on-write fork magic, many multiprocessing configurations (including the default) can "send" that data to child processes in constant* time, if the data's already present in e.g. a global when children are created.

Have you tried this or got it working ? The fly in the ointment is the reference count. Add a reference and BOOM you suddenly have a huge copy. It can be made to work efficiently in certain cases but takes a lot of care.

Re: Parallel Programming with Python

#104

Post by targafarian »

stevesimmons wrote: post #48
auggierose wrote: post #11

Earlier quoted context omitted.

Yeah. Recently switched some Blender Python algorithms I wrote to Swift/Metal, and the speedup was somewhere between 1000 and 1000000 depending on the algorithm.

Speedups of that magnitude suggest the original Python approach was particularly inefficient...

Yeah, properly written Python is at extreme worst O(1000) times slower than speeding it up code with a Numpy/Numba/c/Fortran/etc. implementation. Brute-force loopy code in Python I've seen is 100x slower than the compiled alternatives. So I agree, these extreme numbers are the sign of writing the worst possible Python implementation of a thing and saying Python sucks.

Re: Parallel Programming with Python

#105

Post by lozenge »

srean wrote: post #103
zbentley wrote: post #40

Earlier quoted context omitted.

> What you've created here is pretty much what multiprocessing gives you already in a more performant solution (i.e. no zeromq involved) Minor point of pedantry which I'll state because it's an often-overlooked timesaver for folks developing on multiprocessing: not only is MP potentially faster for transferring data between processes compared to this solution, but it can also be way, way faster in situations where yo…

> Because of copy-on-write fork magic, many multiprocessing configurations (including the default) can "send" that data to child processes in constant* time, if the data's already present in e.g. a global when children are created. Have you tried this or got it working ? The fly in the ointment is the reference count . Add a reference and BOOM you suddenly have a huge copy. It can be made to work efficiently in certa…

You can call gc.freeze that effectively sets all reference counts to infinity.

Re: Parallel Programming with Python

#106

Post by woolvalley »

quietbritishjim wrote: post #19

In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global interpreter lock): This is just not true, because C extension modules (i.e. libraries written to be used from Python but whose implementations are written in C) can release the global interpreter lock while inside a function call. Examples of these include nump…

Because we want first class python multithreading, like many other languages have. If we have to drop down into C, might as well use another language with first class multi-threading like java, kotlin, golang or swift and avoid all the other issues that come with slow GIL languages.

Re: Parallel Programming with Python

#108

Post by devxpy »

TickleSteve wrote: post #89
devxpy wrote: post #70

Earlier quoted context omitted.

That's not my claim man, its written in the zguide http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ Maybe I should've just linked it there,sorry! Okay, I will take that course and get back, thanks for the suggestion. P.S. You just implied Pieter Hintjens is naive. You have to live with that now :(

I think you took that claim out of context: "By "perfect MT programs", I mean code that's easy to write and understand, that works with the same design approach in any programming language, and on any operating system, and that scales across any number of CPUs with zero wait states and no point of diminishing returns." That doesn't mean to say its "perfect" or "solves" multithreading, just that its easy to write and…

> That doesn't mean to say its "perfect" or "solves" multithreading, just that its easy to write and understand

Try saying that out loud?

Re: Parallel Programming with Python

#109

Post by devxpy »

TickleSteve wrote: post #102
devxpy wrote: post #85

Earlier quoted context omitted.

> I would suggest you don't make dramatic claims If you could point out some stuff from ZProc's page, that would be nice! > mpi is the grandfather of message passing libraries Never heard of it before, but just a simple google search reveals that it _might_ be more performant than zmq, but not as fault-tolerant and flexible. It really looks like a niche thing, from this comment by peter hintjens > Why smart cloud bui…

Nothing against zeromq, its good s/w, but like all tools it must be used appropriately. ...also, nanomsg is the 'improved' successor. Also, MPI isn't a 'niche' thing, its the way that a large proportion of high-performance applications have been implemented for a few decades (think Crays & weather prediction). Zeromq has a few simple web-apps using it (I exagerate slightly).

Seems like you know quite a lot about this topic. Do you have any projects of your own that I can see?

Or do you only work for your employer or something?

Re: Parallel Programming with Python

#110

Post by kjeetgill »

ubernostrum wrote: post #43
chrisseaton wrote: post #38

Earlier quoted context omitted.

> complaining that multithreading is impossible in Python without using multiple processes, because of the GIL ... this is not true I think some people's opinions is that if you're writing in C then you're not really writing a Python program, so they think it is impossible in Python. Which seems a reasonable point to make to me. Your argument is that Python is fine for multithreading... as long as you actually write…

A whole lot depends on what exactly it is that someone wants to get out of using threading. The GIL means that a single Python interpreter process can execute at most one Python thread at a time, regardless of the number of CPUs or CPU cores available on the host machine. The GIL also introduces overhead which affects the performance of code using Python threads; how much you're affected by it will vary depending on…

> All of this dates back to design decisions made in the 1990s which presumably seemed reasonable for the time ... Hence, the GIL and the set of tradeoffs it makes. Now, of course, we carry multi-core computers ... Hindsight is great at spotting that, but hindsight doesn't give us a time machine to go back and change the decisions.

Sadly I don't think this is _quite_ true. I believe GILs are used in a number of interpreters and fall prey to the common problem of where either coarsening locks or making them finer ruins somebody's day. I believe Guido Van Rossum hung the GILectomy on two main issues: The interpreter must remain relatively simple, and C extensions cannot be slowed down.

I'm not disagreeing with the decision (necessarily) but it isn't simply a layover from a bygone era. It was a decision that has been reaffirmed and upheld numerous times.

[0]: https://lwn.net/Articles/754577/ [1]: just google Gilectomy, it's been covered in a few places that I don't have handy.

Post reply on HN