Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel Programming with Python

chryswoods.com

Post reply on HN
Print view View original

71–80 of 147 posts

Re: Parallel Programming with Python

#71

Post by devxpy »

TickleSteve wrote: post #32
devxpy wrote: post #23

Earlier quoted context omitted.

You must be some sort of programming GOD, I guess. The problem is that its _hard_ to get right. For example - It's not trivial to use locks when you're working at an abstraction level higher than operating systems. Most people don't even realise there is a race in their application, because locks are inherently non-enforcing. Code written in locks is also really hard to read and reason by. Message passing just makes…

busy-waiting is a valid technique for some use-cases (and gives better performance in those situations) than other techniques. Please research your topic.

Yes, but isn't it more CPU intensive?

(Speaking purely from experience. Don't have a fancy CS degree)

Re: Parallel Programming with Python

#72

Post by devxpy »

TickleSteve wrote: post #55
zbentley wrote: post #41

Earlier quoted context omitted.

You're probably right, but see my comment above: not only is MP possibly superior at being a picking/arbitrating server, but it also supports taking advantage of copy-on-write semantics on Unix-ish systems to transfer memory to children at startup in constant time with no pickling/unpickling necessary.

I agree, multiprocessing will be more performant than ZProc, much more thought has gone into it than the simple 0mq wrapper that is ZProc.

Exactly.

Re: Parallel Programming with Python

#73

Post by [deleted] »

kilon wrote: post #67

One more epic discussion on Python, where we have the unique opportunity to learn that using C libraries from Python is "cheating". I could not agree more It's definitely cheating to use C code with the exception of most Python libraries that already are to a large extent nothing more than thin wrappers over existing C libraries or the tiny fact that the most popular by far implementation of Python , CPython, is almo…

[deleted]

Re: Parallel Programming with Python

#74

Post by devxpy »

zbentley wrote: post #40
TickleSteve wrote: post #30

Earlier quoted context omitted.

You make some extremely large claims about ZProc, what advantages does it have over every other message-passing library for every other language ever built? (including the other zeromq bindings?) TBH, you're claims sound like you've just "discovered" message-passing, of which many, many languages, runtimes and operating systems have been using for many years/decades. ( https://en.wikipedia.org/wiki/Message_passing )…

> What you've created here is pretty much what multiprocessing gives you already in a more performant solution (i.e. no zeromq involved) Minor point of pedantry which I'll state because it's an often-overlooked timesaver for folks developing on multiprocessing: not only is MP potentially faster for transferring data between processes compared to this solution, but it can also be way, way faster in situations where yo…

Performance doesn't equal Better software.

In fact, I think Performance centric development is a lesser known evil.

> have all your data before creating your processes/pool

Zproc exposes the required API for this (Nothing new, just the python API) :)

https://zproc.readthedocs.io/en/latest/api.html#zproc.Proces... (args and kwargs)

> a massive dataset

Wouldn't you be better off using a Database for that kind of work?

> Because of copy-on-write fork magic, many multiprocessing configurations (including the default) can "send" that data to child processes in constant time

Any resources on how to implement that?

Re: Parallel Programming with Python

#76

Post by devxpy »

TickleSteve wrote: post #39
mpweiher wrote: post #5

"...take advantage of the processing power of multicore processors" Step 1: stop using Python. "You can have a second core when you know how to use one" Now don't get me wrong, Python is a perfectly fine language for lots of things, but not for taking optimal advantage of the CPU. https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Relative performance compared to C is somewhere between an order of magnit…

You're absolutely right (but you're probably gonna get some downvotes for saying that). The ratio between the most-performant parallel framework and the least on Python will be a factor of (guessing) 1.5. The ratio between a CPU-bound algorithm written in C and one in Python will be of the order of 10000 (again guessing as it's application-dependent). Where is your time most profitably spent?

Wild Guess: You're not really a python programmer, are you?

Just curious...

Re: Parallel Programming with Python

#77

Post by devxpy »

armitron wrote: post #17
devxpy wrote: post #13

Earlier quoted context omitted.

> deal with serialization and communication I thought a lot about this problem, for over 2 years, and came up with zproc https://github.com/pycampers/zproc Basically, > It lets you do message passing parallelism without the effort of tedious wiring. You'll be doing message passing without ever dealing with sockets! Also, Shared memory parallelism is hard to get right irregardless of which language you use. I would re…

The mantra that shared memory parallelism is hard to get right to the point where such platitudes as "unless you're writing some really really really niche thing" are uttered is entirely erroneous I find, through my own experience. There are idiot-proof thread-safe datastructures and producer/consumer APIs that map extremely well to most problems that come up in practice in the domain, that one should confidently use…

This talk (hopefully) conveys my point across

https://www.youtube.com/watch?v=9zinZmE3Ogk

Re: Parallel Programming with Python

#78

Post by devxpy »

goerz wrote: post #60

The GIL has considerable benefits: I don’t have to worry about whether Python functions are thread-safe. Thread-based parallelism is hard to get right, and given the number of workarounds, Python’s GIL is a total non-issue.

Small correction: It makes the _implementation_ thread-safe.

It also simplifies a lot of CPython code, making it a lot easier to maintain.

Re: Parallel Programming with Python

#79

Post by TickleSteve »

devxpy wrote: post #66
TickleSteve wrote: post #30

Earlier quoted context omitted.

You make some extremely large claims about ZProc, what advantages does it have over every other message-passing library for every other language ever built? (including the other zeromq bindings?) TBH, you're claims sound like you've just "discovered" message-passing, of which many, many languages, runtimes and operating systems have been using for many years/decades. ( https://en.wikipedia.org/wiki/Message_passing )…

> "high performance" I never claimed it to be performant! "Above all, ZProc is written for safety and the ease of use." (Read here - https://github.com/pycampers/zproc?files=1#faq ) > It's not a revolution I totally agree. It's just a better way of doing things zmq already perfected. Like, tell me if you've ever seen a python object that has a `dict` API, but does message passing in the background. > central (pubsub?…

I would suggest you don't make dramatic claims for a subject that has decades of theory behind it with a huge amount of nuance depending on the exact workload and characteristics of the machines in question.

Don't get me wrong, message-passing has some advantages, but they certainly aren't that it 'solves' parallelism. If you wish to know more, investigate:

- Smalltalk and Erlang (for message passing languages).

- QNX (for a message-passing OS)

- mpiPY (for a message-passing Python library, mpi is the grandfather of message passing libraries that runs everywhere).

- Occam & the transputer for an example of a hardware-mp implementation (actually its Communicating Sequential Processes, but for your purposes it would be enlightening).

- golang for a modern-day implementation of CSP.

- Python implementation of CSP (https://github.com/futurecore/python-csp)

- Discussion about MP (http://wiki.c2.com/?MessagePassingConcurrency, for more just google it)

Basically, its great that you want to learn about concurrency & parallelism, but you've come to a gun fight with a blunt butter knife.

Re: Parallel Programming with Python

#80

Post by jwandborg »

Rotareti wrote: post #24
quietbritishjim wrote: post #19

In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global interpreter lock): This is just not true, because C extension modules (i.e. libraries written to be used from Python but whose implementations are written in C) can release the global interpreter lock while inside a function call. Examples of these include nump…

Does anyone know how well Python and Rust team up compared to Python and C in practice?

Subjectively I'm really impressed by PyO3.

If you care about speed, Rust is supposedly as fast as C. The Rust ecosystem also has a lot of supposedly safe(!) tools for parallelism.

Post reply on HN