Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel Programming with Python

chryswoods.com

Post reply on HN
Print view View original

61–70 of 147 posts

Re: Parallel Programming with Python

#61

Post by quietbritishjim »

chrisseaton wrote: post #51
quietbritishjim wrote: post #46

Earlier quoted context omitted.

Let's say I write this: def add_and_mult(a, b, c): return a + b @ c If a, b and c are numpy arrays then this function releases the GIL and so will run in multiple threads with no further work and with little overhead (if a, b and c are large). I would describe this as a function "written in Python", even though numpy uses C under the hood. It seems you describe this snippet as being "written in C instead of Python";…

This only gives you parallelism in one specific situation though - where operations like '+' and '@' take a long time. If they were fine-grained operations, then this doesn't help you. If instead of operating on a numerical matrix, you were instead operating on something like a graph of Python objects, something like a graph traversal would be hard to parallelise as you could not stay out of the GIL long enough to ge…

I agree that not quite all situations will be covered by coarse locks like this. But many will, and my original comment was meant to draw attention to those situations. Your previous comment seemed to be saying that everyone already knew about those, but I still believe that some people commenting or reading here weren't aware you could release the GIL with just a numpy call.

I did also concede that if you do have to write your algorithm completely from scratch, with no scope for using existing C extensions (be they general purpose like numpy or more specialist that implements the whole algorithm) then yes you'll be caught be the GIL, so I agree with you on that. But I also made the point that you'll be caught even more (orders of magintude more!) by the slowness of Python, so any discussion about parallelism or the GIL is a red herring. It's like worrying that you car's windscreen will start to melt if you travel at 500mph; even if that's technically true, it's not the problem you should be focusing on.

It's interesting you mention graphs because the most popular liberally licensed graph library is NetworkX, which is indeed pure Python and so presumably isn't particularly fast. There are graph libraries written as C extension modules but I believe they are less popular and less librally licensed (GPL-style rather than BSD-style). So I definitely agree that this is a big weakness of the Python ecosystem.

Re: Parallel Programming with Python

#62

Post by weberc2 »

sidlls wrote: post #52
ram_rar wrote: post #16

I love python. But its seriously, incapable for doing non trivial concurrent tasks. Multiprocessing module doesnt count. I hope the python core-devs take some inspiration from golang for developing the right abstractions for concurrency.

While I agree that python isn't ideal beyond a certain scope, I think you're overstating how bad it is. My team and I have built a number of non-trivial machine learning products with pipelines that use both the ThreadPool and ProcessPool components successfully. The headaches we have are related more to the fact that Python is dynamic than its concurrency story.

OP probably is overstating a bit, but it is hard to efficiently parallelize computation in Python. For example, if you have a large Python object graph that you need to compute over, you can't easily parallize the computation without paying some significant serialization cost. You can probably alleviate that by carefully choosing algorithms that minimize the amount of serialization per worker process, but at the end of the day, all of this is still quite a lot harder than using shared memory and goroutines. And not to mention Go is 1-2 orders of magnitude faster than Python in single-threaded execution... Python is great for lots of things, but efficient parallel programming in Python is _hard_, even if there are a handful of cases where it's not so hard.

Re: Parallel Programming with Python

#63

Post by weberc2 »

nicolaslem wrote: post #21
ram_rar wrote: post #16

I love python. But its seriously, incapable for doing non trivial concurrent tasks. Multiprocessing module doesnt count. I hope the python core-devs take some inspiration from golang for developing the right abstractions for concurrency.

As a developer working mostly with Python this comment makes no sense to me. There are hundreds of libraries to deal with concurrency and/or parallelism in Python, asyncio, Celery and PySpark being the common ones. All of them provide different approaches to concurrency because the language itself is not tight to one in particular.

These are all quite a lot harder to use than Go, and often they don't play well together. For example, there are lots of sync libraries (the Docker API, the AWS SDK, etc) that can't be turned async, so other folks have had to go through the trouble of forking and porting to async and since those other folks are often not affiliated with the original dev teams, who knows what the quality level of those libraries may be? We've also had a lot of problems with asyncio alone--often developers forgetting to await an async call or doing something (I'm not sure what exactly) that causes processes to hang indefinitely. It's all quite a lot more complex than Go's concurrency model.

And all of that is really just I/O parallelization; there's also CPU parallelization, and I don't believe Python has anything that's quite as easy as "Do these two things in parallel". Pretty much everything requires a lot of marshalling and process management which can easily slow a program down instead of improving it.

Python is great for a lot of things, and the community has found many creative workarounds for its shortcomings, but Go beats Python in I/O and CPU parallelism handily.

Re: Parallel Programming with Python

#64

Post by eternauta3k »

mpweiher wrote: post #5

"...take advantage of the processing power of multicore processors" Step 1: stop using Python. "You can have a second core when you know how to use one" Now don't get me wrong, Python is a perfectly fine language for lots of things, but not for taking optimal advantage of the CPU. https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Relative performance compared to C is somewhere between an order of magnit…

Why? I can run my Python plotting script with multiprocessing in one of the blade servers at work and get the job done quickly. All without translating a big bunch of code to C.

Re: Parallel Programming with Python

#65

Post by kilon »

auggierose wrote: post #11
mpweiher wrote: post #5

"...take advantage of the processing power of multicore processors" Step 1: stop using Python. "You can have a second core when you know how to use one" Now don't get me wrong, Python is a perfectly fine language for lots of things, but not for taking optimal advantage of the CPU. https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Relative performance compared to C is somewhere between an order of magnit…

Yeah. Recently switched some Blender Python algorithms I wrote to Swift/Metal, and the speedup was somewhere between 1000 and 1000000 depending on the algorithm.

Who would have guessed that compiled, static, non-dynamic, hardware accelerated code would be a ton more performant than runtime, highly dynamic, garbage collected and very powerful code that is not hardware accelerated.

Re: Parallel Programming with Python

#66

Post by devxpy »

TickleSteve wrote: post #30
devxpy wrote: post #15

Earlier quoted context omitted.

I believe that since the Advent of zeromq, parallelism is possible in almost any language, including python. My library lets you do parallelism in a unique way, where you do message passing parallelism without being explicit about it. https://github.com/pycampers/zproc/

You make some extremely large claims about ZProc, what advantages does it have over every other message-passing library for every other language ever built? (including the other zeromq bindings?) TBH, you're claims sound like you've just "discovered" message-passing, of which many, many languages, runtimes and operating systems have been using for many years/decades. ( https://en.wikipedia.org/wiki/Message_passing )…

> "high performance"

I never claimed it to be performant!

"Above all, ZProc is written for safety and the ease of use."

(Read here - https://github.com/pycampers/zproc?files=1#faq)

> It's not a revolution

I totally agree. It's just a better way of doing things zmq already perfected. Like, tell me if you've ever seen a python object that has a `dict` API, but does message passing in the background.

> central (pubsub?) server.

Central server, yes. It uses PUB-SUB for state watching and REQ-REP for everything else.

> you've just "discovered" message-passing

Guess you're right? 2 years is a peanut on the time scale...

P.S. Thanks for all the feedback, I've been dying to hear something for a while now.

Re: Parallel Programming with Python

#67

Post by kilon »

One more epic discussion on Python, where we have the unique opportunity to learn that using C libraries from Python is "cheating".

I could not agree more

It's definitely cheating to use C code with the exception of most Python libraries that already are to a large extent nothing more than thin wrappers over existing C libraries or the tiny fact that the most popular by far implementation of Python , CPython, is almost 50% implemented in the C language, including the standard library.The author even dared include "C" in the name of the implementation.

Those cheaters, becoming bolder and bolder every day.

Damn them !!!

Re: Parallel Programming with Python

#68

Post by devxpy »

heavenlyblue wrote: post #25
devxpy wrote: post #15

Earlier quoted context omitted.

I believe that since the Advent of zeromq, parallelism is possible in almost any language, including python. My library lets you do parallelism in a unique way, where you do message passing parallelism without being explicit about it. https://github.com/pycampers/zproc/

>> Zproc uses a Server, which is responsible for storing and communicating the state. >> >> This isolates our resource (state), eliminating the need for locks. So you've just invented a new name for a coordinator process and called it a new fashion in computation?

Great, now just "stating" how things work is equivalent to inventing them!

Re: Parallel Programming with Python

#69

Post by jashmatthews »

goerz wrote: post #60

The GIL has considerable benefits: I don’t have to worry about whether Python functions are thread-safe. Thread-based parallelism is hard to get right, and given the number of workarounds, Python’s GIL is a total non-issue.

> The GIL has considerable benefits: I don’t have to worry about whether Python functions are thread-safe.

Hold on, the GIL doesn't make Python automatically thread-safe!

You can still have classic data races as the VM can pause and resume two threads writing to the same variable.

Re: Parallel Programming with Python

#70

Post by devxpy »

TickleSteve wrote: post #31
devxpy wrote: post #13

Earlier quoted context omitted.

> deal with serialization and communication I thought a lot about this problem, for over 2 years, and came up with zproc https://github.com/pycampers/zproc Basically, > It lets you do message passing parallelism without the effort of tedious wiring. You'll be doing message passing without ever dealing with sockets! Also, Shared memory parallelism is hard to get right irregardless of which language you use. I would re…

you claim "To make utterly perfect MT programs (and I mean that literally)". you've rediscovered message-passing... please take an elementary CS course on parallel systems. That claim is naive in the extreme.

That's not my claim man, its written in the zguide

http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ

Maybe I should've just linked it there,sorry!

Okay, I will take that course and get back, thanks for the suggestion.

P.S. You just implied Pieter Hintjens is naive. You have to live with that now :(

Post reply on HN