Live data from Hacker News

Parallel Programming with Python

chryswoods.com

91–100 of 147 posts

Re: Parallel Programming with Python

#91

Earlier quoted context omitted.

Could you explain the return statement in your example? I only know the '@' as a decorator in Python. This looks like invalid syntax to me what am I missing?

It's matrix multiplication. Has been in Python since version 3.5 https://www.python.org/dev/peps/pep-0465/

Thanks, I'm not quite that up to date on my 3.x Python. Cheers.

Re: Parallel Programming with Python

#92
post #23

Earlier quoted context omitted.

The mantra that shared memory parallelism is hard to get right to the point where such platitudes as "unless you're writing some really really really niche thing" are uttered is entirely erroneous I find, through my own experience. There are idiot-proof thread-safe datastructures and producer/consumer APIs that map extremely well to most problems that come up in practice in the domain, that one should confidently use…

You must be some sort of programming GOD, I guess. The problem is that its _hard_ to get right. For example - It's not trivial to use locks when you're working at an abstraction level higher than operating systems. Most people don't even realise there is a race in their application, because locks are inherently non-enforcing. Code written in locks is also really hard to read and reason by. Message passing just makes…

I do concurrency in Java all the time with CompletableFuture and threadsafe data structures provided by various libraries, e.g. the Guava caches, and I rarely need to use locks or semaphores. It's a good set of abstractions that make concurrency pretty close to idiot-proof.

Futures in particular make it easy to write concurrent code close to the way you would write single-threaded code, because all of the threading is handled behind the scenes.

Re: Parallel Programming with Python

#93
post #66

Earlier quoted context omitted.

> "high performance" I never claimed it to be performant! "Above all, ZProc is written for safety and the ease of use." (Read here - https://github.com/pycampers/zproc?files=1#faq ) > It's not a revolution I totally agree. It's just a better way of doing things zmq already perfected. Like, tell me if you've ever seen a python object that has a `dict` API, but does message passing in the background. > central (pubsub?…

I would suggest you don't make dramatic claims for a subject that has decades of theory behind it with a huge amount of nuance depending on the exact workload and characteristics of the machines in question. Don't get me wrong, message-passing has some advantages, but they certainly aren't that it 'solves' parallelism. If you wish to know more, investigate: - Smalltalk and Erlang (for message passing languages). - QN…

HN comment section shouldn't be a gun fight.

Re: Parallel Programming with Python

#95
post #93

Earlier quoted context omitted.

I would suggest you don't make dramatic claims for a subject that has decades of theory behind it with a huge amount of nuance depending on the exact workload and characteristics of the machines in question. Don't get me wrong, message-passing has some advantages, but they certainly aren't that it 'solves' parallelism. If you wish to know more, investigate: - Smalltalk and Erlang (for message passing languages). - QN…

HN comment section shouldn't be a gun fight.

It isn't, it was more a metaphor for the fact that he's making arguments way out of his league of understanding.

Re: Parallel Programming with Python

#97
post #16

I love python. But its seriously, incapable for doing non trivial concurrent tasks. Multiprocessing module doesnt count. I hope the python core-devs take some inspiration from golang for developing the right abstractions for concurrency.

I successfully do concurrent+parallel computing with Python using asyncio combined with ProcessPoolExecutor. I can see why perhaps that doesn't scratch your itch, but it sure scratches my web-crawling itch.

Re: Parallel Programming with Python

#99
post #5

"...take advantage of the processing power of multicore processors" Step 1: stop using Python. "You can have a second core when you know how to use one" Now don't get me wrong, Python is a perfectly fine language for lots of things, but not for taking optimal advantage of the CPU. https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Relative performance compared to C is somewhere between an order of magnit…

> Considering how much harder and more error-prone multi-core is, maybe first try a fast sequential solution.

Most of the Python programs referenced on that benchmarks game webpage are in-fact using multi-core ?

Re: Parallel Programming with Python

#100
post #71

Earlier quoted context omitted.

busy-waiting is a valid technique for some use-cases (and gives better performance in those situations) than other techniques. Please research your topic.

Yes, but isn't it more CPU intensive? (Speaking purely from experience. Don't have a fancy CS degree)

It uses 100% CPU, true but when the duration of the lock is extremely small (i.e. nanoseconds->microseconds) the total CPU usage is less than arranging for an OS level context-switch. In other words, you use it when synchronising with hardware or when implementing test-and-set primitives for higher level mechanisms. Crucially, the time that the lock is held for must be very short.

Given those restrictions and use cases you get a very efficient low latency locking mechanism.

Post reply on HN