Live data from Hacker News

Python Concurrency Decorators

github.com

21–30 of 51 posts

Re: Python Concurrency Decorators

#21

Earlier quoted context omitted.

It's not Py3 compatible yet, but from the commit log it seems like they're working on it (we could all help?).

I wonder what is keeping them from Python 3 support. I'll take a look, although I am currently stuck on Python 2.7 for my current project.

See this issue comment: https://github.com/alex-sherman/deco/issues/2#issuecomment-2...

Re: Python Concurrency Decorators

#22

Perhaps a strange choice of the word "synchronized" when coming from Java, this typically implies a critical section. Here it seems to initialize a multiprocessing pool for use in the function labelled concurrent (perhaps)?

I think it's like sync in Cilk, meaning that all concurrent jobs must have finished before the part that is annotated synchronised is left. So it's not without precedence.

Re: Python Concurrency Decorators

#26

With Python 3.5 there is native support for concurrency by using the keywords await and async. For simple usage if you are familiar with Go there is this library: https://github.com/pothos/awaitchannel

Asynch and concurrency are very different. Python 3.5 allows first class aysnchronous calls, but concurrency is still "hard".

Re: Python Concurrency Decorators

#27

source looks surprising, for instance the decorators parse the decorated function code and build an AST (still have to find out why)

My best guess is that this is to circumvent issues with naming encountered by using decorated functions in conjunction with multiprocessing pools. Typically you would run into serialization errors.

Re: Python Concurrency Decorators

#28
post #23

I've been looking for a way to replace functools.partial and pool.map with something that could cause me to make bad architectural decisions. This could be the ticket.

Haha, having mucked around with similar mechanisms for dealing with concurrency via decorators I have to agree with you that this is likely to cause hard to debug behavior (especially since direct modification of the AST is going on behind the scenes here). That being said, it's an interesting thought experiment and likely an excellent class project.

Re: Python Concurrency Decorators

#29

A couple of things offhand: - It has Python 2 and 3 support - It's a wrapper for the Python built-in "multiprocessing" library - It spreads out work over all cores (so the abstraction hides the ability to control the pool) Seems like a great way to get your feet wet with multiprocessing in Python, but it likely has limited use in production...although certain infrastructures like resource limited containers might be…

Does anyone know how this library addresses the Global Interpreter Lock (GIL) issue, as multiprocessing really isn't that great/is worse than running on a single thread in many versions of python.

Re: Python Concurrency Decorators

#30

this is such a good idea! the @synchronized decorator to collect the parallelized task at the end of a parent call is very very smart & simple.

It also has a potential for slowing down your code if used willy nilly, of course.

Multiprocessing gets pretty useless for anything outside of independent CPU bound tasks with little IPC and simple data types that can be stuck into shared memory.

If you're using multiprocessing pools so often that you think you need a decorator to clean up your code, then wow, I'd like to see what you're up to. ;)

Post reply on HN