Python Concurrency Decorators

Post by pdonis » Fri, May 13, 2016, 12:43 AM UTC

zjaffee wrote: post #29

gshulegaard wrote: post #8

A couple of things offhand: - It has Python 2 and 3 support - It's a wrapper for the Python built-in "multiprocessing" library - It spreads out work over all cores (so the abstraction hides the ability to control the pool) Seems like a great way to get your feet wet with multiprocessing in Python, but it likely has limited use in production...although certain infrastructures like resource limited containers might be…

Does anyone know how this library addresses the Global Interpreter Lock (GIL) issue, as multiprocessing really isn't that great/is worse than running on a single thread in many versions of python.

Post by awinter-py » Fri, May 13, 2016, 1:12 AM UTC

nomel wrote: post #30

awinter-py wrote: post #5

this is such a good idea! the @synchronized decorator to collect the parallelized task at the end of a parent call is very very smart & simple.

It also has a potential for slowing down your code if used willy nilly, of course. Multiprocessing gets pretty useless for anything outside of independent CPU bound tasks with little IPC and simple data types that can be stuck into shared memory. If you're using multiprocessing pools so often that you think you need a decorator to clean up your code, then wow, I'd like to see what you're up to. ;)

Post by lqdc13 » Fri, May 13, 2016, 1:42 AM UTC

hyperion2010 wrote: post #32

rcthompson wrote: post #13

So, for me, replacing imap with pool.imap is the easy part. The hard part is dealing with things like handling exceptions, catching keyboard interrupts, and so on. Does this module do anything to address these issues?

I have a hack for dealing with KeyboardInterrupt's on a ProcessPoolExecutor: https://github.com/tgbugs/desc/blob/master/util/process_fixe... . I used this in concert with asyncio run_in_executor which helps with some of the exception handling.

Post by m_mueller » Fri, May 13, 2016, 2:18 AM UTC

nomel wrote: post #30

awinter-py wrote: post #5

this is such a good idea! the @synchronized decorator to collect the parallelized task at the end of a parent call is very very smart & simple.

It also has a potential for slowing down your code if used willy nilly, of course. Multiprocessing gets pretty useless for anything outside of independent CPU bound tasks with little IPC and simple data types that can be stuck into shared memory. If you're using multiprocessing pools so often that you think you need a decorator to clean up your code, then wow, I'd like to see what you're up to. ;)

Post by tbarbugli » Fri, May 13, 2016, 6:12 AM UTC

madisonmay wrote: post #27

tbarbugli wrote: post #24

source looks surprising, for instance the decorators parse the decorated function code and build an AST (still have to find out why)

My best guess is that this is to circumvent issues with naming encountered by using decorated functions in conjunction with multiprocessing pools. Typically you would run into serialization errors.