Perhaps a strange choice of the word "synchronized" when coming from Java, this typically implies a critical section. Here it seems to initialize a multiprocessing pool for use in the function labelled concurrent (perhaps)?
Python Concurrency Decorators
11–20 of 51 posts
Re: Python Concurrency Decorators
#12A couple of things offhand: - It has Python 2 and 3 support - It's a wrapper for the Python built-in "multiprocessing" library - It spreads out work over all cores (so the abstraction hides the ability to control the pool) Seems like a great way to get your feet wet with multiprocessing in Python, but it likely has limited use in production...although certain infrastructures like resource limited containers might be…
Is there no way to specify the number of cores to use?
Re: Python Concurrency Decorators
#13So, for me, replacing imap with pool.imap is the easy part. The hard part is dealing with things like handling exceptions, catching keyboard interrupts, and so on. Does this module do anything to address these issues?
Re: Python Concurrency Decorators
#14A couple of things offhand: - It has Python 2 and 3 support - It's a wrapper for the Python built-in "multiprocessing" library - It spreads out work over all cores (so the abstraction hides the ability to control the pool) Seems like a great way to get your feet wet with multiprocessing in Python, but it likely has limited use in production...although certain infrastructures like resource limited containers might be…
It's not Py3 compatible yet, but from the commit log it seems like they're working on it (we could all help?).
Re: Python Concurrency Decorators
#15A couple of things offhand: - It has Python 2 and 3 support - It's a wrapper for the Python built-in "multiprocessing" library - It spreads out work over all cores (so the abstraction hides the ability to control the pool) Seems like a great way to get your feet wet with multiprocessing in Python, but it likely has limited use in production...although certain infrastructures like resource limited containers might be…
Is there no way to specify the number of cores to use?
You could certainly limit the process pool in the multiprocessing library itself: https://docs.python.org/2/library/multiprocessing.html ... but it doesn't seem like this decorator abstraction accounts for it.
> That's it, two lines of changes is all we need in order to parallelize this program. Now this program will make use of all the cores on the machine it's running on, allowing it to run significantly faster.
> As an overview, DECO is mainly just a smart wrapper for Python's multiprocessing.pool. When @concurrent is applied to a function it replaces it with calls to pool.apply_async. Additionally when arguments are passed to pool.apply_async, DECO replaces any index mutable objects with proxies, allowing it to detect and synchronize mutations of these objects. The results of these calls can then be obtained by calling wait() on the concurrent function, invoking a synchronization event.
I haven't really dug around the source code, but it sounds like not really.
Re: Python Concurrency Decorators
#16A couple of things offhand: - It has Python 2 and 3 support - It's a wrapper for the Python built-in "multiprocessing" library - It spreads out work over all cores (so the abstraction hides the ability to control the pool) Seems like a great way to get your feet wet with multiprocessing in Python, but it likely has limited use in production...although certain infrastructures like resource limited containers might be…
Is there no way to specify the number of cores to use?
Looks like there is:
https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee...
Re: Python Concurrency Decorators
#17A couple of things offhand: - It has Python 2 and 3 support - It's a wrapper for the Python built-in "multiprocessing" library - It spreads out work over all cores (so the abstraction hides the ability to control the pool) Seems like a great way to get your feet wet with multiprocessing in Python, but it likely has limited use in production...although certain infrastructures like resource limited containers might be…
Is there no way to specify the number of cores to use?
If you check out the test file[1] in the repository for deco, it looks like you can specify the number of cores by setting the "processes" attribute of the concurrent-decorated function [object].
1. https://github.com/alex-sherman/deco/blob/master/conc_test.p...
https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee...
Re: Python Concurrency Decorators
#18Earlier quoted context omitted.
Is there no way to specify the number of cores to use?
Looks like there is: https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee...
Interesting, so when they claim it automatically scales out to all cores what they mean is it defaults to 3 unless overridden.
Re: Python Concurrency Decorators
#19Re: Python Concurrency Decorators
#20A couple of things offhand: - It has Python 2 and 3 support - It's a wrapper for the Python built-in "multiprocessing" library - It spreads out work over all cores (so the abstraction hides the ability to control the pool) Seems like a great way to get your feet wet with multiprocessing in Python, but it likely has limited use in production...although certain infrastructures like resource limited containers might be…
It's not Py3 compatible yet, but from the commit log it seems like they're working on it (we could all help?).
I wonder what is keeping them from Python 3 support. I'll take a look, although I am currently stuck on Python 2.7 for my current project.