Python Concurrency Decorators • Hacker News Board

Re: Python Concurrency Decorators

#41

Post by elpres » Fri, May 13, 2016, 8:08 AM UTC

thebigspacefuck wrote: post #10

Another page you might be interested in bookmarking: https://wiki.python.org/moin/PythonDecoratorLibrary A similar decorator to thread function calls for concurrency: https://wiki.python.org/moin/PythonDecoratorLibrary#Lazy_Thu...

Doesn't this second decorator only work with functions doing IO or releasing the GIL in any other way (like time.sleep() that they use in the example)? If all the function is doing is actual computation (i.e. using the CPU), no other code can run in parallel with it because of the GIL.

Re: Python Concurrency Decorators

#42

Post by [deleted] » Fri, May 13, 2016, 8:46 AM UTC

elpres wrote: post #41

thebigspacefuck wrote: post #10

Another page you might be interested in bookmarking: https://wiki.python.org/moin/PythonDecoratorLibrary A similar decorator to thread function calls for concurrency: https://wiki.python.org/moin/PythonDecoratorLibrary#Lazy_Thu...

Doesn't this second decorator only work with functions doing IO or releasing the GIL in any other way (like time.sleep() that they use in the example)? If all the function is doing is actual computation (i.e. using the CPU), no other code can run in parallel with it because of the GIL.

[deleted]

Re: Python Concurrency Decorators

#43

Post by cyberfart » Fri, May 13, 2016, 9:16 AM UTC

"We have proposed DECO, a simplification of concurrent programming techniques targeted at programmers with little understanding of concurrent programming." (from paper)

gave me the chills

Re: Python Concurrency Decorators

#44

Post by [deleted] » Fri, May 13, 2016, 10:12 AM UTC

elpres wrote: post #41

thebigspacefuck wrote: post #10

Another page you might be interested in bookmarking: https://wiki.python.org/moin/PythonDecoratorLibrary A similar decorator to thread function calls for concurrency: https://wiki.python.org/moin/PythonDecoratorLibrary#Lazy_Thu...

Doesn't this second decorator only work with functions doing IO or releasing the GIL in any other way (like time.sleep() that they use in the example)? If all the function is doing is actual computation (i.e. using the CPU), no other code can run in parallel with it because of the GIL.

[deleted]

Re: Python Concurrency Decorators

#45

Post by thebigspacefuck » Fri, May 13, 2016, 2:40 PM UTC

elpres wrote: post #41

thebigspacefuck wrote: post #10

Another page you might be interested in bookmarking: https://wiki.python.org/moin/PythonDecoratorLibrary A similar decorator to thread function calls for concurrency: https://wiki.python.org/moin/PythonDecoratorLibrary#Lazy_Thu...

Doesn't this second decorator only work with functions doing IO or releasing the GIL in any other way (like time.sleep() that they use in the example)? If all the function is doing is actual computation (i.e. using the CPU), no other code can run in parallel with it because of the GIL.

Yes, that's right, it is best used for I/O-bound applications. You could use it for:

* running multiple Database queries

* SSH-ing into multiple devices to run a command

* loading multiple web pages

* calling multiple APIs

Re: Python Concurrency Decorators

#46

Post by hcarvalhoalves » Fri, May 13, 2016, 4:53 PM UTC

thebigspacefuck wrote: post #10

Another page you might be interested in bookmarking: https://wiki.python.org/moin/PythonDecoratorLibrary A similar decorator to thread function calls for concurrency: https://wiki.python.org/moin/PythonDecoratorLibrary#Lazy_Thu...

Never heard this name for this pattern (thunk). Isn't this the same as a Future/Promise object?

Re: Python Concurrency Decorators

#47

Post by hcarvalhoalves » Fri, May 13, 2016, 4:54 PM UTC

Re: Python Concurrency Decorators

#48

Post by [deleted] » Fri, May 13, 2016, 6:41 PM UTC

elpres wrote: post #41

thebigspacefuck wrote: post #10

Another page you might be interested in bookmarking: https://wiki.python.org/moin/PythonDecoratorLibrary A similar decorator to thread function calls for concurrency: https://wiki.python.org/moin/PythonDecoratorLibrary#Lazy_Thu...

Doesn't this second decorator only work with functions doing IO or releasing the GIL in any other way (like time.sleep() that they use in the example)? If all the function is doing is actual computation (i.e. using the CPU), no other code can run in parallel with it because of the GIL.

[deleted]

Re: Python Concurrency Decorators

#49

Post by issaria » Fri, May 13, 2016, 7:11 PM UTC

hcarvalhoalves wrote: post #46

thebigspacefuck wrote: post #10

Another page you might be interested in bookmarking: https://wiki.python.org/moin/PythonDecoratorLibrary A similar decorator to thread function calls for concurrency: https://wiki.python.org/moin/PythonDecoratorLibrary#Lazy_Thu...

Never heard this name for this pattern (thunk). Isn't this the same as a Future/Promise object?

Thunk is from haskell,.

Re: Python Concurrency Decorators

#50

Post by vector57 » Fri, May 13, 2016, 7:25 PM UTC

rcthompson wrote: post #12

gshulegaard wrote: post #8

A couple of things offhand: - It has Python 2 and 3 support - It's a wrapper for the Python built-in "multiprocessing" library - It spreads out work over all cores (so the abstraction hides the ability to control the pool) Seems like a great way to get your feet wet with multiprocessing in Python, but it likely has limited use in production...although certain infrastructures like resource limited containers might be…

Is there no way to specify the number of cores to use?

There is in the latest version. Using like @concurrent(...) passes all arguments directly to Pool(...)