Do people here use pypy in production? What are the benefits?
Let's Remove the Global Interpreter Lock
11–20 of 326 posts
Re: Let's Remove the Global Interpreter Lock
#12Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is
1. Big enough to need concurrency
2. Not big enough to require multiple boxes.
3. Running in a situation that can not spare the resources for multiprocessing.
4. You want to share memory instead of designing your workflow to handle messages or working off a queue.
Re: Let's Remove the Global Interpreter Lock
#13This would be great if it means we can run the C portions of Python in threads without performance hits. I recently started a little project that is a cross-platform GUI for batch bzip2 compression, and Python did it quite well with its built-in bzip2 module. But, once I tried to do it parallel, the performance impacts of GIL were obvious. Yes, you can work around that with multi-process, but I'd rather not be spamming the running processes list and have to actually handle seperate processes that should be threads.
In the end I settled for C++ and QT with the native bzip2 library with a few modifications.
Re: Let's Remove the Global Interpreter Lock
#14Would this work with cpython extensions that were ported to PyPy?
They would run under GIL (I can't see CPython C API being thread-friendly unless gilectomy succeeds)
Ah, ok. So this approach doesn't completely remove the GIL, but removes it as a barrier for pure python code running in PyPy?
Or does it break the current support for porting cpython extensions?
Re: Let's Remove the Global Interpreter Lock
#15Do people here use pypy in production? What are the benefits?
Sure. Free 2-5x speedup. pypy + pypy's pip generally works as a transparent drop-in replacement to python + python's pip, so it's free speed.
It doesn't (or didn't) work when you need to rely on an extension that uses Python's C API. I haven't followed the scene in awhile so maybe that's changed. pypy's pip has so many libraries that I hardly notice, so maybe they solved that.
Unfortunately python is fundamentally slower than lua or JS, possibly due to the object model. Python traps all method calls, but even integer addition, comparisons, and so on are treated as metamethods. That's the case for Lua too, but e.g. it's absurdly easy to make a Python object have a custom length, whereas Lua didn't have a __len__ metamethod until after 5.1. I'm not sure it even works on LuaJIT either. Probably in the newer versions.
Re: Let's Remove the Global Interpreter Lock
#16Just curious: if they solve it in Python, would it be possible to solve it in Ruby too ?
This is in the cards for Ruby 3. The plan is to migrate away from a global interpreter lock to "guilds" which are akin to execution contexts. These guilds also have a locking mechanism that allows for parallelism which they call the "global guild lock."
You can learn more about concurrency in Ruby 3 at this wonderful blog post: http://olivierlacan.com/posts/concurrency-in-ruby-3-with-gui...
Re: Let's Remove the Global Interpreter Lock
#17Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…
There are many cases where the objects are too big to be passed around. Python is used a huge amount in Machine learning and datascience, where being able to do parallel work on stuff already in memory would be great.
Re: Let's Remove the Global Interpreter Lock
#18Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…
Using multiprocessing is a pain to use, and it's slow.
Re: Let's Remove the Global Interpreter Lock
#19Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…
Using multiprocessing is a pain to use, and it's slow.
in what was is it a pain that threading is not?
Re: Let's Remove the Global Interpreter Lock
#20Super glad they're going to try using mutexes and not that STM approach which was looking to be immensely complicated. Was not looking forward to the kinds of interpreter bugs that was going to produce.