First Python 2.7 interpreter to use multiple cores
mail.python.org
First Python 2.7 interpreter to use multiple cores
1–10 of 42 posts
Re: First Python 2.7 interpreter to use multiple cores
#2I'm not following this. Does it mean the GIL is finally gone?
Re: First Python 2.7 interpreter to use multiple cores
#3I'm not following this. Does it mean the GIL is finally gone?
Yes (although so far only in this branch of PyPy). It basically uses fine-grained locking instead of a global lock. I can't explain it simply so I'll just point to recent discussion: https://news.ycombinator.com/item?id=3941642 and https://news.ycombinator.com/item?id=3464484 and the beginning of this project https://news.ycombinator.com/item?id=2710235
Re: First Python 2.7 interpreter to use multiple cores
#4I'm not following this. Does it mean the GIL is finally gone?
This is the first of what hopes to be many releases of pypy using STM for multithreaded safety. Up until this release Armin had a special API for creating and running transactions. Now he has wrapped python's threading module to use STM, see http://morepypy.blogspot.com/2012/05/stm-update-back-to-thre... for more information.
Note that this is independent of CPython development.
Re: First Python 2.7 interpreter to use multiple cores
#5To be a bit pedantic: the multiprocess module works fine for multiple cores in regular cpython, I actually used it for a little graphics program I wrote. Nice to see the GIL gone for multithreading though :)
Re: First Python 2.7 interpreter to use multiple cores
#6I much prefer share-nothing approach and use sockets + subprocess for communication. It scales nicely.
Re: First Python 2.7 interpreter to use multiple cores
#7To be a bit pedantic: the multiprocess module works fine for multiple cores in regular cpython, I actually used it for a little graphics program I wrote. Nice to see the GIL gone for multithreading though :)
It isn't pedantic just irrelevant. Using multiprocess is an entirely different beast than using threads. All shared values have to explicitly allocated that way in shared memory. Since there is a separate interpreter for each process there is no problem with internal state getting messed up. Threads are just a harder problem for python and one which has never seen an effective solution.[1] That is why this kind of work is exciting.
[1] You may argue (correctly) that multiprocessing and using pipes to communicate is a better solution than threads to begin with (for many applications). However, sometimes a threaded solution is the right solution and previously if you needed threads python didn't work for you.
Re: First Python 2.7 interpreter to use multiple cores
#8To be a bit pedantic: the multiprocess module works fine for multiple cores in regular cpython, I actually used it for a little graphics program I wrote. Nice to see the GIL gone for multithreading though :)
It isn't pedantic just irrelevant. Using multiprocess is an entirely different beast than using threads. All shared values have to explicitly allocated that way in shared memory. Since there is a separate interpreter for each process there is no problem with internal state getting messed up. Threads are just a harder problem for python and one which has never seen an effective solution.[1] That is why this kind of wo…
Using multiprocess is different from using threads. But that doesn't make it irrelevant, because it is a completely functional, already-existing way of using multiple cores.
Re: First Python 2.7 interpreter to use multiple cores
#9I'm not following this. Does it mean the GIL is finally gone?
That is the goal of the effort (see the pypy blog for the history of the effort, http://morepypy.blogspot.com ). But as noted in TFA, it is currently very, very slow, so no the problem is not solved, creating a very very slow GIL-free interpreter has been possible for a long time, just... very very slow so not acceptable.
Also it's not CPython. Technically, if you want the GIL to be gone on Python in general you can already use IronPyton or Jython which do not use a global interpreter lock.
Re: First Python 2.7 interpreter to use multiple cores
#10I'm not following this. Does it mean the GIL is finally gone?
It is a branch of PyPy (a fast and compliant Python 2.7 interpreter) that behaves as if the GIL was still there — statements are always run atomically, so are the new with atomic blocks — but will actually run multithreaded programs on multiple cores (as long as contention is low). It will also provide a transaction module (implemented with threads and atomic blocks) so that existing reactors can be ported, allowing coroutine-based programs to run on multiple cores too. This isn't faster at the moment, that will become the focus once this is stable.
Current documentation: https://bitbucket.org/pypy/pypy/raw/stm-thread/pypy/doc/stm....
The plan: http://pypy.org/tmdonate.html