Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

First Python 2.7 interpreter to use multiple cores

mail.python.org

Post reply on HN
Print view View original

21–30 of 42 posts

Re: First Python 2.7 interpreter to use multiple cores

#21

Post by d0mine »

seunosewa wrote: post #16
jandrewrogers wrote: post #14

Earlier quoted context omitted.

You do not need threads for shared memory. I know of quite a few high-performance software systems that use process isolation and yet leverage shared memory via mechanisms like mmap() for IPC.

Good point, but shared memory via mmap isn't very Pythonic since you have to manage the shared memory manually, you can't allocate Python objects on the shared region, etc.

You don't need to use mmap explicitly. multiprocessing.Array and numpy arrays can share the same memory that can be used from different processes e.g., http://stackoverflow.com/a/9849971

Re: First Python 2.7 interpreter to use multiple cores

#22

Post by viraptor »

slurgfest wrote: post #11
tocomment wrote: post #2

I'm not following this. Does it mean the GIL is finally gone?

I will get excited when it actually works out to be faster than interpreters with the GIL. I'm a bit disappointed that Python seems to have no interest in standardizing a lightweight-thread (greenlet, coroutine) interface that isn't an awful mess. yield is powerful but coroutine implementations with it are baroque and opaque. Java-style threading has a million gotchas and the interface is just a dog. I want to be abl…

Most likely it never will be faster than just GIL. It may be very close to the same performance, but nothing is likely to beat a "one lock every N thousands of instructions" approach when you have a single thread.

If it's within 5% though, I'll be happy - especially if the backend is possible to switch. It seems that it's possible now - so if you know you're running only in a single thread you could just choose the cut-down version.

Re: First Python 2.7 interpreter to use multiple cores

#24

Post by j_baker »

sp332 wrote: post #5

To be a bit pedantic: the multiprocess module works fine for multiple cores in regular cpython, I actually used it for a little graphics program I wrote. Nice to see the GIL gone for multithreading though :)

To be even more pedantic, python already runs on multiple cores. They poll the GIL until they can run. I believe some version of Python 3 fixes this behavior so they don't have to poll the GIL.

Re: First Python 2.7 interpreter to use multiple cores

#25

Post by chimeracoder »

My understanding of the GIL is that its inclusion was justified by the fact that removing it would speed up multithreaded programs at the cost of slowing down single-threaded programs (the latter being more common than the former).

Given that, and given that the patches for removing the GIL were submitted for an earlier version a while back (2.4 or before, I believe), is it unfeasible/impossible to design an interpreter that detects whether a program requires support for multiple threads and then act accordingly? Since this currently requires the inclusion of a module, it seems like it would be unambiguous.

Re: First Python 2.7 interpreter to use multiple cores

#26

Post by dbecker »

This is incredibly interesting, and the people working on it are incredibly smart.

But I think it's a slight oversell to call this a python 2.7 interpreter. For instance, most scientific computing code that runs in cpython 2.7 won't run on pypy.

It's impressive from a computer science point of view. But, pypy has a ways to go before I'd call it a python interpreter without adding qualifications.

Re: First Python 2.7 interpreter to use multiple cores

#27

Post by sanxiyn »

chimeracoder wrote: post #25

My understanding of the GIL is that its inclusion was justified by the fact that removing it would speed up multithreaded programs at the cost of slowing down single-threaded programs (the latter being more common than the former). Given that, and given that the patches for removing the GIL were submitted for an earlier version a while back (2.4 or before, I believe), is it unfeasible/impossible to design an interpre…

Detection will be non-trivial. On the other hand, it could be a simple command line option, like specifying the number of cores you want your program to run on.

Re: First Python 2.7 interpreter to use multiple cores

#28

Post by ori_b »

sanxiyn wrote: post #27
chimeracoder wrote: post #25

My understanding of the GIL is that its inclusion was justified by the fact that removing it would speed up multithreaded programs at the cost of slowing down single-threaded programs (the latter being more common than the former). Given that, and given that the patches for removing the GIL were submitted for an earlier version a while back (2.4 or before, I believe), is it unfeasible/impossible to design an interpre…

Detection will be non-trivial. On the other hand, it could be a simple command line option, like specifying the number of cores you want your program to run on.

I'm not sure about that. If you hook thread creation into invalidating the code that was was JITted without STM, you might be able to handle it reasonably well.

Of course, the slow path would just use STM all the time.

Re: First Python 2.7 interpreter to use multiple cores

#29

Post by Peaker »

j_baker wrote: post #24
sp332 wrote: post #5

To be a bit pedantic: the multiprocess module works fine for multiple cores in regular cpython, I actually used it for a little graphics program I wrote. Nice to see the GIL gone for multithreading though :)

To be even more pedantic, python already runs on multiple cores. They poll the GIL until they can run. I believe some version of Python 3 fixes this behavior so they don't have to poll the GIL.

The GIL is not a spinlock. Since only one core gets the GIL at a time, it's still just using one core (even if that one core is being switched around).

The GIL is released during the execution of some C code, so another thread may execute Python code in the meantime, but that's probably besides the point too.

Re: First Python 2.7 interpreter to use multiple cores

#30

Post by sanxiyn »

ori_b wrote: post #28
sanxiyn wrote: post #27

Earlier quoted context omitted.

Detection will be non-trivial. On the other hand, it could be a simple command line option, like specifying the number of cores you want your program to run on.

I'm not sure about that. If you hook thread creation into invalidating the code that was was JITted without STM, you might be able to handle it reasonably well. Of course, the slow path would just use STM all the time.

> ... invalidating the code that was JITted ...

I mean, that is non-trivial.

Post reply on HN