Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

First Python 2.7 interpreter to use multiple cores

mail.python.org

Post reply on HN
Print view View original

31–40 of 42 posts

Re: First Python 2.7 interpreter to use multiple cores

#31

Post by willvarfar »

Peaker wrote: post #29
j_baker wrote: post #24

Earlier quoted context omitted.

To be even more pedantic, python already runs on multiple cores. They poll the GIL until they can run. I believe some version of Python 3 fixes this behavior so they don't have to poll the GIL.

The GIL is not a spinlock. Since only one core gets the GIL at a time, it's still just using one core (even if that one core is being switched around). The GIL is released during the execution of some C code, so another thread may execute Python code in the meantime, but that's probably besides the point too.

As we are in the sub-thread, we can point out that actually a Python process can use more than one core at the same time, just trying to acquire the GIL.

This is how Python programs can show up using more than 100% CPU in `top`.

http://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2010-und... covers this

Re: First Python 2.7 interpreter to use multiple cores

#32

Post by fijal »

dbecker wrote: post #26

This is incredibly interesting, and the people working on it are incredibly smart. But I think it's a slight oversell to call this a python 2.7 interpreter. For instance, most scientific computing code that runs in cpython 2.7 won't run on pypy. It's impressive from a computer science point of view. But, pypy has a ways to go before I'd call it a python interpreter without adding qualifications.

PyPy is officially sanctioned as python 2.7 interpreter. Cannot run scientific code does not rule it out. This is more a problem on the scientific code being too implementation specific (for good reasons though). This is known to be a problem to a lot of people (including me), however it does not rule pypy as a python 2.7 interpreter (without qualifications).

Re: First Python 2.7 interpreter to use multiple cores

#33

Post by lloeki »

dbecker wrote: post #26

This is incredibly interesting, and the people working on it are incredibly smart. But I think it's a slight oversell to call this a python 2.7 interpreter. For instance, most scientific computing code that runs in cpython 2.7 won't run on pypy. It's impressive from a computer science point of view. But, pypy has a ways to go before I'd call it a python interpreter without adding qualifications.

> most scientific computing code that runs in cpython 2.7 won't run on pypy

Pure python? or stuff written in/depending on C modules like NumPy? PyPy is supposed to be 2.7 compliant if pure python, while C extension support is still experimental.

Re: First Python 2.7 interpreter to use multiple cores

#34

Post by exDM69 »

jandrewrogers wrote: post #14
freyrs3 wrote: post #13

Earlier quoted context omitted.

Depends on what you're doing. Shoving a numpy array with 10^7 elements over the wire isn't going to scale nicely at all. There are still a lot of computing scenarios having shared memory is the best approach.

You do not need threads for shared memory. I know of quite a few high-performance software systems that use process isolation and yet leverage shared memory via mechanisms like mmap() for IPC.

A context switch from one thread to another is less expensive than from one process to another. And there are other advantages to running in a shared memory address space. It tends to make some things complicated, but if getting the most out of your CPUs is the goal, threads are the way to go.

Re: First Python 2.7 interpreter to use multiple cores

#35

Post by andreasvc »

sp332 wrote: post #5

To be a bit pedantic: the multiprocess module works fine for multiple cores in regular cpython, I actually used it for a little graphics program I wrote. Nice to see the GIL gone for multithreading though :)

I wouldn't call it fine. Try hitting ctrl-c when running a process Pool. Worse, it's not just ctrl-c, but any abnormal termination such as a segfault as well which will screw everything up. The bug is acknowledged, but it seems they won't fix it. In my opinion that makes it better to just leave it out of the standard library.

Re: First Python 2.7 interpreter to use multiple cores

#36

Post by d0mine »

exDM69 wrote: post #34
jandrewrogers wrote: post #14

Earlier quoted context omitted.

You do not need threads for shared memory. I know of quite a few high-performance software systems that use process isolation and yet leverage shared memory via mechanisms like mmap() for IPC.

A context switch from one thread to another is less expensive than from one process to another. And there are other advantages to running in a shared memory address space. It tends to make some things complicated, but if getting the most out of your CPUs is the goal, threads are the way to go.

On some systems (Linux) there is no difference once threads/processes are initialized. Simple benchmark confirms it (see my comment about mmap).

Re: First Python 2.7 interpreter to use multiple cores

#37

Post by jlgreco »

Tobu wrote: post #20
jlgreco wrote: post #18

Earlier quoted context omitted.

In my humble opinion, the greatest problem with all of them is that there are so many. Most of them are rather fine I think, but it can just be a serious pain in the ass getting your head around all of them.

I've only needed one (Twisted, which is actually slightly outside the space occupied by gevent et al). You can't stop people from building more, but feel free to bet on a popular project if you don't want to deal with choice.

In a world where I only work with my own code, that works. Unfortunately that is not my reality.

Re: First Python 2.7 interpreter to use multiple cores

#38

Post by chimeracoder »

sanxiyn wrote: post #27
chimeracoder wrote: post #25

My understanding of the GIL is that its inclusion was justified by the fact that removing it would speed up multithreaded programs at the cost of slowing down single-threaded programs (the latter being more common than the former). Given that, and given that the patches for removing the GIL were submitted for an earlier version a while back (2.4 or before, I believe), is it unfeasible/impossible to design an interpre…

Detection will be non-trivial. On the other hand, it could be a simple command line option, like specifying the number of cores you want your program to run on.

Not really - we already have __future__ imports, which must be declared at the top of the file before any other modules.

I think this could be achieved in a simple way in a similar fashion - some sort of a straightforward 'multithreaded' pragma shouldn't be too much of a burden.

Re: First Python 2.7 interpreter to use multiple cores

#39

Post by ori_b »

sanxiyn wrote: post #30
ori_b wrote: post #28

Earlier quoted context omitted.

I'm not sure about that. If you hook thread creation into invalidating the code that was was JITted without STM, you might be able to handle it reasonably well. Of course, the slow path would just use STM all the time.

> ... invalidating the code that was JITted ... I mean, that is non-trivial.

Maybe. If you determine that you only need to do it on spawning the first thread, and never go back, then you can probably spare the expense of throwing out ALL of the jit cache and starting from nothing.

Re: First Python 2.7 interpreter to use multiple cores

#40

Post by sanxiyn »

chimeracoder wrote: post #38
sanxiyn wrote: post #27

Earlier quoted context omitted.

Detection will be non-trivial. On the other hand, it could be a simple command line option, like specifying the number of cores you want your program to run on.

Not really - we already have __future__ imports, which must be declared at the top of the file before any other modules. I think this could be achieved in a simple way in a similar fashion - some sort of a straightforward 'multithreaded' pragma shouldn't be too much of a burden.

__future__ is per-module, so while it is good for syntax changes, I think it is inappropriate for runtime pragmas.
Post reply on HN