Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

251–260 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#251

Post by gnaritas »

slantedview wrote: post #226
metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

> multiprocessing instead of multithreading There's a reason threads exist.

Those reasons aren't what they used to be, resources aren't nearly as limited these days and we now have the hindsight to see that threads lead to very buggy code due to shared state. Processes are better.

Re: Let's Remove the Global Interpreter Lock

#252

Post by riri-au »

eslaught wrote: post #161

The comments here are missing a massive use case: shared memory. Shared memory isn't just about programmer convenience. It's about using a machine's memory resources more effectively. Yes, shared memory is available in multi-processing, but it doesn't necessarily interact well with existing codes. I've been working on adding Python support to Legion [1], a task-based runtime system for HPC. Legion wants to manage sha…

Agreed, shared programming is an immensely useful feature for numerical programming, including data science and machine learning. Lots of people will say that those should be written in C++, but I think the rise of machine learning & data science in high level languages argues against their point.

Re: Let's Remove the Global Interpreter Lock

#253

Post by jzwinck »

sambe wrote: post #223
detroitcoder wrote: post #210

Earlier quoted context omitted.

^This. It is a very common usecase for applications I work with to create a very large in memory read-only pd dataframe and then put a flask interface to operations on that dataframe using gunicorn and expose as an API. If I use async workers, the dataframe operations are bound by GIL restraints. If I use sync workers, each process needs a copy of the pd dataframe which the server cannot handle (I have never seen pre…

I rarely hear people complain about genuine use-cases but this would seem to be one. However, aren't most/all of the dataframe operations done in C extensions in these cases?

While a lot of NumPy is C and Fortran, Pandas is mostly pure Python and some Cython. And mostly it does not release the GIL.

You often end up having to implement your own C extensions or use Numba for the core of your processing. Even with BLAS enabled, NumPy has almost zero intrinsic parallelism, np.dot() being the notable exception which releases the GIL and uses multicore by itself.

Re: Let's Remove the Global Interpreter Lock

#254

Post by throwaway613834 »

jzwinck wrote: post #253
sambe wrote: post #223

Earlier quoted context omitted.

I rarely hear people complain about genuine use-cases but this would seem to be one. However, aren't most/all of the dataframe operations done in C extensions in these cases?

While a lot of NumPy is C and Fortran, Pandas is mostly pure Python and some Cython. And mostly it does not release the GIL. You often end up having to implement your own C extensions or use Numba for the core of your processing. Even with BLAS enabled, NumPy has almost zero intrinsic parallelism, np.dot() being the notable exception which releases the GIL and uses multicore by itself.

> Even with BLAS enabled, NumPy has almost zero intrinsic parallelism, np.dot() being the notable exception which releases the GIL and uses multicore by itself.

Is there any sort of list (comprehensive or otherwise) that denotes which NumPy functions are parallelism-friendly? I mean this whether it's in terms of releasing the GIL, in terms of SIMD support, or in terms of being multi-core.

Re: Let's Remove the Global Interpreter Lock

#255

Post by throwaway613834 »

peterkelly wrote: post #183
Animats wrote: post #173

Earlier quoted context omitted.

Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload. "Multiprocessing" is…

If CPU load is an issue, why would you be using an interpreter in the first place?

> If CPU load is an issue, why would you be using an interpreter in the first place?

You're basically asking why NumPy, SciPy, Numba, etc. even exist.

They exist because Python is ridiculously fast to develop in compared to, say, C++.

Re: Let's Remove the Global Interpreter Lock

#256

Post by m_mueller »

njharman wrote: post #164
m_mueller wrote: post #32

Earlier quoted context omitted.

In my five years of python I've run up against this boundary at least once. In your list I would * take out #2. if something can make use of multiple nodes it can usually make even better use of multi-core parallelization (which affects both computational and memory bandwidth performance). multi-node comes with a much higher communications overhead, so there's a relatively wide range of applications that scale well o…

Once (or a few times) in 5 years puts this problem into the "not worth(ROI) solving" bucket for me. Those few times, put down the hammer and use some other tool for those not naillike jobs.

Here's the thing: Python, especially 3.6, is such a well rounded language that all other major limitations have IMO been solved already. In my view the GIL is the main one left, and reason to pause and think whether python is a good idea at the start of a project. Removing it is therefore worth it, and would also give a nice additional incentive for everyone to switch to python 3.x, so we don't have to keep on maintaining 2.7 with the same code (i.e. the worst of both worlds).

Re: Let's Remove the Global Interpreter Lock

#257

Post by abecedarius »

jhayward wrote: post #211
optimusclimb wrote: post #134

Earlier quoted context omitted.

Except when your use case requires a massive shared data cache that needs to be atomically updated.

> Except when your use case requires a massive shared data cache that needs to be atomically updated You can delete the last 6 words. Anything where multiple processes would have to read in/acquire a massive dataset to do some independent work qualifies. For instance, running some number (e.g., hundreds to hundreds of thousands) of analytical or statistical tests over a set to pick parameters, etc.

Read-only shared memory can cover that. Python's ref counting does make it a nuisance: you can't share it as a Python object graph.

Re: Let's Remove the Global Interpreter Lock

#258

Post by stuaxo »

I liked "STM" as a big-idea approach, but can see how going the traditional way may bear fruit more quickly.

Could the experience gained this way (and by other projects such as the gilectemy) help with a future STM attempt ?

I wonder if we need better hardware for STM to work well too.

Re: Let's Remove the Global Interpreter Lock

#259

Post by zeptomu »

detroitcoder wrote: post #210
eslaught wrote: post #161

The comments here are missing a massive use case: shared memory. Shared memory isn't just about programmer convenience. It's about using a machine's memory resources more effectively. Yes, shared memory is available in multi-processing, but it doesn't necessarily interact well with existing codes. I've been working on adding Python support to Legion [1], a task-based runtime system for HPC. Legion wants to manage sha…

^This. It is a very common usecase for applications I work with to create a very large in memory read-only pd dataframe and then put a flask interface to operations on that dataframe using gunicorn and expose as an API. If I use async workers, the dataframe operations are bound by GIL restraints. If I use sync workers, each process needs a copy of the pd dataframe which the server cannot handle (I have never seen pre…

> create a very large in memory read-only pd dataframe and then put a flask interface to operations on that dataframe using gunicorn and expose as an API. [...]

May I ask what you consider large memory - MByte, GByte, TByte? The simplest solution is to store it as a blob on a SSD, and read it via simple file IO or put it into a DB. But I assume this was too slow, so it would be interesting to go into more details.

In the end you can do shared memory with multiprocessing in Python, which - I have to admit - requires some setup and bookkeeping work.

Re: Let's Remove the Global Interpreter Lock

#260

Post by cjbillington »

This seems like a good place to spruik something I made, a Python package for profiling how much the GIL is held:

https://github.com/chrisjbillington/gil_load

In my experience, the GIL is not held for nearly as high a proportion of the time as people think it is, because properly written C extensions and blocking io always releases the GIL. So long as the proportion of time the GIL is held is not approaching 100%, then you can still get gains from threading. This is almost always the case in numerically heavy code that uses numpy or scipy, since the extensions release the GIL. Threads work almost just as well at speeding up this code as in any GIL-free interpreter.

And usually long before you consider multithreaded code, you'll want to move the bottlenecks of your code over into Cython or something, since that can give speedup factors much larger than multithreading. In which case all you need is a "with nogil:" around the the meaty bit of you Cython code, and then it too will be able to get speedups from multithreading.

Post reply on HN