Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

191–200 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#191

Post by zerkten »

peterkelly wrote: post #183
Animats wrote: post #173

Earlier quoted context omitted.

Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload. "Multiprocessing" is…

If CPU load is an issue, why would you be using an interpreter in the first place?

It seems like people never make this assessment, or use the GIL argument to put interpreted languages down. I personally run into I/O bound problems way more often than CPU bound ones. That said, I'm mainly doing things in the realm of a Python web developer. Scientists probably hit CPU bound problems more often with Python, but seem to drop down to C/C++ extensions without needing to complain about the problems.

Re: Let's Remove the Global Interpreter Lock

#192

Post by optimusclimb »

omarforgotpwd wrote: post #141
optimusclimb wrote: post #134

Earlier quoted context omitted.

Except when your use case requires a massive shared data cache that needs to be atomically updated.

Redis could help. Obviously not perfect for every use case but covers many of them.

It doesn't if you need to manage atomic data across the processes, as there's no way to lock and block the other cache consumers (think the data you need to handle cache evictions, etc.)

Also, you're describing multiple python processes + an extra server (redis) process - as a "simpler" solution for the limitation that Python doesn't do multi-threads well.

Of course there are a ton of use cases out there where you can scale in other ways, but threads and shared memory exist for a reason - there's no reason not to call a spade a spade and say the GIL is still a limitation.

Re: Let's Remove the Global Interpreter Lock

#193

Post by cool-RR »

Python's GIL issue is like the Israeli-Palestinian conflict.

1. People like to talk about it a lot, complain about it and say their opinion of what should be done with it.

2. It's not likely to be resolved for years to come.

3. In the end, the problem has very little effect on people's lives, much much less than the amount of hype around the issue.

Re: Let's Remove the Global Interpreter Lock

#194

Post by khedoros1 »

peterkelly wrote: post #183
Animats wrote: post #173

Earlier quoted context omitted.

Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload. "Multiprocessing" is…

If CPU load is an issue, why would you be using an interpreter in the first place?

A lot of the heavy lifting is done through calls to C libraries anyhow, with Python just being a convenient way to pass the data around.

Re: Let's Remove the Global Interpreter Lock

#195

Post by optimusclimb »

abecedarius wrote: post #145
optimusclimb wrote: post #134

Earlier quoted context omitted.

Except when your use case requires a massive shared data cache that needs to be atomically updated.

I think it's ok to not write everything in Python, and this is a long way from the top of my problems with it.

Of course it is - and that's what people do. The reason for the parent article is that there ARE people that would like to continue to use Python the language, and their existing source code/libraries, but would like not to deal with the GIL. Just because it is not a priority for you doesn't mean it isn't for others.

Re: Let's Remove the Global Interpreter Lock

#196

Post by sevensor »

dguaraglia wrote: post #186
sevensor wrote: post #33

Earlier quoted context omitted.

With threading, all of your threads can refer to the same objects. Multiprocessing means you have multiple interpreters running. That means no shared memory, and communication over pretty slow queues. I've definitely wanted to have multithreaded Python programs where all threads referred to the same large read-only data structure. But I can't do this because of the GIL. I mean, I can, but it's pointless. I can't do t…

Yeah, sharing memory between processes is a very delicate ballet to perform. That said, sharing a read-only piece of data is way simpler than you'd expect, depending on size and your forking chain. The documentation could do a better job of explaining the nuances and provide more examples.

Care to elaborate? All I've seen in the docs is how to share arrays or C structures between processes. It would take a substantial rewrite to use either. Is there some kind of CoW mechanism I'm missing?

Re: Let's Remove the Global Interpreter Lock

#197

Post by ryanjl »

xenadu02 wrote: post #160
fijal wrote: post #83

Earlier quoted context omitted.

C API... You can argue it's not a part of the language, but PyPy was forced to support it at the end

IronPython interoperates with a whole host of C and C++ code. I'm not sure why this would matter? The initial implementation may need to assume single-threaded C interface support and take a global lock but it wouldn't be a stretch to have these things declare they are multithread aware and relax that restriction. Forgive me but most of these objections seem like post-hoc rationalizations. The first step is deciding…

Jython uses the JNI and IronPython does it through C++/CLI, neither of them support the CPython extension interface, meaning the C modules aren't compatible. Because of this, Jython and IronPython inherit the interface properties of their respective VMs and they can remain thread safe without the GIL.

Re: Let's Remove the Global Interpreter Lock

#200

Post by JulianWasTaken »

sandGorgon wrote: post #81
JulianWasTaken wrote: post #69

Earlier quoted context omitted.

Certainly! It's a bit hard to answer some of those questions because it's been so long since we've run CPython, and also because we've now got ~10 apps or so that run on PyPy. Initially memory tradeoff was definitely significant, somewhere around 40% or so -- it's going to vary across applications though certainly, and in a lot of cases I'm a bit happy our memory usage went up because it forces us more towards "nicer…

are you using any math packages like Numpy/Pandas or Opencv ?

Not in any production workloads.

They do work these days in PyPy though, so I'd feel comfortable doing so if we did, although I'd probably feel just as comfortable writing whatever numerics in pure-Python too unless it was stuff that already existed easily elsewhere.

On a personal note I've played with OpenCV as well (and done so with PyPy to do some real-time facial analysis on a video stream), but yeah also not for $PRODUCTION_WORK.

Post reply on HN