Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

201–210 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#201

Post by zeptomu »

crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

I am a heavy user of Python and its scientific libraries (numpy, etc.), and although I know about the GIL, I have to add, that for us (we do a lot of scientific code-prototyping to evaluate remote sensing processing methods) the GIL hasn't been a problem so far.

E.g. in the remote sensing and earth observation domain you can simply divide your problem (e.g. semantic segmentation) into (maybe over-lapping) subproblems (via e.g. tiling) and start separate processes for each image processing tool-chain.

Granted you may not utilize your resources to the full extent by only applying multiprocessing (and ignoring threading), but in my experience you can solve a lot of problems by simply applying map-reduce-like programs and optimizing for throughput.

Re: Let's Remove the Global Interpreter Lock

#202

Post by DigitalJack »

dsfyu404ed wrote: post #190
peterkelly wrote: post #183

Earlier quoted context omitted.

If CPU load is an issue, why would you be using an interpreter in the first place?

Just because those resources exist does not mean you get to park your 1997 Chevy Cavalier diagonal across three parking spaces. I'm not going to run your code on my server if your code uses resources so poorly that I can't run other things I want to run on my server.

You are getting downvotes, I suspect, because your comment makes no sense in the context of the post you replied to.

Perhaps you meant to reply to GP?

Re: Let's Remove the Global Interpreter Lock

#203

Post by vram22 »

ericfrederich wrote: post #198

Perhaps a little unrelated, I used the rpyc package to get Jython and CPython working together. In the end I was able to use Java libraries from CPython pretty much seamlessly.

You mean you used RPyC at both ends, on the CPython side and on the Jython side. Cool idea. I knew about RPyC but had not thought of using it in this way. And getting access to Java libraries by doing this, can be very useful, I can see.

Re: Let's Remove the Global Interpreter Lock

#204

Post by VectorLock »

gshulegaard wrote: post #146

I feel like the GIL is, at this point, Python's most infamous attribute. For a long time I thought it was also the biggest flaw with Python...but over time I care less and less about it. I think the first thing to realize is that single-threaded performance is often significantly better with the GIL than without it. I think Larry Hasting's first Gilectomy talk was extremely insightful (about the GIL in general and ab…

The GIL has been a much bigger problem for perception than it ever has been for performance. Python has lost more mindshare over it than anything else. The few machine cycles that were ever saved by moving away from it were far outweighed by the waste of human cycles.

Re: Let's Remove the Global Interpreter Lock

#205

Post by nine_k »

heavenlyblue wrote: post #177
nine_k wrote: post #159

Earlier quoted context omitted.

You have web workers, generators, all the async stuff, futures and promises — plenty enough from the language perspective. Maybe node.js does not happen to be multi-threaded, but it's not about the language.

By the same logic Python is multithreading-ready as well, since it's only a matter of it's major implementation why it doesn't support multithreading.

> By the same logic Python is multithreading-ready as well, since it's only a matter of it's major implementation why it doesn't support multithreading.

How web workers aren't threads? Browsers are more widely deployed than Node, even with the same V8 engine.

Re: Let's Remove the Global Interpreter Lock

#206

Post by zeptomu »

Animats wrote: post #173
metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload. "Multiprocessing" is…

See my other replay in this thread.

> Parallelizing your number crunching is probably not going to work very well. [...]

The question is, what exactly does "number crunching" mean? We do aerial imagery analysis, so image processing in essence, which I would classify as a "number crunching" problem. A common thing e.g. is to do a time-series analysis and you can simply start multiple (2, 4, ..., N with clusters, etc.) processes for each problem. Obviously this works because most methods are computation and/or memory heavy - the additional memory requirements and "overhead" of Python itself (IMHO people overestimate the weight of starting new processes instead of threads) is completely dwarfed by the requirements (memory and CPU) of the method itself.

Re: Let's Remove the Global Interpreter Lock

#207

Post by sametmax »

eslaught wrote: post #161

The comments here are missing a massive use case: shared memory. Shared memory isn't just about programmer convenience. It's about using a machine's memory resources more effectively. Yes, shared memory is available in multi-processing, but it doesn't necessarily interact well with existing codes. I've been working on adding Python support to Legion [1], a task-based runtime system for HPC. Legion wants to manage sha…

But then multi-interpreters would allow that, and the article discard it as a valid solution. I find it harsh. It seems much easier to implement, doesn't have the same serialization problem than multiprocessing has and allow to utilize all the CPUs. Yes it's not as good proper threads because you do have more overhead, but it's an order of magnitude better than what we currently have, while being way easier to do that getting rid of the GIL.

Too bad the current project is on hold.

Re: Let's Remove the Global Interpreter Lock

#208

Post by sametmax »

gshulegaard wrote: post #146

I feel like the GIL is, at this point, Python's most infamous attribute. For a long time I thought it was also the biggest flaw with Python...but over time I care less and less about it. I think the first thing to realize is that single-threaded performance is often significantly better with the GIL than without it. I think Larry Hasting's first Gilectomy talk was extremely insightful (about the GIL in general and ab…

Which is what multi-interpreters is a good solution. You keep the GIL and the benefits of it, but you loose the cost of serialization and can share memory.

Re: Let's Remove the Global Interpreter Lock

#209

Post by dekhn »

Animats wrote: post #173
metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload. "Multiprocessing" is…

The other issue with multiprocessing is that it requires the enclosing code to be pickleable, and many Python objects are not pickleable. For example, if I have a thread-safe RPC client and want to send thousands of RPCs using the client, I can't do that with multiprocessing (subprocess pool; threading pools work). RPC clients manage a TCP connection, if you use multiprocess you end up having to make many TCP connections.

Re: Let's Remove the Global Interpreter Lock

#210

Post by detroitcoder »

eslaught wrote: post #161

The comments here are missing a massive use case: shared memory. Shared memory isn't just about programmer convenience. It's about using a machine's memory resources more effectively. Yes, shared memory is available in multi-processing, but it doesn't necessarily interact well with existing codes. I've been working on adding Python support to Legion [1], a task-based runtime system for HPC. Legion wants to manage sha…

^This. It is a very common usecase for applications I work with to create a very large in memory read-only pd dataframe and then put a flask interface to operations on that dataframe using gunicorn and expose as an API. If I use async workers, the dataframe operations are bound by GIL restraints. If I use sync workers, each process needs a copy of the pd dataframe which the server cannot handle (I have never seen pre-fork shared memory work for this problem). I don't want to introduce another technology to solve this problem.
Post reply on HN