Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

151–160 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#151

Post by myusernameisok »

shwouchk wrote: post #72
myusernameisok wrote: post #8

Earlier quoted context omitted.

It's the same issue with Python. AFAIK there are a number of Python libraries that are not thread-safe, and the GIL prevents them from being an issue.

GIL does not help thread safety in application code (and external libraries), just in the VM.

I'm fairly certain you're incorrect. With the GIL you don't have to lock shared memory because the assumption is that only one thread will be running at a time. For example shared data structures won't be changed while being being read/written to by multiple threads, because only one thread is actually running.

Re: Let's Remove the Global Interpreter Lock

#152

Post by cjhanks »

devwastaken wrote: post #13

This would be great if it means we can run the C portions of Python in threads without performance hits. I recently started a little project that is a cross-platform GUI for batch bzip2 compression, and Python did it quite well with its built-in bzip2 module. But, once I tried to do it parallel, the performance impacts of GIL were obvious. Yes, you can work around that with multi-process, but I'd rather not be spammi…

That doesn't seem quite right. C extensions can release the GIL and still continue running. So long as they are not operating on Python objects directly it is safe.

Re: Let's Remove the Global Interpreter Lock

#153

Post by carapace »

crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

I can't tell you how happy I was to see your comment at the top of this discussion.

Relevant: "Python is Only Slow If You Use it Wrong" http://apenwarr.ca/diary/2011-10-pycodeconf-apenwarr.pdf

Re: Let's Remove the Global Interpreter Lock

#154

Post by weberc2 »

dom0 wrote: post #131
weberc2 wrote: post #127

Earlier quoted context omitted.

> If you need to close that gap between the performance of multiprocessing, and multithreading, then you probably shouldn't be using Python, or any language of the same shape, in the first place. Unfortunately, these performance concerns often manifest well after the "rewrite it in a different language" date has expired. There are a lot of people in that boat, and they need better options. > There is one other option…

> There are a lot of people in that boat, and they need better options. Land isn't coming to you, folks, you must start rowing if you want to get there. Rewrite bit for bit. Module for module. Package for package.

Sounds like you're saying this is infeasible; care to explain why?

Re: Let's Remove the Global Interpreter Lock

#155

Post by sbeckeriv »

crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

I would like to know more.

Re: Let's Remove the Global Interpreter Lock

#156

Post by xenadu02 »

gshulegaard wrote: post #146

I feel like the GIL is, at this point, Python's most infamous attribute. For a long time I thought it was also the biggest flaw with Python...but over time I care less and less about it. I think the first thing to realize is that single-threaded performance is often significantly better with the GIL than without it. I think Larry Hasting's first Gilectomy talk was extremely insightful (about the GIL in general and ab…

Many projects have solved this problem with dual compilation modes and provide two binaries the user can select from at runtime.

Eliminating the GIL doesn't have to mean actually eliminating it. You could certainly have #defines and/or alternate implementations that make the fine-grained locks no-ops when compiling in GIL mode. Conversely make the GIL a no-op in multithreaded mode.

Re: Let's Remove the Global Interpreter Lock

#157

Post by dguaraglia »

crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

Whoa, thanks for bringing up MPI2, you might have just saved me a lot of painstaking development with the mmap and multiprocessing libraries.

Re: Let's Remove the Global Interpreter Lock

#158

Post by funkymike »

weberc2 wrote: post #154
dom0 wrote: post #131

Earlier quoted context omitted.

> There are a lot of people in that boat, and they need better options. Land isn't coming to you, folks, you must start rowing if you want to get there. Rewrite bit for bit. Module for module. Package for package.

Sounds like you're saying this is infeasible; care to explain why?

I took it to mean that it is feasible. Instead of saying "well we used the wrong language, I guess we're screwed," you rewrite one component at a time, piece by piece, until the whole has been replaced.

This is the approach I try to use myself. It's nearly impossible to replace an entire system all at once. But replacing one part at a time is doable and you can see the improvements much sooner.

Re: Let's Remove the Global Interpreter Lock

#159

Post by nine_k »

rlander wrote: post #126
nine_k wrote: post #78

Earlier quoted context omitted.

If you want other multiplatform, open-source, highly parallel languages with nice syntax and quick turnaround, we already have a few, like Elixir, Racket, or, well, even ES6. Much of the Python's appeal is in its huge, colossal, powerful ecosystem, with modules for everything, and things like numpy or tensorflow using it as the high-level interface language. Not breaking this is probably more important for success th…

I don't mean to be pedantic, but please explain how ES6 is a "highly parallel" language.

You have web workers, generators, all the async stuff, futures and promises — plenty enough from the language perspective. Maybe node.js does not happen to be multi-threaded, but it's not about the language.

Re: Let's Remove the Global Interpreter Lock

#160

Post by xenadu02 »

fijal wrote: post #83
ryanjl wrote: post #80

Earlier quoted context omitted.

Jython and Iron Python lack the GIL. It's just an implementation detail of the underlying VM. There's nothing in the language itself which requires a GIL.

C API... You can argue it's not a part of the language, but PyPy was forced to support it at the end

IronPython interoperates with a whole host of C and C++ code. I'm not sure why this would matter?

The initial implementation may need to assume single-threaded C interface support and take a global lock but it wouldn't be a stretch to have these things declare they are multithread aware and relax that restriction.

Forgive me but most of these objections seem like post-hoc rationalizations. The first step is deciding to support a GIL-less multithreaded mode. After that, solve the problems one step at a time.

It is amazing how many times accomplishing "magic" boils down to:

1. Decide we're going to solve this problem. 2. Iterate toward the solution in manageable steps.

#1 is by far the most difficult aspect :)

Post reply on HN