Let's Remove the Global Interpreter Lock • Hacker News Board

Re: Let's Remove the Global Interpreter Lock

#131

Post by dom0 » Mon, Aug 14, 2017, 5:00 PM UTC

weberc2 wrote: post #127

pslam wrote: post #121

Earlier quoted context omitted.

Absolutely agree. Almost all tasks will perform very well when using multiprocessing. It also has a nice side-effect of steering you towards explicitly coding data flows without fine-grained sharing. If you need to close that gap between the performance of multiprocessing, and multithreading, then you probably shouldn't be using Python, or any language of the same shape, in the first place. There is one other option…

> If you need to close that gap between the performance of multiprocessing, and multithreading, then you probably shouldn't be using Python, or any language of the same shape, in the first place. Unfortunately, these performance concerns often manifest well after the "rewrite it in a different language" date has expired. There are a lot of people in that boat, and they need better options. > There is one other option…

> There are a lot of people in that boat, and they need better options.

Land isn't coming to you, folks, you must start rowing if you want to get there.

Rewrite bit for bit. Module for module. Package for package.

Re: Let's Remove the Global Interpreter Lock

#132

Post by weberc2 » Mon, Aug 14, 2017, 5:01 PM UTC

seunosewa wrote: post #53

The ideal solution is for someone to design a new programming language that is as similar to Python as possible without requiring a global lock. Rarely used features that make it hard to parallelize Python would be dropped. STM might be built into the language instead of being hacked into one implementation, etc.

If we're talking about creating an entirely new language, I don't think the "ideal" is Python with a few tweaks; it's going to be radically different. The point is, you have to draw the line somewhere, and if you're going to build an entirely new language, you should probably address as many problems as you can; few will switch to an incrementally better Python (unless you can give strong compatibility guarantees, in which case it's arguably not a new language).

Re: Let's Remove the Global Interpreter Lock

#133

Post by [deleted] » Mon, Aug 14, 2017, 5:03 PM UTC

andreasgonewild wrote: post #130

I just can't stop thinking that somewhere along the line one of the Guidos should have reacted to handing out global locks left and right. I mean, as long as its only you and your friends using it. But once it starts spreading, these are the kind of issues that need to be kicked out of the way asap. Lock granularity affects the entire design of client code, reducing it basically means rewriting everything. Ah well, a…

[deleted]

Re: Let's Remove the Global Interpreter Lock

#134

Post by optimusclimb » Mon, Aug 14, 2017, 5:03 PM UTC

metalliqaz wrote: post #112

crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Except when your use case requires a massive shared data cache that needs to be atomically updated.

Re: Let's Remove the Global Interpreter Lock

#135

Post by sametmax » Mon, Aug 14, 2017, 5:05 PM UTC

kevingadd wrote: post #110

crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

The post addresses this strategy and describes why they consider it insufficient. You can already do this in Python, anyway.

Yes but they discard multiple interpreters as having no real advantages. This is dishonest since it should be able to share objects with much, much less overhead as multiprocessing, while allowing to use multiple CPU. It's not perfect, but honestly it seems like a very good deal for Python.

Re: Let's Remove the Global Interpreter Lock

#136

Post by zepolen » Mon, Aug 14, 2017, 5:05 PM UTC

optimusclimb wrote: post #134

metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Except when your use case requires a massive shared data cache that needs to be atomically updated.

Of which there are plenty well defined ones to do the job already, and as a plus they can communicate with any language not just Python.

Re: Let's Remove the Global Interpreter Lock

#137

Post by sametmax » Mon, Aug 14, 2017, 5:07 PM UTC

dec0dedab0de wrote: post #12

Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…

It's just a low hanging fruit for perfs from the dev point of view. It's nice and useful, just nowhere as needed as most people asking for it pretend it is.

Re: Let's Remove the Global Interpreter Lock

#138

Post by ddustin » Mon, Aug 14, 2017, 5:10 PM UTC

dec0dedab0de wrote: post #12

Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…

My god. This website is full of web script-kiddies.

I've run into this need at least a dozen times.

Re: Let's Remove the Global Interpreter Lock

#139

Post by btilly » Mon, Aug 14, 2017, 5:10 PM UTC

sevensor wrote: post #33

dec0dedab0de wrote: post #19

Earlier quoted context omitted.

in what was is it a pain that threading is not?

With threading, all of your threads can refer to the same objects. Multiprocessing means you have multiple interpreters running. That means no shared memory, and communication over pretty slow queues. I've definitely wanted to have multithreaded Python programs where all threads referred to the same large read-only data structure. But I can't do this because of the GIL. I mean, I can, but it's pointless. I can't do t…

And yet, if you could have what you want, would it actually be faster?

The costs of synchronizing mutable data between cores is surprisingly high. Any time your CPU thinks that the data that it has in its cache might not be what some other CPU has in its cache, the two have to coordinate what they are doing. And thanks to the fact that Python uses reference counting, data is constantly being changed even though you don't think that you're changing it.

Furthermore if you throw out the GIL for fine-grained locking, you then open up a world of potential problems such as deadlocks. Which look like "my program mysteriously froze". Life just got a lot more complicated.

It is easy to look at all of those cores and say, "I just want my program to use all of them!" But doing that and actually GETTING better performance is a lot trickier than it might seem.

Re: Let's Remove the Global Interpreter Lock

#140

Post by omarforgotpwd » Mon, Aug 14, 2017, 5:12 PM UTC

crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

Yes, multi-processing is much easier anyway. Not to mention how complicated, not backwards compatible and thorny trying to get rid of the GIL is...