Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

121–130 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#121

Post by pslam »

metalliqaz wrote: post #112
crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Absolutely agree. Almost all tasks will perform very well when using multiprocessing. It also has a nice side-effect of steering you towards explicitly coding data flows without fine-grained sharing.

If you need to close that gap between the performance of multiprocessing, and multithreading, then you probably shouldn't be using Python, or any language of the same shape, in the first place.

There is one other option I'd like to see: multiprocessing style, but with multiple Python interpreter instances in the same process — one per thread. There would still be the hard delineation of data boundaries between instances, but less overhead for pushing data between them.

Re: Let's Remove the Global Interpreter Lock

#122

Post by rguillebert »

taeric wrote: post #90
rguillebert wrote: post #79

Earlier quoted context omitted.

It's not the PyPy developers' job to make every Python library threadsafe, people writing libraries will have to make their code threadsafe, like in every other language.

There is a clear difference here, though. Making a change that could lead to poorly written libraries now being broken is clearly the fault of the change. Userspace for these libraries is defined by how it is, not how it was intended. (And really, was it intended to be dangerous in this way?)

Then just use the version of PyPy with a GIL?

Re: Let's Remove the Global Interpreter Lock

#123

Post by sqeaky »

kllrnohj wrote: post #116
dec0dedab0de wrote: post #12

Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…

#1 & #2: Consumer CPUs are now pushing 16 cores & 32 threads. Python is limited to ~1/20th of what a single box is capable of. That's a pretty big bottleneck. #4: Even if you're just talking message passing sending a message between threads is in the 10s of nanoseconds while between processes is 10s of microseconds. That's a ~1000x slowdown on core communication. Given that CPU cores are not getting any faster, that'…

You are right about the majority of what you said, but I am pedantically picking on one point. CPU cores are getting faster, but they aren't doing it with clock speed, they are dispatching more instructions per cycle or otherwise making the work faster.

Re: Let's Remove the Global Interpreter Lock

#124

Post by bb101 »

metalliqaz wrote: post #112
crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

I can see how using multiprocessing trumps threads for smaller programs. However it can become memory inefficient to have larger programs running in multiple processes, especially on servers with less resources.

Re: Let's Remove the Global Interpreter Lock

#125

Post by metalliqaz »

pslam wrote: post #121
metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Absolutely agree. Almost all tasks will perform very well when using multiprocessing. It also has a nice side-effect of steering you towards explicitly coding data flows without fine-grained sharing. If you need to close that gap between the performance of multiprocessing, and multithreading, then you probably shouldn't be using Python, or any language of the same shape, in the first place. There is one other option…

What would that get you?

Re: Let's Remove the Global Interpreter Lock

#126

Post by rlander »

nine_k wrote: post #78
seunosewa wrote: post #53

The ideal solution is for someone to design a new programming language that is as similar to Python as possible without requiring a global lock. Rarely used features that make it hard to parallelize Python would be dropped. STM might be built into the language instead of being hacked into one implementation, etc.

If you want other multiplatform, open-source, highly parallel languages with nice syntax and quick turnaround, we already have a few, like Elixir, Racket, or, well, even ES6. Much of the Python's appeal is in its huge, colossal, powerful ecosystem, with modules for everything, and things like numpy or tensorflow using it as the high-level interface language. Not breaking this is probably more important for success th…

I don't mean to be pedantic, but please explain how ES6 is a "highly parallel" language.

Re: Let's Remove the Global Interpreter Lock

#127

Post by weberc2 »

pslam wrote: post #121
metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Absolutely agree. Almost all tasks will perform very well when using multiprocessing. It also has a nice side-effect of steering you towards explicitly coding data flows without fine-grained sharing. If you need to close that gap between the performance of multiprocessing, and multithreading, then you probably shouldn't be using Python, or any language of the same shape, in the first place. There is one other option…

> If you need to close that gap between the performance of multiprocessing, and multithreading, then you probably shouldn't be using Python, or any language of the same shape, in the first place.

Unfortunately, these performance concerns often manifest well after the "rewrite it in a different language" date has expired. There are a lot of people in that boat, and they need better options.

> There is one other option I'd like to see: multiprocessing style, but with multiple Python interpreter instances in the same process — one per thread. There would still be the hard delineation of data boundaries between instances, but less overhead for pushing data between them.

If I understand correctly, the article discusses this ("subinterpreters"), but claims that there is no advantage to this approach vs multiprocessing. Presumably any overhead savings are eaten by GIL contention or some such?

Re: Let's Remove the Global Interpreter Lock

#128

Post by jononor »

kllrnohj wrote: post #116
dec0dedab0de wrote: post #12

Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…

#1 & #2: Consumer CPUs are now pushing 16 cores & 32 threads. Python is limited to ~1/20th of what a single box is capable of. That's a pretty big bottleneck. #4: Even if you're just talking message passing sending a message between threads is in the 10s of nanoseconds while between processes is 10s of microseconds. That's a ~1000x slowdown on core communication. Given that CPU cores are not getting any faster, that'…

The efficiency hit is very dependent on how large your computation chunks are. If the computation per message batch is on order of 100 ms, it would be <10% loss.

Re: Let's Remove the Global Interpreter Lock

#129

Post by ehsankia »

dTal wrote: post #87
rburhum wrote: post #68

Earlier quoted context omitted.

So basically recreate an entire language and library ecosystem because there is one feature that is less than ideal? I hope you realize why a better approach may be to reengineer that one component...

Python has many less-than-ideal features. Do you think we finally got it right, that we will use Python forever, and that the library work of the past decade or so is irreplaceable? "Is it possible that software is not like anything else, that it is meant to be discarded: that the whole point is to see it as a soap bubble?" -- Alan Perlis

Just to go from py2 to py3, which was relatively a MUCH smaller change than a whole new language, it's taken a decade and it's still far from over. I don't see how a whole new language would be any better. And it's not like there's a lack of new languages popping up left and right. There's a reason most of them just die out. It's insanely hard to gain critical mass unless you have a huge backer, like a whole organization or company using the language.

Re: Let's Remove the Global Interpreter Lock

#130

Post by andreasgonewild »

I just can't stop thinking that somewhere along the line one of the Guidos should have reacted to handing out global locks left and right. I mean, as long as its only you and your friends using it. But once it starts spreading, these are the kind of issues that need to be kicked out of the way asap. Lock granularity affects the entire design of client code, reducing it basically means rewriting everything.

Ah well, at least it serves as a warning sign for budding language composers as myself. Snabel did full threading before it walked or talked:

https://github.com/andreas-gone-wild/snackis/blob/master/sna...

And to any Pythoneers with sore toes out there: pick a better language or learn to live with it, down-voting me will do nothing to solve your problems. It's a tool, we're supposed to pick the best one for the job; not decide on one for life and defend it to death. Imagine what could happen if language communities started working together rather than competing. There is no price to be won, we're all being taken for a ride.

Post reply on HN