Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

171–180 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#171

Post by neolefty »

njharman wrote: post #164
m_mueller wrote: post #32

Earlier quoted context omitted.

In my five years of python I've run up against this boundary at least once. In your list I would * take out #2. if something can make use of multiple nodes it can usually make even better use of multi-core parallelization (which affects both computational and memory bandwidth performance). multi-node comes with a much higher communications overhead, so there's a relatively wide range of applications that scale well o…

Once (or a few times) in 5 years puts this problem into the "not worth(ROI) solving" bucket for me. Those few times, put down the hammer and use some other tool for those not naillike jobs.

Certainly not worth it for one person to tackle the GIL, but a million people running into it a few times in 5 years, and I think it's economical.

Re: Let's Remove the Global Interpreter Lock

#172

Post by breatheoften »

kllrnohj wrote: post #116
dec0dedab0de wrote: post #12

Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…

#1 & #2: Consumer CPUs are now pushing 16 cores & 32 threads. Python is limited to ~1/20th of what a single box is capable of. That's a pretty big bottleneck. #4: Even if you're just talking message passing sending a message between threads is in the 10s of nanoseconds while between processes is 10s of microseconds. That's a ~1000x slowdown on core communication. Given that CPU cores are not getting any faster, that'…

Moving data between threads is only free to the extent that synchronization is free. Maybe you could say that moving immutable data between threads is free but I don't think you can say its free in general ... Doing so significantly undersells the complexity that comes with shared memory concurrency.

Re: Let's Remove the Global Interpreter Lock

#173

Post by Animats »

metalliqaz wrote: post #112
crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload.

"Multiprocessing" is useful when you have a lot of work to do concurrently and not too much data to pass between processes. I've used Python subprocesses that way. Parallelizing your number crunching is probably not going to work very well.

Re: Let's Remove the Global Interpreter Lock

#174

Post by rburhum »

dTal wrote: post #87
rburhum wrote: post #68

Earlier quoted context omitted.

So basically recreate an entire language and library ecosystem because there is one feature that is less than ideal? I hope you realize why a better approach may be to reengineer that one component...

Python has many less-than-ideal features. Do you think we finally got it right, that we will use Python forever, and that the library work of the past decade or so is irreplaceable? "Is it possible that software is not like anything else, that it is meant to be discarded: that the whole point is to see it as a soap bubble?" -- Alan Perlis

At the end of the day, the purpose of why we write software as software engineers is to solve real-world problems, not to have a perfect beautiful language. What you are describing is equivalent to doing an amputation when all you need is antibiotics.

There are several things I personally _hate_ about python, but there is a cost-benefit that comes from engineering new things. What new problems are we going to be able to solve by using a new language? If the answer is clear (e.g. imperative programming vs declarative/functional programming let you solve different kind of problems) then it makes sense to do. If certain constructs enable you to completely avoid a recurring mistake (e.g. garbage collection), then it may make sense.

But this?!?!? No man, you don't need a new language to fix this.

Re: Let's Remove the Global Interpreter Lock

#175

Post by jerf »

nine_k wrote: post #159
rlander wrote: post #126

Earlier quoted context omitted.

I don't mean to be pedantic, but please explain how ES6 is a "highly parallel" language.

You have web workers, generators, all the async stuff, futures and promises — plenty enough from the language perspective. Maybe node.js does not happen to be multi-threaded, but it's not about the language.

That's all true about Python, too, and it's been true since before Node existed. If Node were adequate, Python would be adequate too.

In fact, other than "run lots of Javascript", I'm not sure I can name a single thing Node did before Python.

Re: Let's Remove the Global Interpreter Lock

#176

Post by neolefty »

omarforgotpwd wrote: post #142
dec0dedab0de wrote: post #12

Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…

Motivation for removing the GIL is basically that when people hear about it they go "hmmm that doesn't sound good". Obviously many applications have been written in GIL languages and there aren't really many practical problems that can't be overcome easily.

I think it may be some Stockholm Syndrome -- people have worked very hard to get around the GIL, and they've come to expect its limitations and respect those solutions.

But I've never heard of someone asking for a GIL to be added to the JVM.

Re: Let's Remove the Global Interpreter Lock

#177

Post by heavenlyblue »

nine_k wrote: post #159
rlander wrote: post #126

Earlier quoted context omitted.

I don't mean to be pedantic, but please explain how ES6 is a "highly parallel" language.

You have web workers, generators, all the async stuff, futures and promises — plenty enough from the language perspective. Maybe node.js does not happen to be multi-threaded, but it's not about the language.

By the same logic Python is multithreading-ready as well, since it's only a matter of it's major implementation why it doesn't support multithreading.

Re: Let's Remove the Global Interpreter Lock

#178

Post by heavenlyblue »

wiremine wrote: post #58

> We estimate a total cost of $50k... Just looking at it from a financial perspective, having a great Python interpreter that doesn't have a GIL seems like a no brainer for $50,000, and it creates another reason why people should take a look at PyPy. Side note: if you haven't looked at PyPy, check it out, along with RPython https://rpython.readthedocs.io/en/latest/

Andrew Godwin had raised £17K out of £2.5K expected in order to implement (I believe excellent) Django migrations that are now part of the official repository: https://www.kickstarter.com/projects/andrewgodwin/schema-mig...

Neither do I think that raising $50K for Python interpreter would be an issue.

PS: I don't find Django an excellent ORM per se. On the other hand it's highly pragmatic, and their implementation of automatically-generated migrations have saved a good chunk of my time.

Re: Let's Remove the Global Interpreter Lock

#179

Post by vladf »

njharman wrote: post #167
vladf wrote: post #143

There seem to be a lot of naysayers in the comments about removing the GIL. Multiprocess parallelism isn't always appropriate, so I find this to be a very promising change that will definitely make me want to switch to PyPy. Here are the use cases I've found multiprocessing to be inappropriate: * High-contention parallel operations. Doing synchronization through a Manager (a separate IPC-based synchronizing broker pr…

> Multiprocess parallelism isn't always appropriate Using Python isn't always appropriate.

Are you saying that because a language is missing something, when considering a fix for that thing, the existence of other languages/solutions is an argument against that fix?

Re: Let's Remove the Global Interpreter Lock

#180

Post by orangejewce »

ams6110 wrote: post #109
smaddox wrote: post #106

Earlier quoted context omitted.

Can't this already be handled by calling out to a C/C++ or FORTRAN procedure that processes the data in multiple threads? For number crunching, Python is almost exclusively used as glue.

Today's machine learning and data science students don't know how to code in those languages. They know python, and maybe java.

Don't forget R... shudders
Post reply on HN