Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

161–170 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#161

Post by eslaught »

The comments here are missing a massive use case: shared memory. Shared memory isn't just about programmer convenience. It's about using a machine's memory resources more effectively.

Yes, shared memory is available in multi-processing, but it doesn't necessarily interact well with existing codes.

I've been working on adding Python support to Legion [1], a task-based runtime system for HPC. Legion wants to manage shared memory so that multiple cores don't necessarily need multiple copies of the data, when the executing tasks don't conflict (all are read-only, or access disjoint data). Legion is C++, so this mostly "just works". Some additional work is required to support GPUs, but it's still not so difficult. But with Python, if we go with multiprocessing, we have to switch to a different mechanism. Worse, Python is an optional dependency for Legion, so we can't depend on Python's multiprocessing support either.

If you have a large existing project, and a use case that can take advantage of shared memory, being forced into Python's multiprocessing scheme for parallelism is a pain.

We've been investigating using a dlmopen approach as well, based on this proof of concept [2]. Turns out that dlmopen in every available version of libc has a critical bug that prevents it from being practically useful, if you have any desire to make use of native modules. You can build a custom libc with this patch [3] but rolling a custom libc is also a massive pain.

In all likelihood we'll end up rolling our own multiprocessing to make this work. If the GIL were truly gone though, we could potentially avoid many of these issues.

[1]: http://legion.stanford.edu/

[2]: https://news.ycombinator.com/item?id=11844268

[3]: https://patchwork.ozlabs.org/patch/496559/

Re: Let's Remove the Global Interpreter Lock

#162

Post by lemoncucumber »

andruby wrote: post #62
wiremine wrote: post #58

> We estimate a total cost of $50k... Just looking at it from a financial perspective, having a great Python interpreter that doesn't have a GIL seems like a no brainer for $50,000, and it creates another reason why people should take a look at PyPy. Side note: if you haven't looked at PyPy, check it out, along with RPython https://rpython.readthedocs.io/en/latest/

How can they estimate this? What about all the libraries that might not be compatible with the solution PyPy comes up with? This feels like a number that might in the end blow up to 10x the original estimate.

Would you say the Python ecosystem is stuffed to the GILs with incompatible libraries?

Re: Let's Remove the Global Interpreter Lock

#163

Post by njharman »

bb101 wrote: post #124
metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

I can see how using multiprocessing trumps threads for smaller programs. However it can become memory inefficient to have larger programs running in multiple processes, especially on servers with less resources.

If I run N versions of program that occupies 8mb of memory the memory footprint of the code is much less than N*8mb due to shared libraries/memory pages.

It's a factor, sure. But, one you should weigh with other factors to determine what is best.

Re: Let's Remove the Global Interpreter Lock

#164

Post by njharman »

m_mueller wrote: post #32
dec0dedab0de wrote: post #12

Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…

In my five years of python I've run up against this boundary at least once. In your list I would * take out #2. if something can make use of multiple nodes it can usually make even better use of multi-core parallelization (which affects both computational and memory bandwidth performance). multi-node comes with a much higher communications overhead, so there's a relatively wide range of applications that scale well o…

Once (or a few times) in 5 years puts this problem into the "not worth(ROI) solving" bucket for me.

Those few times, put down the hammer and use some other tool for those not naillike jobs.

Re: Let's Remove the Global Interpreter Lock

#165

Post by dr_zoidberg »

macrael wrote: post #11

Do people here use pypy in production? What are the benefits?

I tried in digital forensics. Depends on the project. May get up to 5x speedup in the software that runs, after a lot (a loooooooooot) of complaining by it. Many proejcts didn't manage to run though. In the end, not truly significant speedup (the bottleneck tends to lie somewhere else) for the effort that is required to get everything to work.

PS: I do realize "digital forensics" is probably not the kind of "production environment" you were thinking. Just a small datapoint about a particular branch of software that, while getting good speedups, may not benefit as much as the "X times faster" line would suggest.

Re: Let's Remove the Global Interpreter Lock

#166

Post by sevensor »

btilly wrote: post #139
sevensor wrote: post #33

Earlier quoted context omitted.

With threading, all of your threads can refer to the same objects. Multiprocessing means you have multiple interpreters running. That means no shared memory, and communication over pretty slow queues. I've definitely wanted to have multithreaded Python programs where all threads referred to the same large read-only data structure. But I can't do this because of the GIL. I mean, I can, but it's pointless. I can't do t…

And yet, if you could have what you want, would it actually be faster? The costs of synchronizing mutable data between cores is surprisingly high. Any time your CPU thinks that the data that it has in its cache might not be what some other CPU has in its cache, the two have to coordinate what they are doing. And thanks to the fact that Python uses reference counting, data is constantly being changed even though you d…

Right, but like I said, I'd be fine with a read-only shared data structure. I have a problem that has a hefty data model. The problem can be decomposed and attacked in parallel, but the decomposition doesn't cut across the data. Right now I run n instances on n cores, but that means making n copies of a large data structure. This requires a lot of system memory, ruins any chance I have of not wrecking the cache (not that I have high hopes there, but still), and forces me into certain patterns, like using long-lived processes because it's expensive to set up the model, that I'd prefer to avoid.

Re: Let's Remove the Global Interpreter Lock

#167

Post by njharman »

vladf wrote: post #143

There seem to be a lot of naysayers in the comments about removing the GIL. Multiprocess parallelism isn't always appropriate, so I find this to be a very promising change that will definitely make me want to switch to PyPy. Here are the use cases I've found multiprocessing to be inappropriate: * High-contention parallel operations. Doing synchronization through a Manager (a separate IPC-based synchronizing broker pr…

> Multiprocess parallelism isn't always appropriate

Using Python isn't always appropriate.

Re: Let's Remove the Global Interpreter Lock

#168

Post by weberc2 »

funkymike wrote: post #158
weberc2 wrote: post #154

Earlier quoted context omitted.

Sounds like you're saying this is infeasible; care to explain why?

I took it to mean that it is feasible. Instead of saying "well we used the wrong language, I guess we're screwed," you rewrite one component at a time, piece by piece, until the whole has been replaced. This is the approach I try to use myself. It's nearly impossible to replace an entire system all at once. But replacing one part at a time is doable and you can see the improvements much sooner.

By "it is infeasible", I meant, "removing the GIL is infeasible"; not "rewriting is infeasible".

Re: Let's Remove the Global Interpreter Lock

#169

Post by sillysaurus3 »

sillysaurus3 wrote: post #60
JulianWasTaken wrote: post #55

Earlier quoted context omitted.

Ah! Yes, agreed, Python does certainly make it too easy to do things that cannot reasonably be sped up.

Twist: Lua makes it trivial to overload arithmetic using metatables, but LuaJIT seems to have solved that. If there is any warmup time, it's hard to tell. Mike Pall is a JIT god, and I wish we had more insight into everything that went into producing one of the best JIT's of all time. I'd love a comment/post that highlights the differences between JS and Lua as the reason why LuaJIT was able be so effective. There mu…

More: https://www.reddit.com/r/programming/comments/1r2s82/lua_fun...

Re: Let's Remove the Global Interpreter Lock

#170

Post by agumonkey »

metalliqaz wrote: post #112
crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

hettinger said that multiprocess used pickle for every communication and that it must be accounted for when optimizing
Post reply on HN