Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

51–60 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#51

Post by colesbury »

dec0dedab0de wrote: post #12

Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…

Your criteria 2, 3, and 4 doesn't make much sense to me. We often have workloads that require multiple boxes, but we still want to make effective use of each box. Common server hardware has dozens of cores, which requires a lot of parallelism to fully utilize. The GIL hinders that, even when most of the work doesn't hold the GIL (see Amdahl's law)

Python multiprocessing doesn't work well with a lot of external libraries. For example, CUDA doesn't work across forks and many system resources can be shared across threads but not processes. Python objects must be pickled to be sent to another process, but not all objects can be pickled (including some built-in objects like tracebacks).

A lot of different parallel programming models can be built on top of threads (shared memory, fork-join, message passing), and to a certain extent they can be mixed. That's not true of Python multiprocessing, which only allows a narrow form of message passing. (It's also buggy, has internal race conditions, and easily leaks resources.)

The problem for CPython is that it may not be possible to remove the GIL without breaking the C API, and a lot of the benefit of Python is the huge number of high-quality packages, many of which use the C API.

Re: Let's Remove the Global Interpreter Lock

#52

Post by ant6n »

dec0dedab0de wrote: post #44
cyphar wrote: post #24

Earlier quoted context omitted.

You can't just use functions defined in your tool, you need to create a faux-cli interface in order to run each parallel worker. Also, copying large datasets between processes is not efficient. And also, there are cases where the fan-out approach is not the best way of parallelizing a task, and passing information back up to a parent task is more complicated than necessary.

"You can't just use functions defined in your tool, you need to create a faux-cli interface in order to run each parallel worker." the multiprocessing library allows you to launch multiple processes using your function definitions. It's almost the same as the multithreading library but does not share data. It seems the real problem, as you pointed out, is the additional memory. I didn't consider situations where each…

It gets more interesting when you have a large data set that's required for the computation, but as you compute, you may discover partial solutions that can be cached and used by other workers.

So not only a large read-only data set, but also a read-write cache used by all workers. This sort of thing is relatively easy with threads, but basically impossible with multiprocessing.

Re: Let's Remove the Global Interpreter Lock

#53

Post by seunosewa »

The ideal solution is for someone to design a new programming language that is as similar to Python as possible without requiring a global lock. Rarely used features that make it hard to parallelize Python would be dropped. STM might be built into the language instead of being hacked into one implementation, etc.

Re: Let's Remove the Global Interpreter Lock

#54

Post by fijal »

tyingq wrote: post #14
fijal wrote: post #9

Earlier quoted context omitted.

They would run under GIL (I can't see CPython C API being thread-friendly unless gilectomy succeeds)

Ah, ok. So this approach doesn't completely remove the GIL, but removes it as a barrier for pure python code running in PyPy? Or does it break the current support for porting cpython extensions?

It removes it for pure python code. The C extensions run under the lock (which is unfair to call interpreter any more)

Re: Let's Remove the Global Interpreter Lock

#55

Post by JulianWasTaken »

sillysaurus3 wrote: post #39
JulianWasTaken wrote: post #34

Earlier quoted context omitted.

I can't tell what you mean by the last paragraph there, but oftentimes PyPy's speedups come exactly from inlining stuff like what you refer to there -- Python's not fundamentally slower, it's those kinds of stuff that you can speed up. (And yeah the CPython API is still a pain point if you've got a library that uses it, although some stuff will still work using PyPy's emulation layer. It'd be great if people stopped…

For example, Python makes it fairly easy to trap a call to a missing method, both via __getattr__ and __missing__. In JS the only way you can do that is via Proxy objects, and even those have limits. You can't always inline the arithmetic ops effectively. You can recompile the method each time it's called with different types, but that's why the warmup time is an issue. This wouldn't be a problem if Python didn't mak…

Ah! Yes, agreed, Python does certainly make it too easy to do things that cannot reasonably be sped up.

Re: Let's Remove the Global Interpreter Lock

#57

Post by akx »

apenwarr wrote: post #35
devwastaken wrote: post #13

This would be great if it means we can run the C portions of Python in threads without performance hits. I recently started a little project that is a cross-platform GUI for batch bzip2 compression, and Python did it quite well with its built-in bzip2 module. But, once I tried to do it parallel, the performance impacts of GIL were obvious. Yes, you can work around that with multi-process, but I'd rather not be spammi…

In normal CPython, you can design your C extension (such as bzip2) to release the GIL while it runs. This is one of the few times when threads are useful in Python. It's also why scipy etc are as fast as they are. I don't know if the bzip2 module does this, but it probably should.

Yeah, the stdlib bz2 module does release the GIL for the time of the compression (though it locks the compressor object simultaneously): https://github.com/python/cpython/blob/d4b93e21c2664d6a78e06...

Re: Let's Remove the Global Interpreter Lock

#58

Post by wiremine »

> We estimate a total cost of $50k...

Just looking at it from a financial perspective, having a great Python interpreter that doesn't have a GIL seems like a no brainer for $50,000, and it creates another reason why people should take a look at PyPy.

Side note: if you haven't looked at PyPy, check it out, along with RPython

https://rpython.readthedocs.io/en/latest/

Re: Let's Remove the Global Interpreter Lock

#59

Post by jrs95 »

seunosewa wrote: post #53

The ideal solution is for someone to design a new programming language that is as similar to Python as possible without requiring a global lock. Rarely used features that make it hard to parallelize Python would be dropped. STM might be built into the language instead of being hacked into one implementation, etc.

Perl 6 doesn't have a GIL, and already has a sane concurrency model, but the lack of libraries and community interest seems to make that pretty much a non-starter.

Re: Let's Remove the Global Interpreter Lock

#60

Post by sillysaurus3 »

JulianWasTaken wrote: post #55
sillysaurus3 wrote: post #39

Earlier quoted context omitted.

For example, Python makes it fairly easy to trap a call to a missing method, both via __getattr__ and __missing__. In JS the only way you can do that is via Proxy objects, and even those have limits. You can't always inline the arithmetic ops effectively. You can recompile the method each time it's called with different types, but that's why the warmup time is an issue. This wouldn't be a problem if Python didn't mak…

Ah! Yes, agreed, Python does certainly make it too easy to do things that cannot reasonably be sped up.

Twist: Lua makes it trivial to overload arithmetic using metatables, but LuaJIT seems to have solved that. If there is any warmup time, it's hard to tell. Mike Pall is a JIT god, and I wish we had more insight into everything that went into producing one of the best JIT's of all time.

I'd love a comment/post that highlights the differences between JS and Lua as the reason why LuaJIT was able be so effective. There must be differences that make Lua possible to speed up so much. There are easy ones to think of, but the details matter a lot.

EDIT: I found some discussion at https://news.ycombinator.com/item?id=1188246 but it left me wanting more.

Related:

https://stackoverflow.com/questions/4911762/why-is-luajit-so...

http://article.gmane.org/gmane.comp.lang.lua.general/58908

http://lua-users.org/lists/lua-l/2010-03/msg00305.html

Post reply on HN