Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

231–240 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#231

Post by kerkeslager »

metalliqaz wrote: post #112
crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Your parent comment gives good advice, because the GIL is probably here to stay and so there's no use complaining about it. But the idea that multiprocessing gives better results than multithreading is ridiculous.

In languages which don't have a GIL, threads are almost as capable as processes, but lighter weight. Threads are almost always preferable to processes in most languages.

I understand why the GIL is still around, and don't necessarily support removing it, but it's definitely not there because it produces "better, more stable [Python] code".

Re: Let's Remove the Global Interpreter Lock

#232

Post by d0mine »

glic3rinu wrote: post #218
metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Lately I've found out that multiprocessing will not help you if your program is multithreaded. There is no sane way of forking a multithreaded program. For one, the child process will inherit a copy of all locks in the state they where at forking time, possibly causing random crashes and deadlocks.

that is why 'forkserver' start method exists. https://docs.python.org/3/library/multiprocessing.html#conte...

Re: Let's Remove the Global Interpreter Lock

#233

Post by kerkeslager »

zeptomu wrote: post #206
Animats wrote: post #173

Earlier quoted context omitted.

Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload. "Multiprocessing" is…

See my other replay in this thread. > Parallelizing your number crunching is probably not going to work very well. [...] The question is, what exactly does "number crunching" mean? We do aerial imagery analysis, so image processing in essence, which I would classify as a "number crunching" problem. A common thing e.g. is to do a time-series analysis and you can simply start multiple (2, 4, ..., N with clusters, etc.)…

...which is true, but doesn't mean you can just ignore it.

Interpreter state is among the most frequently accessed memory in many applications, meaning it's ideal to have it in cache. The difference between two interpreter states and one might not be big compared to the data being processed, but it's big enough to bump a lot of interpreter state out of cache, which for many programs can have drastic performance implications.

If you don't think cache locality is important, look at radix sort versus quicksort. Radix sort has a much lower O, but performs worse in most cases because of its poor cache locality.

Look, I get that there are fairly easy ways to work around these problems, but let's not just blithely pretend they aren't problems.

Re: Let's Remove the Global Interpreter Lock

#234

Post by lstyls »

glic3rinu wrote: post #218
metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Lately I've found out that multiprocessing will not help you if your program is multithreaded. There is no sane way of forking a multithreaded program. For one, the child process will inherit a copy of all locks in the state they where at forking time, possibly causing random crashes and deadlocks.

> There is no sane way of forking a multithreaded program

The sane way of forking a multithreaded process is to exec immediately after.

Re: Let's Remove the Global Interpreter Lock

#235

Post by astrodust »

crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

Counterpoint: Threads in JRuby work as threads should work. No GIL. No grinding of gears.

Multi-process is just one form of concurrency, and it's not always the best one.

Re: Let's Remove the Global Interpreter Lock

#236

Post by zeptomu »

kerkeslager wrote: post #233
zeptomu wrote: post #206

Earlier quoted context omitted.

See my other replay in this thread. > Parallelizing your number crunching is probably not going to work very well. [...] The question is, what exactly does "number crunching" mean? We do aerial imagery analysis, so image processing in essence, which I would classify as a "number crunching" problem. A common thing e.g. is to do a time-series analysis and you can simply start multiple (2, 4, ..., N with clusters, etc.)…

...which is true, but doesn't mean you can just ignore it. Interpreter state is among the most frequently accessed memory in many applications, meaning it's ideal to have it in cache. The difference between two interpreter states and one might not be big compared to the data being processed, but it's big enough to bump a lot of interpreter state out of cache, which for many programs can have drastic performance impli…

Agreed, but there is a lot of misinformation about the topic. I met developers that thought the GIL prevents you from running your program in multiple instances at the same time on one machine - which is obviously not the case.

Sure, it's a problem for specific workloads, and Python will get there eventually - I just don't think it is a deal breaker.

Re: Let's Remove the Global Interpreter Lock

#237

Post by hyperbovine »

Animats wrote: post #173
metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload. "Multiprocessing" is…

Actually things are not as bad as they used to be. Since 3.4 you can alter the way multiprocessing starts processes:

https://docs.python.org/3/library/multiprocessing.html#conte...

The ``forkserver`` method eliminates most of the problems you mention: child processes are only started once, and they fork() from a totally separate process so they don't inherit all of the resources of the main process (in particular, they don't copy the whole heap). I've found this eliminates 90% of the performance-related issues I used to experience with multiprocessing.

Re: Let's Remove the Global Interpreter Lock

#238

Post by lstyls »

eslaught wrote: post #161

The comments here are missing a massive use case: shared memory. Shared memory isn't just about programmer convenience. It's about using a machine's memory resources more effectively. Yes, shared memory is available in multi-processing, but it doesn't necessarily interact well with existing codes. I've been working on adding Python support to Legion [1], a task-based runtime system for HPC. Legion wants to manage sha…

This is a good point and one of the few convincing arguments I've heard against the GIL. Thanks for providing so much detail!

Did you consider just mounting a ramdisk and storing data as files? At first glance it seems like a decent fit for sharing read-only data in memory.

Re: Let's Remove the Global Interpreter Lock

#239

Post by hyperbovine »

glic3rinu wrote: post #218
metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Lately I've found out that multiprocessing will not help you if your program is multithreaded. There is no sane way of forking a multithreaded program. For one, the child process will inherit a copy of all locks in the state they where at forking time, possibly causing random crashes and deadlocks.

If the child program is multithreaded then it's almost certainly not pure Python in the first place. So, wrap it up in `with nogil:` Cython statements and use the threading module (or concurrent.futures.ThreadPoolExecutor).

Re: Let's Remove the Global Interpreter Lock

#240

Post by Pxtl »

Ick, I'd forgotten how monkeypatchable the core of Python was.

If not for that, I'd focus on supporting some kind of pseudo-process where multiple instances of the Python interpreter could be loaded but they would only share pure-functional libs which, I assume, could be used in a threadsafe fashion... but then you run into the mutability of those libs. Well, the mutability of everything in python. Plus what happens if those libs expose anythign that you could hold a reference to - what happens to refcounting in a multithreaded Python?

Honestly, I feel like the world has passed Python by. At this point the cost of its performance limitations don't seem to be worth its payoff. Not that it's a bad language - I like Python. I just don't really feel the need to use it for anything anymore.

Post reply on HN