Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

211–220 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#211

Post by jhayward »

optimusclimb wrote: post #134
metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Except when your use case requires a massive shared data cache that needs to be atomically updated.

> Except when your use case requires a massive shared data cache that needs to be atomically updated

You can delete the last 6 words. Anything where multiple processes would have to read in/acquire a massive dataset to do some independent work qualifies. For instance, running some number (e.g., hundreds to hundreds of thousands) of analytical or statistical tests over a set to pick parameters, etc.

Re: Let's Remove the Global Interpreter Lock

#212

Post by std_throwaway »

dom0 wrote: post #92
PrimHelios wrote: post #64

Earlier quoted context omitted.

CPython doesn't have any reservations about breaking the Python API between minor versions, so why care about the C API? I get where you're coming from, but they've already shown they don't care much for compatibility, so I don't see why that's a big obstacle.

Removing the GIL (in a non-braindead way) likely entails breaking all existing code using the C API. PyPy could do so without breaking cpyext, by maintaining the illusion of a GIL whenever control passes to cpyext.

Does it lock the GIL so numpy can release it again immediately afterwards?

Re: Let's Remove the Global Interpreter Lock

#213

Post by jhayward »

abecedarius wrote: post #145
optimusclimb wrote: post #134

Earlier quoted context omitted.

Except when your use case requires a massive shared data cache that needs to be atomically updated.

I think it's ok to not write everything in Python, and this is a long way from the top of my problems with it.

This is self-fulfilling. As long as Python is useless for a set of tasks that are intrinsic and important to some domains, they won't use it.

Re: Let's Remove the Global Interpreter Lock

#214

Post by zeptomu »

bobwaycott wrote: post #93
ant6n wrote: post #18

Earlier quoted context omitted.

Using multiprocessing is a pain to use, and it's slow.

If you’re looking for simple threaded multiprocessing, it’s not that hard/painful: from multiprocessing.dummy import Pool pool = Pool(num_threads) result = pool.map(your_func, your_objects) pool.close() pool.join() Improve and/or complicate things from there.

This is a nice pattern and there are surprisingly many problems that can be solved that way. AFAIK you do not have to join() here as the processes die after the map call.

Often the challenge is a big amount of (hopefully read-only) data that you want to access in every 'your_func'. The naive solution is to copy the data, but this might blow your memory.

Re: Let's Remove the Global Interpreter Lock

#215

Post by digitalzombie »

rburhum wrote: post #68
seunosewa wrote: post #53

The ideal solution is for someone to design a new programming language that is as similar to Python as possible without requiring a global lock. Rarely used features that make it hard to parallelize Python would be dropped. STM might be built into the language instead of being hacked into one implementation, etc.

So basically recreate an entire language and library ecosystem because there is one feature that is less than ideal? I hope you realize why a better approach may be to reengineer that one component...

> I hope you realize why a better approach may be to reengineer that one component...

Top comment is proposing basically Erlang or an actor model.

As for immutability... well they have to either have it or manage mutable state.

That task of engineering is not something to scoff at and I think building a new language or using an existing language with those ability would help. Erlang is not a number crunching language. But there are others such as Pony.

Re: Let's Remove the Global Interpreter Lock

#216

Post by asperous »

Animats wrote: post #173
metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload. "Multiprocessing" is…

If you're CPU bound (only reason to care about the GIL anyway), then you want one process per core. So at least the L1 memory cache isn't shared. The separate memory consumption is minimal (3-5MB*N cores).

You don't need to do setup/destroy more then once.

Re: Let's Remove the Global Interpreter Lock

#217

Post by valarauca1 »

Animats wrote: post #173
metalliqaz wrote: post #112

Earlier quoted context omitted.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload. "Multiprocessing" is…

>Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive

Statically and dynamically loaded binaries are resident in the kernel's page cache. Which while each process will have different locations within its process address space for each process (b/c ALSR), they _should_ be de-duplicated in RAM, ultimately all these seperate in process images will be pointing at the same physical RAM page(s).

So from a hardware cache standpoint you're mostly okay.

Re: Let's Remove the Global Interpreter Lock

#218

Post by glic3rinu »

metalliqaz wrote: post #112
crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

Lately I've found out that multiprocessing will not help you if your program is multithreaded. There is no sane way of forking a multithreaded program. For one, the child process will inherit a copy of all locks in the state they where at forking time, possibly causing random crashes and deadlocks.

Re: Let's Remove the Global Interpreter Lock

#219

Post by kllrnohj »

breatheoften wrote: post #172
kllrnohj wrote: post #116

Earlier quoted context omitted.

#1 & #2: Consumer CPUs are now pushing 16 cores & 32 threads. Python is limited to ~1/20th of what a single box is capable of. That's a pretty big bottleneck. #4: Even if you're just talking message passing sending a message between threads is in the 10s of nanoseconds while between processes is 10s of microseconds. That's a ~1000x slowdown on core communication. Given that CPU cores are not getting any faster, that'…

Moving data between threads is only free to the extent that synchronization is free. Maybe you could say that moving immutable data between threads is free but I don't think you can say its free in general ... Doing so significantly undersells the complexity that comes with shared memory concurrency.

You seem to be conflating moving with sharing. Moving between threads is always free[1] regardless of if it's mutable or immutable, and there's no concurrency issues at all since it's a move.

Move means the sender no longer has a reference. As in, std::move, rust's ownership transfer, webworker's transferables, etc...

1: Yes there's a single synchronize point where the handoff happens, but this is part of sending a message at all. It's also independent to the size & complexity of the payload itself when we're talking multi-threaded instead of multi-process. You have that exact same sync point that costs the exact same regardless of whether your message consists of a single byte or a multi-gigabyte structure.

Re: Let's Remove the Global Interpreter Lock

#220

Post by cjbillington »

apenwarr wrote: post #35
devwastaken wrote: post #13

This would be great if it means we can run the C portions of Python in threads without performance hits. I recently started a little project that is a cross-platform GUI for batch bzip2 compression, and Python did it quite well with its built-in bzip2 module. But, once I tried to do it parallel, the performance impacts of GIL were obvious. Yes, you can work around that with multi-process, but I'd rather not be spammi…

In normal CPython, you can design your C extension (such as bzip2) to release the GIL while it runs. This is one of the few times when threads are useful in Python. It's also why scipy etc are as fast as they are. I don't know if the bzip2 module does this, but it probably should.

This. Any part of my numerical code that is a bottleneck, is either already coming from scipy or numpy, or I'm going to write in Cython if possible. Rewriting in Cython is already the opimisation you would do before going to multithreaded, because it can get you factors of 10, 100, etc, whereas multithreaded gets me a factor of 4 to 8 depending on how many cores I have and how independent the workload is.

So by the time it comes to consider multiple threads, the bottlenecks that I want to paralellise are already non-GIL-holding.

I wrote a tool to measure what proportion of the time the GIL is held in a program:

https://github.com/chrisjbillington/gil_load

I encourange people to measure what fraction of the time the GIL is actually held in their multithreaded programs. Unless it's approaching 100%, go ahead and use more threads! You will get a speedup. It's my experience that this is true more often than not. The biggest exception is poorly written C extensions that do not release the GIL even though they have no need for it. But if you're writing your own in Cython it's a matter of just typing `with nogil:`.

Post reply on HN