Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

gh-116167: Allow disabling the GIL

github.com

Post reply on HN
Print view View original

31–40 of 259 posts

Re: gh-116167: Allow disabling the GIL

#31

Post by znpy »

helsinki wrote: post #2

While the title is correct, it is a bit misleading, because disabling the GIL breaks the asyncio tests. It's like saying the engine can be removed from my car. Sure, it can, but the car won't work.

> While the title is correct, it is a bit misleading, because disabling the GIL breaks the asyncio tests. It's like saying the engine can be removed from my car. Sure, it can, but the car won't work.

You're not supposed to drive a car that hasn't got out of the research and development laboratory either, so there's that.

Re: gh-116167: Allow disabling the GIL

#32

Post by sparks1970 »

behnamoh wrote: post #25

I've been programming in Python for over 6 years now and every week I learn something new. But recently I've been thinking about moving to a more capable language with proper concurrency for backend API requests (FastAPI sucked). I also want types, so Elixir is not in the picture. I dabbled in Rust a bit. Although I was able to get the hang of things and build a CLI tool pretty quickly, I'm worried I'll have to deal…

Golang? Pretty easy to pick up coming from Python and proper concurrency.

Re: gh-116167: Allow disabling the GIL

#33

Post by pillusmany »

kroolik wrote: post #19
protomikron wrote: post #9

Although this is nice, the problems with the GIL are often blown out of proportion: people stating that you couldn't do efficient (compute-bounded) multi-processing, which was never the case as the `multiprocessing` module works just fine.

Managing processes is more annoying than threads, though. Incl. data passing and so forth.

The "ray" library makes running python code on multi core and clusters very easy.

Re: gh-116167: Allow disabling the GIL

#34

Post by pillusmany »

ynik wrote: post #28
protomikron wrote: post #9

Although this is nice, the problems with the GIL are often blown out of proportion: people stating that you couldn't do efficient (compute-bounded) multi-processing, which was never the case as the `multiprocessing` module works just fine.

multiprocessing only works fine when you're working on problems that don't require 10+ GB of memory per process . Once you have significant memory usage, you really need to find a way to share that memory across multiple CPU cores. For non-trivial data structures partly implemented in C++ (as optimization, because pure python would be too slow), that means messing with allocators and shared memory. Such GIL-workaroun…

"Ray" can share python objects memory between processes. It's also much easier to use than multi processing.

Re: gh-116167: Allow disabling the GIL

#35

Post by ebiester »

ynik wrote: post #28
protomikron wrote: post #9

Although this is nice, the problems with the GIL are often blown out of proportion: people stating that you couldn't do efficient (compute-bounded) multi-processing, which was never the case as the `multiprocessing` module works just fine.

multiprocessing only works fine when you're working on problems that don't require 10+ GB of memory per process . Once you have significant memory usage, you really need to find a way to share that memory across multiple CPU cores. For non-trivial data structures partly implemented in C++ (as optimization, because pure python would be too slow), that means messing with allocators and shared memory. Such GIL-workaroun…

And I guess what I don't understand is why people choose Python for these use cases. I am not in the "Rustify" everything camp, but Go + C, Java + JNI, Rust, and C++ all seem like more suitable solutions.

Re: gh-116167: Allow disabling the GIL

#38

Post by oivey »

ebiester wrote: post #35
ynik wrote: post #28

Earlier quoted context omitted.

multiprocessing only works fine when you're working on problems that don't require 10+ GB of memory per process . Once you have significant memory usage, you really need to find a way to share that memory across multiple CPU cores. For non-trivial data structures partly implemented in C++ (as optimization, because pure python would be too slow), that means messing with allocators and shared memory. Such GIL-workaroun…

And I guess what I don't understand is why people choose Python for these use cases. I am not in the "Rustify" everything camp, but Go + C, Java + JNI, Rust, and C++ all seem like more suitable solutions.

Notably, all of those are static languages and none of them have array types as nice as PyTorch or NumPy, among many other packages in the Python ecosystem. Those two facts are likely closely related.

Re: gh-116167: Allow disabling the GIL

#39

Post by scubbo »

behnamoh wrote: post #25

I've been programming in Python for over 6 years now and every week I learn something new. But recently I've been thinking about moving to a more capable language with proper concurrency for backend API requests (FastAPI sucked). I also want types, so Elixir is not in the picture. I dabbled in Rust a bit. Although I was able to get the hang of things and build a CLI tool pretty quickly, I'm worried I'll have to deal…

Python supports types! https://www.mypy-lang.org/

Re: gh-116167: Allow disabling the GIL

#40

Post by sapiogram »

gtirloni wrote: post #12
ollien wrote: post #8

Earlier quoted context omitted.

How do single-threaded programs benefit from a lack of GIL?

It could remove the locking/unlocking operations.

Removing the GIL requires more locking/unlocking operations. For single-threaded program, it's a performance penalty on average: https://peps.python.org/pep-0703/#performance
Post reply on HN