Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intent to approve PEP 703: making the GIL optional

discuss.python.org

Post reply on HN
Print view View original

131–140 of 513 posts

Re: Intent to approve PEP 703: making the GIL optional

#131

Post by TX81Z »

Galanwe wrote: post #57
miraculixx wrote: post #16

Unpopular opinion: This is a missed opportunity. What? Python could have been the one language with a sane multithreading model. Now it risks becoming a second version of Java. I fear this will make it a less attractive programming language, not least because it might lose its beginner friendlyness. For example, without the GIL a lot more care must be put into designing your programs. This can be true even though you…

I don't quite get your "unpopular opinion". First, I don't see how the GIL would have much to do with free multithreading. The GIL should not have much observable logical impact on multithreaded _Python_ code. It should not make it more or less susceptible to race conditions. It's only practical impact should be slowness. Second, your proposed "one GIL per thread" is pretty much the equivalent of the current state of…

Queues are a HUGE pain in the arse though.

Re: Intent to approve PEP 703: making the GIL optional

#132

Post by qbasic_forever »

paulddraper wrote: post #127
qbasic_forever wrote: post #107

Earlier quoted context omitted.

That's the exact attitude that lead to decades of pain with the 2 to 3 transition and libraries though. There was zero plan for how to help libraries migrate from python 2 to 3, or more importantly how does one library support _both_ python 2 and 3 from one codebase as their users take time to switch their python interpretor. The attitude of "just turn off GIL if you don't need it, just use libraries that support tur…

> how does one library support _both_ python 2 and 3 from one codebase How does one library support both GIL and no-GIL? Easy, it supports no-GIL, so it supports both. Done.

You're saying, "just rewrite and re-release all your dependencies, it's easy!" which was exactly what happened disastrously with python 2 to 3.

Re: Intent to approve PEP 703: making the GIL optional

#133

Post by slt2021 »

travisjungroth wrote: post #82
Waterluvian wrote: post #23

I’m glad they’re very conscious about how easily this could turn into a Python 4 debacle. They’ll have to be intensely careful not to accidentally affect yes-GIL behaviour. All kinds of weird cases are possible if any sort of emulated GIL isn’t exactly like with a GIL.

I’ve seen no description of how this won’t be like 2 -> 3 except: 1. We don’t want it to be. 2. We’ll give up quickly if it is. Those are both important points. But there seems to be an important missing third piece of “and we’ll achieve this by…”.

quite a large set of users are using default python installation. There is large number of companies and non-trivial codebases that run on default python that comes with the oldest LTS version of Linux distro.

currently quite a lot of companies use python3.6 only because it comes standard with the Ubuntu 14.04.6 which happens to be the oldest LTS version - and companies have habit of migrating from out-of-support LTS version to currently-supported-oldest-LTS

this can repeat 2->3 because there will be users stuck with older versions of python and library maintainers will have to maintain both versions: with GIL and without GIL (just like PHP extension developers did with thread safe methods)

Re: Intent to approve PEP 703: making the GIL optional

#134

Post by schneems »

miraculixx wrote: post #16

Unpopular opinion: This is a missed opportunity. What? Python could have been the one language with a sane multithreading model. Now it risks becoming a second version of Java. I fear this will make it a less attractive programming language, not least because it might lose its beginner friendlyness. For example, without the GIL a lot more care must be put into designing your programs. This can be true even though you…

> IMHO the far better and still my preferred approach would have been to favor a per-thread GIL with an explicit mode to share particular objects.

You just described Ractors in Ruby, which didn’t turn out great. The setup cost for either freezing or copying memory to the target ractor to guarantee memory safety is often higher than the perf gains of the parallelism.

Not that it can’t work or won’t be improved. But there is a real world case study of what you’re recommending that we can reference without having to guess.

Re: Intent to approve PEP 703: making the GIL optional

#135

Post by n2d4 »

miraculixx wrote: post #120
qbasic_forever wrote: post #101

Earlier quoted context omitted.

Yep there are a ton of issues like that to be found, and unfortunately they will manifest as difficult to find and debug race conditions. This is why the proposal and work is to make non-GIL mode entirely optional and not the default. It just means for the brave few that flip it on and use it, be prepared to spend a huge amount of time finding and fixing subtle race conditions in decades of old python library code. T…

The intent is to make no GIL the default eventually.

I don't think this is true. There are fairly strong voices on both sides inside the community, at this time it's pretty uncertain.

To quote Guido:

>Let’s not blow it this time. If we’re going forward with nogil (and I’m not saying we are, but I can’t exclude it), let’s make sure there is a way to be able to import extensions requiring the GIL in a nogil interpreter without any additional shenanigans

https://discuss.python.org/t/pep-703-making-the-global-inter...

Re: Intent to approve PEP 703: making the GIL optional

#136

Post by ies7 »

qbasic_forever wrote: post #85
nnx wrote: post #72

I hope this won't make Python's dependency hell even worse, but I'm not hopeful.

Yeah it's going to be weird for some years where some libraries support no-GIL and others don't, while folks cry about the ones that don't support it holding them back. Like asyncio's introduction we'll probably see core stuff like http requests, file IO etc. all now have an entirely new permutation of libraries made to support non-GIL mode. This is going to get pretty spicy as stuff like http already has regular (bl…

> while folks cry about the ones that don't support it holding them back.

And someone may prefer to make NON GIL wall of shame/fame instead of directly contributing to those libraries

Re: Intent to approve PEP 703: making the GIL optional

#137

Post by qbasic_forever »

MarkMarine wrote: post #128
qbasic_forever wrote: post #107

Earlier quoted context omitted.

That's the exact attitude that lead to decades of pain with the 2 to 3 transition and libraries though. There was zero plan for how to help libraries migrate from python 2 to 3, or more importantly how does one library support _both_ python 2 and 3 from one codebase as their users take time to switch their python interpretor. The attitude of "just turn off GIL if you don't need it, just use libraries that support tur…

I’m no fan of Java, but in comparison with python, Java’s focus on extreme backwards compatibility and their ability to actually execute on this promise year after year stands in stark relief with how python has handled the same challenges. I have low confidence, despite their claims this won’t be python 4, that this will actually be executed well. Looking forward to having homebrew deliver python@3.25_GIL and python…

Yep, and you'll have pip3-gil and pip3-nogil binaries because each permutation of python has separate and incompatible site-packages folders and libraries. It could get really ugly.

Re: Intent to approve PEP 703: making the GIL optional

#138

Post by ggm »

qbasic_forever wrote: post #123
n2d4 wrote: post #81

Earlier quoted context omitted.

I didn't dive deep on this, but I assume that GIL mode can still run anything, including no-GIL code (it is one of their promises, at least). So, unlike 2->3, there is forwards-compatibility. It also seems like the latter isn't meant as a replacement (for the moment), but rather as an option.

Presumably C extensions will have a different API name/ABI to prevent accidentally calling into GIL code when in non-GIL mode and vice/versa so that's going to complicate the compatibility story.

  stat = rt_oldFunctionName(&newStateArg, oldarg[,...])
mostly (again from memory. I'd not live or die by this)

Re: Intent to approve PEP 703: making the GIL optional

#139

Post by slt2021 »

miraculixx wrote: post #42
Waterluvian wrote: post #23

I’m glad they’re very conscious about how easily this could turn into a Python 4 debacle. They’ll have to be intensely careful not to accidentally affect yes-GIL behaviour. All kinds of weird cases are possible if any sort of emulated GIL isn’t exactly like with a GIL.

I am sure the intent is good. I am not so sure it is possible to avoid. They already say it could take 5+ years of having gil + nogil exist in parallel. For any tool builder that means their cost has just doubled for the next five years, at least. Why? Because people will want to use tools in either mode, no matter if it is deemed productive or experimental.

essentially the adoption of No-GIL Python will depend on:

  1. In which version No-GIL will become default option in CPython
  2. When that CPython version will come standard in LTS Linux distro
  3. When all earlier LTS distros will go out of support
  4. When companies switch from outdated to target LTS version of distro
currently quite a lot of companies use python3.6 only because it comes standard with the Ubuntu 14.04.6 which happens to be the oldest LTS version - and companies have habit of migrating from out-of-support LTS version to currently-supported-oldest-LTS

Re: Intent to approve PEP 703: making the GIL optional

#140

Post by Galanwe »

miraculixx wrote: post #79
Galanwe wrote: post #57

Earlier quoted context omitted.

I don't quite get your "unpopular opinion". First, I don't see how the GIL would have much to do with free multithreading. The GIL should not have much observable logical impact on multithreaded _Python_ code. It should not make it more or less susceptible to race conditions. It's only practical impact should be slowness. Second, your proposed "one GIL per thread" is pretty much the equivalent of the current state of…

Yes, used for decades - and for good reason and benefit. No, no the same thing. Sharing objects between processes is not easily achieved for various reasons (at least in Python). It would be easier to get it in multithreading with an arena based allocation model where objects live in a shared or non-shared area of memory. Also it's not my idea. I am just advocating it as the better model for Python to advance to. It…

> No, no the same thing.

Well to me it is the same thing.

Two threads sharing state through a common heap, or two processes sharing state through a shared memory is pretty much indistinguishable, at least on Linux.

The question is not multithreading or multiprocessing anymore, the difference to me is more semantic than real.

The question is then just how these threads/processes communicate.

I would argue that shared mutable state is rarely a good idea, and an equivalent message based system is often preferable.

For the few use cases that remain where you would want shared mutable states, as mentioned in my original answer, Python has shared memory support, though with non-built-in types. Improving these shared types should be the only thing you advocate for.

Post reply on HN