Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intent to approve PEP 703: making the GIL optional

discuss.python.org

Post reply on HN
Print view View original

151–160 of 513 posts

Re: Intent to approve PEP 703: making the GIL optional

#151

Post by MarkMarine »

kelipso wrote: post #146
qbasic_forever wrote: post #137

Earlier quoted context omitted.

Yep, and you'll have pip3-gil and pip3-nogil binaries because each permutation of python has separate and incompatible site-packages folders and libraries. It could get really ugly.

Lol the correct statement is it will get really ugly. Pretty much no question it will be a bigger mess than 2 to 3 transition. Here's hoping my subfield will move to a different language in the meanwhile because I don't want to deal with this shit again.

All your hopes of GIL free code ruined by a GIL requiring left pad dep buried deep in the dependency tree.

Re: Intent to approve PEP 703: making the GIL optional

#152

Post by raminf »

Remember the transition of text to Unicode? 32 to 64-bit? Intel to ARM? Y2K?

No-GIL is a much smaller shift. It can follow the same transition path without radically breaking things. And if some things do break, there would be a well-defined way to handle those cases.

We all somehow survived those. Glad to see forward motion on this. It will open up a lot more terrain that has been marked off as untenable.

One of the things about early Swift that they got right was building breaking changes into the promise. Everyone knew where they stood and adjusted just fine. Sometimes I wish Python would take the same path.

Re: Intent to approve PEP 703: making the GIL optional

#153

Post by wokwokwok »

qbasic_forever wrote: post #132
paulddraper wrote: post #127

Earlier quoted context omitted.

> how does one library support _both_ python 2 and 3 from one codebase How does one library support both GIL and no-GIL? Easy, it supports no-GIL, so it supports both. Done.

You're saying, "just rewrite and re-release all your dependencies, it's easy!" which was exactly what happened disastrously with python 2 to 3.

You can down vote this til the cows come home but it’s historically what happened.

This “that’s not how it went” stuff going down is quite just blatant historical revisionism.

Maybe it’s a good idea? Maybe it’s not?

…but anyone down voting “this reminds me of the Python 2/3” fiasco has no idea what they’re talking about.

Re: Intent to approve PEP 703: making the GIL optional

#154

Post by qbasic_forever »

xmonkee wrote: post #148
qbasic_forever wrote: post #107

Earlier quoted context omitted.

That's the exact attitude that lead to decades of pain with the 2 to 3 transition and libraries though. There was zero plan for how to help libraries migrate from python 2 to 3, or more importantly how does one library support _both_ python 2 and 3 from one codebase as their users take time to switch their python interpretor. The attitude of "just turn off GIL if you don't need it, just use libraries that support tur…

no-GIL libs will work with the GIL on surely? So you won’t have to support both?

Where will you get no-GIL libraries, especially in the early days? Just yell at the maintainers of core libs like flask and requests until they use their volunteer and spare time to implement incredibly complex and tricky locking semantics all over their codebases, AND test it all with both GIL and non-GIL interpretors at scale to suss out race conditions? That just happens for free and overnight because a lot of people are plus one mashing on GitHub issues I guess?

Re: Intent to approve PEP 703: making the GIL optional

#155

Post by dboreham »

eru wrote: post #105
neilv wrote: post #61

Earlier quoted context omitted.

I remember scouring those C runtime docs, for every non-reentrant function. It might be what got me in the habit of checking docs when using some API that I know moderately well, just in case there's some important detail I missed before, or something had changed. Around that time, doing cross-platform C++, I got an early look at Java, with concurrency built in from the start, along with GC and various other nice fea…

> I got an early look at Java, with concurrency built in from the start, [...] and I "knew" it was going to be huge. And Java doesn't even have good (conceptual) support for concurrency. Compared with eg Erlang, Rust or even Haskell. But Java was still better at it than C or C++ at the time.

In your opinion. Some of us believe the exact opposite.

Re: Intent to approve PEP 703: making the GIL optional

#156

Post by Galanwe »

nyanpasu64 wrote: post #45
miraculixx wrote: post #34

Earlier quoted context omitted.

How do you suggest we could teach proper threading (what is that?) to the masses?

Rust is a brilliant lesson in using traditional threading safely. It uses & for thread-shared types and constrains &mut to a single thread, which naturally causes people to keep single-threaded data on an object only accessible from a single thread, and make multithreaded data either immutable, mutex-protected, or atomic. Alternatively, message-passing isn't traditional threading, but Erlang/Go-style languages are an…

I don't quite see how Rust approach to multithreading is that amazing.

Rust does nothing to magically prevent race conditions for you. Rust safety does not encompass race conditions, starvation, etc.

What it does is defensively prevent you from sharing mutable variables without explicitly opting for it.

It's good for catching careless sharing issues, but not much more.

Re: Intent to approve PEP 703: making the GIL optional

#157

Post by qbasic_forever »

wokwokwok wrote: post #153
qbasic_forever wrote: post #132

Earlier quoted context omitted.

You're saying, "just rewrite and re-release all your dependencies, it's easy!" which was exactly what happened disastrously with python 2 to 3.

You can down vote this til the cows come home but it’s historically what happened. This “that’s not how it went” stuff going down is quite just blatant historical revisionism. Maybe it’s a good idea? Maybe it’s not? …but anyone down voting “this reminds me of the Python 2/3” fiasco has no idea what they’re talking about.

It's good if there's a benefit for everyone, like python 3 fixing the terrible Unicode story. It's not clear non-GIL will even be a net performance improvement for most people--you are effectively moving syncronization from the core runtime to each and every library and program at the edge. Writing safe code at that level doesn't come for free, your basic program will be slower (and likely buggier) if every call into a library is now doing its own little bespoke GIL instead of relying on python's global one like now.

Re: Intent to approve PEP 703: making the GIL optional

#158

Post by donio »

ahgamut wrote: post #147

"Python 4, but not really", because we want to squeeze out more multithreading performance and be cool again. Some questions from reading the OP: - How much does performance improve due to this No-GIL thing? Is it greater than 2x? For what workloads? - Do I have to compile two versions of every extension (gil/nogil)? I would prefer building extensions does not get any more complicated. - Can I automatically update my…

The estimate in the PEP is that it will be 5-8% slower. Having to use more granular locks and atomic operations has a cost.

https://peps.python.org/pep-0703/#performance

Re: Intent to approve PEP 703: making the GIL optional

#159

Post by wtetzner »

dpedu wrote: post #43
wtetzner wrote: post #38

Earlier quoted context omitted.

Of course Java has thread safe collections in the standard library. ConcurrentHashMap etc. are popular in multithreaded contexts.

The difference is that it's a language feature in python whereas the java equivalents had to be written with locking or other approaches for handling concurrent access.

It's not a language feature in Python though. The GIL doesn't automatically make code thread safe.

Re: Intent to approve PEP 703: making the GIL optional

#160

Post by viraptor »

raminf wrote: post #152

Remember the transition of text to Unicode? 32 to 64-bit? Intel to ARM? Y2K? No-GIL is a much smaller shift. It can follow the same transition path without radically breaking things. And if some things do break, there would be a well-defined way to handle those cases. We all somehow survived those. Glad to see forward motion on this. It will open up a lot more terrain that has been marked off as untenable. One of the…

I think that's a bit different. 32 to 64 - you could test whether it works. Same for arm. Same for y2k. Sure, maybe the testing wouldn't cover the failing case, but the testing you did would be deterministic. But here? Test all you want and the answer is: it's either correct or you haven't triggered the right race yet.
Post reply on HN