Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intent to approve PEP 703: making the GIL optional

discuss.python.org

Post reply on HN
Print view View original

101–110 of 513 posts

Re: Intent to approve PEP 703: making the GIL optional

#101

Post by qbasic_forever »

bjourne wrote: post #96

This can (and I think will) cause issues for C extensions because many are written without multi-threading in mind. Here is a small example which is unsafe if lst can be accessed from another thread: https://news.ycombinator.com/item?id=36649769 Note that the code may cause a context switch even today if the C code callbacks into Python bytecode (via a __del__ method) and the bytecode is long enough (100 instructions…

Yep there are a ton of issues like that to be found, and unfortunately they will manifest as difficult to find and debug race conditions. This is why the proposal and work is to make non-GIL mode entirely optional and not the default.

It just means for the brave few that flip it on and use it, be prepared to spend a huge amount of time finding and fixing subtle race conditions in decades of old python library code. The early adopters are going to be in for a lot of pain, or more likely they'll restrict their use of non-GIL processes to very specialized and dedicated processes that have as few dependencies as possible.

Re: Intent to approve PEP 703: making the GIL optional

#102

Post by winter_blue »

ggm wrote: post #33

Lots of C library code for decades carried man page warnings it was unstable for use in async, re-entrant and recursive contexts. We learned how to cope and incrementally re-entrant safe versions deployed without too much API instability. Maybe time has healed wounds and caused me memory loss of the pain of discovery you'd tripped over them. String parsing which tokenised in-place. DNS calls which used static buffers…

I wonder why so many library developers even chose to build native libraries on the shaky and poorly-architected foundation that Python is.

Even writing a JVM native JNI library would have allowed to avoid a lot of that pain (and the library would have been useable from Clojure, Kotlin , Scala, JRuby[1], Jython[2], Java, etc) without any painful threading issues.

[1] which I’m aware of having been used in production by companies in the past

[2] which I’m aware has been quite a bit under maintained for the last several years

Re: Intent to approve PEP 703: making the GIL optional

#103

Post by miraculixx »

travisjungroth wrote: post #82
Waterluvian wrote: post #23

I’m glad they’re very conscious about how easily this could turn into a Python 4 debacle. They’ll have to be intensely careful not to accidentally affect yes-GIL behaviour. All kinds of weird cases are possible if any sort of emulated GIL isn’t exactly like with a GIL.

I’ve seen no description of how this won’t be like 2 -> 3 except: 1. We don’t want it to be. 2. We’ll give up quickly if it is. Those are both important points. But there seems to be an important missing third piece of “and we’ll achieve this by…”.

Indeed. Also once that Genie is out putting it back will be next to impossible.

Re: Intent to approve PEP 703: making the GIL optional

#104

Post by a_nop »

tremon wrote: post #12

I know they say specifically that they don't want a repeat of the Python3 transition scenario, but the approach they're taking now still veers eerily close to that path, at least it looks that way to me. A lot will depend on the Python community and the distribution channels. I could see the community struggling to adopt it in a timely fashion, or distributions jumping the gun (Ubuntu, Fedora, Anaconda). Maybe it's t…

They kind of burned a breaking major version transition for no good reason with 2-to-3, now they are prefacing a major change with "it won't be like 2-to-3". It sounds like they may be maintaining two operating modes in CPython 3 instead of going forward with another major transition, just because of that history.

Re: Intent to approve PEP 703: making the GIL optional

#105

Post by eru »

neilv wrote: post #61
ggm wrote: post #33

Lots of C library code for decades carried man page warnings it was unstable for use in async, re-entrant and recursive contexts. We learned how to cope and incrementally re-entrant safe versions deployed without too much API instability. Maybe time has healed wounds and caused me memory loss of the pain of discovery you'd tripped over them. String parsing which tokenised in-place. DNS calls which used static buffers…

I remember scouring those C runtime docs, for every non-reentrant function. It might be what got me in the habit of checking docs when using some API that I know moderately well, just in case there's some important detail I missed before, or something had changed. Around that time, doing cross-platform C++, I got an early look at Java, with concurrency built in from the start, along with GC and various other nice fea…

> I got an early look at Java, with concurrency built in from the start, [...] and I "knew" it was going to be huge.

And Java doesn't even have good (conceptual) support for concurrency. Compared with eg Erlang, Rust or even Haskell.

But Java was still better at it than C or C++ at the time.

Re: Intent to approve PEP 703: making the GIL optional

#106

Post by miraculixx »

qbasic_forever wrote: post #85
nnx wrote: post #72

I hope this won't make Python's dependency hell even worse, but I'm not hopeful.

Yeah it's going to be weird for some years where some libraries support no-GIL and others don't, while folks cry about the ones that don't support it holding them back. Like asyncio's introduction we'll probably see core stuff like http requests, file IO etc. all now have an entirely new permutation of libraries made to support non-GIL mode. This is going to get pretty spicy as stuff like http already has regular (bl…

import this

;)

Re: Intent to approve PEP 703: making the GIL optional

#107

Post by qbasic_forever »

paulddraper wrote: post #63
ggm wrote: post #33

Lots of C library code for decades carried man page warnings it was unstable for use in async, re-entrant and recursive contexts. We learned how to cope and incrementally re-entrant safe versions deployed without too much API instability. Maybe time has healed wounds and caused me memory loss of the pain of discovery you'd tripped over them. String parsing which tokenised in-place. DNS calls which used static buffers…

No-GIL mode is optional, and libraries will be marked "no-GIL compatible" and the ecosystem will gradually support more and more of these. No one's flipping a switch and breaking mountains of sketchy C.

That's the exact attitude that lead to decades of pain with the 2 to 3 transition and libraries though. There was zero plan for how to help libraries migrate from python 2 to 3, or more importantly how does one library support _both_ python 2 and 3 from one codebase as their users take time to switch their python interpretor. The attitude of "just turn off GIL if you don't need it, just use libraries that support turning it off" means library authors will be asked to provide versions of their library that do and don't support non-GIL mode. That's a big burden to dump on library maintainers.

Re: Intent to approve PEP 703: making the GIL optional

#108

Post by miraculixx »

lostdog wrote: post #71
vasili111 wrote: post #24

What are advantages of non-GIL Python?

In GIL Python, you might think you could speed something up by multithreading, but it turns out you can't. The GIL will just run it serially anyways. No-GIL means it is possible to run things in parallel (without resorting to fancy C extensions).

In theory it's easy. In practice you need to design programs very carefully to leverage multiple threads.

The same is true for multiple processes, yet that is much easier to reason about and a lot more likely to get it right.

Re: Intent to approve PEP 703: making the GIL optional

#110

Post by lanstin »

winter_blue wrote: post #102
ggm wrote: post #33

Lots of C library code for decades carried man page warnings it was unstable for use in async, re-entrant and recursive contexts. We learned how to cope and incrementally re-entrant safe versions deployed without too much API instability. Maybe time has healed wounds and caused me memory loss of the pain of discovery you'd tripped over them. String parsing which tokenised in-place. DNS calls which used static buffers…

I wonder why so many library developers even chose to build native libraries on the shaky and poorly-architected foundation that Python is. Even writing a JVM native JNI library would have allowed to avoid a lot of that pain (and the library would have been useable from Clojure, Kotlin , Scala, JRuby[1], Jython[2], Java, etc) without any painful threading issues. [1] which I’m aware of having been used in production…

Because with the GIL it was dead simple to glue a C library into Python, and also was the canonical way to address hot inner loops in Python: rewrite into C. Nothing fancy but a little trial and error with module loading and you get 30% speed ups without being a great C programmer.

I don’t think it’s false to say that the ease of moving hot spots into C is part of the reason Python has been so successful for thirty years.

I hope this works, but I am very sceptical about being able to port code that worked with a locking solution provided for the enthusiast trying out C working without running into concurrency bugs.

Post reply on HN