Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intent to approve PEP 703: making the GIL optional

discuss.python.org

Post reply on HN
Print view View original

211–220 of 513 posts

Re: Intent to approve PEP 703: making the GIL optional

#211

Post by gary_0 »

tremon wrote: post #12

I know they say specifically that they don't want a repeat of the Python3 transition scenario, but the approach they're taking now still veers eerily close to that path, at least it looks that way to me. A lot will depend on the Python community and the distribution channels. I could see the community struggling to adopt it in a timely fashion, or distributions jumping the gun (Ubuntu, Fedora, Anaconda). Maybe it's t…

They say they want no-gil to be the only build mode 5 years from when it becomes available. That is both too long and too short.

That is too long for Python to have 2 modes. Half a decade is more than enough time for 2 modes to become the status quo. For one thing, think of all the outdated Stackoverflow threads that will be hanging around after that 5 years. I am not optimistic that 5 years won't turn into 10 years of uncertainty and breakage.

But 5 years might be too short for everyone to dredge up all that C code, update it, test it, and call it mature.

I guess we'll see.

Re: Intent to approve PEP 703: making the GIL optional

#212

Post by matsemann »

miraculixx wrote: post #91
slt2021 wrote: post #78

Earlier quoted context omitted.

even if they get "free multithreading" with no-GIL, their system eventually will overgrow one beefy machine and will need to be deployed across a fleet of 10/100/1000 machines. at which point you lose benefit of no-GIL, because you now have to introduce redis and kafka into the system

I kept making this point as well as the other arguments above (and others did too) in the Core Dev discussion group. Unfortuately to no avail. To be sure I am not a core dev.

Well I don't agree that just because one needs >1 servers, no-gil is suddenly useless.

Still lots of complexity and awkwardness that can be avoided if you can do threading instead of processes. Like Promotheus scraping from a non-webserver python app is a pita, as you need a new process and lots of communication, vs just plug and play as in other languages.

Or just the insane resource usage. Had a java app serving multiple orders of magnitude more customers running on a few containers. Our current python app needs multiple deployments with different entry points, and about 15x the amount of containers.

Re: Intent to approve PEP 703: making the GIL optional

#213

Post by pyuser583 »

Steltek wrote: post #69
paulddraper wrote: post #63

Earlier quoted context omitted.

No-GIL mode is optional, and libraries will be marked "no-GIL compatible" and the ecosystem will gradually support more and more of these. No one's flipping a switch and breaking mountains of sketchy C.

Well someone has to say it so I might as well: Is this going to be another Python 2 -> 3 cat herding exercise again? I suppose the need to run your app in "no-GIL" mode is less than needing to jump from 2.7 to 3.

From the article:

> We want to be very careful with backward compatibility. We do not want another Python 3 situation …

Re: Intent to approve PEP 703: making the GIL optional

#214

Post by quickthrower2 »

pseudocomposer wrote: post #191
eru wrote: post #105

Earlier quoted context omitted.

> I got an early look at Java, with concurrency built in from the start, [...] and I "knew" it was going to be huge. And Java doesn't even have good (conceptual) support for concurrency. Compared with eg Erlang, Rust or even Haskell. But Java was still better at it than C or C++ at the time.

I agree that Rust handles concurrency about as well as Java (in addition to its memory safety). But could you provide docs on Erlang or Haskell concurrent data structures that were better than Java’s, say, 10-20 years ago? Java’s support for concurrent data structures was pretty unprecedented in its heyday.

STM, 2006? https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.67...

Still way after Java was released I guess

Re: Intent to approve PEP 703: making the GIL optional

#215

Post by kgeist »

Is the following possible?

- library author marks their library "no-GIL" after making sure it's thread-safe without GIL

- if the interpreter sees this metainformation, it temporarily disables GIL for the current OS thread while running the library's code

- result: old versions of Python can still run no-GIL libraries under GIL, while new versions of Python allow to gradually remove GIL

Or it's not how CPython works?

Re: Intent to approve PEP 703: making the GIL optional

#216

Post by yellowapple »

qbasic_forever wrote: post #137
MarkMarine wrote: post #128

Earlier quoted context omitted.

I’m no fan of Java, but in comparison with python, Java’s focus on extreme backwards compatibility and their ability to actually execute on this promise year after year stands in stark relief with how python has handled the same challenges. I have low confidence, despite their claims this won’t be python 4, that this will actually be executed well. Looking forward to having homebrew deliver python@3.25_GIL and python…

Yep, and you'll have pip3-gil and pip3-nogil binaries because each permutation of python has separate and incompatible site-packages folders and libraries. It could get really ugly.

Or it could motivate total abandonment of system-level Python installations in favor of per-app virtualenvs or whatever the new hotness is, and we'll finally achieve world peace.

Re: Intent to approve PEP 703: making the GIL optional

#217

Post by mlyle »

slt2021 wrote: post #78
miraculixx wrote: post #74

Earlier quoted context omitted.

Fully agree. People put far too much emphasis & expectations on the "free" part in "free multithreading".

even if they get "free multithreading" with no-GIL, their system eventually will overgrow one beefy machine and will need to be deployed across a fleet of 10/100/1000 machines. at which point you lose benefit of no-GIL, because you now have to introduce redis and kafka into the system

I think this argument presumes that everything is the sort of problem that maps well to redis and kafka. Scientific computing doesn't. And while things like numpy might lower contention on the GIL a bit, it's not a cure-all.

Finely-grained locks are useful. Even when you end up scaling between machines, it can be useful to have many threads in one memory space to maximize what you get out of one machine.

We're moving up to hundreds of cores; Python often being stuck only being able to use a couple while tightly coupling state has been unfortunate.

Re: Intent to approve PEP 703: making the GIL optional

#218

Post by miraculixx »

smashed wrote: post #205
qbasic_forever wrote: post #107

Earlier quoted context omitted.

That's the exact attitude that lead to decades of pain with the 2 to 3 transition and libraries though. There was zero plan for how to help libraries migrate from python 2 to 3, or more importantly how does one library support _both_ python 2 and 3 from one codebase as their users take time to switch their python interpretor. The attitude of "just turn off GIL if you don't need it, just use libraries that support tur…

As I understand it, as a library author you either do absolutely nothing and your library will be marked as requiring GIL by default. Nothing to do, you keep on working with that good old GIL and nothing changes. Or you make the extra effort of being thread safe and you can declare your library as not requiring the GIL. Now if a user script mixes your GIL free lib with an older lib that has not been updated, well, to…

Where can I read up on this planned way of working? This is not what the PEP says

Re: Intent to approve PEP 703: making the GIL optional

#219

Post by BGINBarbarian »

kgeist wrote: post #215

Is the following possible? - library author marks their library "no-GIL" after making sure it's thread-safe without GIL - if the interpreter sees this metainformation, it temporarily disables GIL for the current OS thread while running the library's code - result: old versions of Python can still run no-GIL libraries under GIL, while new versions of Python allow to gradually remove GIL Or it's not how CPython works?

Afaik, nogil will be a compile flag, which means that when there are two builds, you separately compile Gil and nogil builds. They will be two separate programs/binaries/packages. It could be possible for something like conda to install both binaries, then run your program with the one that matches the library flags, but python itself could not do this (afaik).

Re: Intent to approve PEP 703: making the GIL optional

#220

Post by miraculixx »

ctoth wrote: post #164
wokwokwok wrote: post #153

Earlier quoted context omitted.

You can down vote this til the cows come home but it’s historically what happened. This “that’s not how it went” stuff going down is quite just blatant historical revisionism. Maybe it’s a good idea? Maybe it’s not? …but anyone down voting “this reminds me of the Python 2/3” fiasco has no idea what they’re talking about.

Except they're not at all analogous situations. In python 2 -> 3, if a new version of a library came out Python 3 only, you couldn't use both it, and your old Python 2 code at the same time. Here, you can use the GIL until every one of your dependencies have migrated, even as new versions of your dependencies come out with nogil support until one magical day they all have nogil and you can migrate. But since you have…

* removed comment as it was factually false *
Post reply on HN