Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intent to approve PEP 703: making the GIL optional

discuss.python.org

Post reply on HN
Print view View original

171–180 of 513 posts

Re: Intent to approve PEP 703: making the GIL optional

#171

Post by pyeri »

Let us wait and watch but I somehow feel that this no-GIL mode is just a band-aid solution to Python's performance problem. The cause goes deep inside the core of Python, it gradually came to this stage as more and more features got added to the language since the 3.x transition.

I think new language features shouldn't just be added to provide syntactic sugars or coding shortcuts to programmers or just because a certain feature has become very cool (like lambda functions, for eg).

I'm glad that the Python community has realized that performance is an issue and started working on things like no-GIL mode.

People often say that Python's biggest strength is its readability and easy syntax but I disagree. Python's real strength is the enormous third party library ecosystem, popular packages like numpy, pandas, scikit, etc. which have almost become addictive in most data science projects. But now, people are thinking of other alternatives to these due to Python's performance issues. Other ecosystems like golang and rust are getting built at rapid pace and at some point, they will also have (more performant) equivalents of these packages if public shows enough interest.

Re: Intent to approve PEP 703: making the GIL optional

#172

Post by Cacti »

lanstin wrote: post #110
winter_blue wrote: post #102

Earlier quoted context omitted.

I wonder why so many library developers even chose to build native libraries on the shaky and poorly-architected foundation that Python is. Even writing a JVM native JNI library would have allowed to avoid a lot of that pain (and the library would have been useable from Clojure, Kotlin , Scala, JRuby[1], Jython[2], Java, etc) without any painful threading issues. [1] which I’m aware of having been used in production…

Because with the GIL it was dead simple to glue a C library into Python, and also was the canonical way to address hot inner loops in Python: rewrite into C. Nothing fancy but a little trial and error with module loading and you get 30% speed ups without being a great C programmer. I don’t think it’s false to say that the ease of moving hot spots into C is part of the reason Python has been so successful for thirty y…

My understanding is the thing about the binding system that drove adoption more than anything was due to how easy it was to generate bindings for an existing C library. This drove stdlib expansion and allowed rapid repurpose of huge amounts of long-standing and popular C libraries, and it also added instant credibility for those that already trusted those libraries. This helped avoid the chicken and egg problem of not enough devs for a serious stdlib, but not serious enough stdlib to draw dev (which is where most languages, even good ones, die).

I distinctly remember that during the time period where Python grew from “minor” to “dominate” (roughly 1995-2005), doing Python dev was often a process of answering “are there bindings for that?” And usually _there were_, because of that tooling.

Re: Intent to approve PEP 703: making the GIL optional

#173

Post by thayne »

miraculixx wrote: post #16

Unpopular opinion: This is a missed opportunity. What? Python could have been the one language with a sane multithreading model. Now it risks becoming a second version of Java. I fear this will make it a less attractive programming language, not least because it might lose its beginner friendlyness. For example, without the GIL a lot more care must be put into designing your programs. This can be true even though you…

> my preferred approach would have been to favor a per-thread GIL with an explicit mode to share particular objects.

From what I understand, this is how threading worked in perl. But that functionality is now "discouraged".

I do think that would have been a good way to do it. Especially with an emphasis on message passing.

Re: Intent to approve PEP 703: making the GIL optional

#174

Post by yjftsjthsd-h »

KRAKRISMOTT wrote: post #9
frfl wrote: post #8

If I remember Guido van Rossum did mention the status of the GIL in one of the Lex Friedman episodes [1] he was on (it's been a while, so I may be misremembering). Surprised to see a big decision like this happen so quickly. Did Meta's announcement play a big role in this [2]? [1]: https://www.youtube.com/watch?v=-DVyjdw4t9I [2]: https://news.ycombinator.com/item?id=36643670

Removing him as BDFL was probably the best thing to have happened to Python. He never prioritized performance as a top priority, at least not the same way Lua, JavaScript and Java did. Even Ruby has a JIT now.

> Removing him as BDFL was probably the best thing to have happened to Python. He never prioritized performance as a top priority

That doesn't follow; you're assuming that Python should prioritize performance as a top priority, which is very much not a given. Python has always excelled at being easy to use, being flexible, being a great glue language - but performant? An interpreted, dynamically typed language? That's like making a C interpreter - you can do it, but that doesn't make it a good idea.

Re: Intent to approve PEP 703: making the GIL optional

#175

Post by PaulDavisThe1st »

neilv wrote: post #61
ggm wrote: post #33

Lots of C library code for decades carried man page warnings it was unstable for use in async, re-entrant and recursive contexts. We learned how to cope and incrementally re-entrant safe versions deployed without too much API instability. Maybe time has healed wounds and caused me memory loss of the pain of discovery you'd tripped over them. String parsing which tokenised in-place. DNS calls which used static buffers…

I remember scouring those C runtime docs, for every non-reentrant function. It might be what got me in the habit of checking docs when using some API that I know moderately well, just in case there's some important detail I missed before, or something had changed. Around that time, doing cross-platform C++, I got an early look at Java, with concurrency built in from the start, along with GC and various other nice fea…

> I remember scouring those C runtime docs, for every non-reentrant function.

hint:

  % cd /usr/include
  % ag --no-color '[^A-Z]_REENTRANT'

Re: Intent to approve PEP 703: making the GIL optional

#176

Post by ehsankia »

pyeri wrote: post #171

Let us wait and watch but I somehow feel that this no-GIL mode is just a band-aid solution to Python's performance problem. The cause goes deep inside the core of Python, it gradually came to this stage as more and more features got added to the language since the 3.x transition. I think new language features shouldn't just be added to provide syntactic sugars or coding shortcuts to programmers or just because a cert…

Hasn't python been getting much faster since 3.x? Where is your evidence that new py3 features is making python slower?

Re: Intent to approve PEP 703: making the GIL optional

#178

Post by Too »

Galanwe wrote: post #140
miraculixx wrote: post #79

Earlier quoted context omitted.

Yes, used for decades - and for good reason and benefit. No, no the same thing. Sharing objects between processes is not easily achieved for various reasons (at least in Python). It would be easier to get it in multithreading with an arena based allocation model where objects live in a shared or non-shared area of memory. Also it's not my idea. I am just advocating it as the better model for Python to advance to. It…

> No, no the same thing. Well to me it is the same thing. Two threads sharing state through a common heap, or two processes sharing state through a shared memory is pretty much indistinguishable, at least on Linux. The question is not multithreading or multiprocessing anymore, the difference to me is more semantic than real. The question is then just how these threads/processes communicate. I would argue that shared…

Right now, you often get stuck on using multiprocessing because you need to share one object that is unpickleable and can't be sent over queues.

Improving shared types would be a great addition. Problem is the reason for objects being unpicklable in the first place, is often a more underlying thing that can't be shared across processes, like open file handles.

Re: Intent to approve PEP 703: making the GIL optional

#179

Post by hinkley »

qbasic_forever wrote: post #101
bjourne wrote: post #96

This can (and I think will) cause issues for C extensions because many are written without multi-threading in mind. Here is a small example which is unsafe if lst can be accessed from another thread: https://news.ycombinator.com/item?id=36649769 Note that the code may cause a context switch even today if the C code callbacks into Python bytecode (via a __del__ method) and the bytecode is long enough (100 instructions…

Yep there are a ton of issues like that to be found, and unfortunately they will manifest as difficult to find and debug race conditions. This is why the proposal and work is to make non-GIL mode entirely optional and not the default. It just means for the brave few that flip it on and use it, be prepared to spend a huge amount of time finding and fixing subtle race conditions in decades of old python library code. T…

Does Python have a lot of secondary dependencies? I could see someone pulling in two dependencies, not realize they both use the same unsafe library, and end up having them step all over each other.

Re: Intent to approve PEP 703: making the GIL optional

#180

Post by pmontra »

slt2021 wrote: post #139
miraculixx wrote: post #42

Earlier quoted context omitted.

I am sure the intent is good. I am not so sure it is possible to avoid. They already say it could take 5+ years of having gil + nogil exist in parallel. For any tool builder that means their cost has just doubled for the next five years, at least. Why? Because people will want to use tools in either mode, no matter if it is deemed productive or experimental.

essentially the adoption of No-GIL Python will depend on: 1. In which version No-GIL will become default option in CPython 2. When that CPython version will come standard in LTS Linux distro 3. When all earlier LTS distros will go out of support 4. When companies switch from outdated to target LTS version of distro currently quite a lot of companies use python3.6 only because it comes standard with the Ubuntu 14.04.6…

A customer on 18.04 just told me to wait another year and migrate their servers to 24.04 so they can stay there until 2029, or will that be 2030? They are on the standard 5 years LTS support, not the extended 10 years one.
Post reply on HN