Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intent to approve PEP 703: making the GIL optional

discuss.python.org

Post reply on HN
Print view View original

81–90 of 513 posts

Re: Intent to approve PEP 703: making the GIL optional

#81

Post by n2d4 »

Steltek wrote: post #69
paulddraper wrote: post #63

Earlier quoted context omitted.

No-GIL mode is optional, and libraries will be marked "no-GIL compatible" and the ecosystem will gradually support more and more of these. No one's flipping a switch and breaking mountains of sketchy C.

Well someone has to say it so I might as well: Is this going to be another Python 2 -> 3 cat herding exercise again? I suppose the need to run your app in "no-GIL" mode is less than needing to jump from 2.7 to 3.

I didn't dive deep on this, but I assume that GIL mode can still run anything, including no-GIL code (it is one of their promises, at least). So, unlike 2->3, there is forwards-compatibility.

It also seems like the latter isn't meant as a replacement (for the moment), but rather as an option.

Re: Intent to approve PEP 703: making the GIL optional

#82

Post by travisjungroth »

Waterluvian wrote: post #23

I’m glad they’re very conscious about how easily this could turn into a Python 4 debacle. They’ll have to be intensely careful not to accidentally affect yes-GIL behaviour. All kinds of weird cases are possible if any sort of emulated GIL isn’t exactly like with a GIL.

I’ve seen no description of how this won’t be like 2 -> 3 except:

1. We don’t want it to be.

2. We’ll give up quickly if it is.

Those are both important points. But there seems to be an important missing third piece of “and we’ll achieve this by…”.

Re: Intent to approve PEP 703: making the GIL optional

#84

Post by miraculixx »

nyanpasu64 wrote: post #45
miraculixx wrote: post #34

Earlier quoted context omitted.

How do you suggest we could teach proper threading (what is that?) to the masses?

Rust is a brilliant lesson in using traditional threading safely. It uses & for thread-shared types and constrains &mut to a single thread, which naturally causes people to keep single-threaded data on an object only accessible from a single thread, and make multithreaded data either immutable, mutex-protected, or atomic. Alternatively, message-passing isn't traditional threading, but Erlang/Go-style languages are an…

Great examples. Unfortunately that's not what the Python SC has decided to go for.

Re: Intent to approve PEP 703: making the GIL optional

#85

Post by qbasic_forever »

nnx wrote: post #72

I hope this won't make Python's dependency hell even worse, but I'm not hopeful.

Yeah it's going to be weird for some years where some libraries support no-GIL and others don't, while folks cry about the ones that don't support it holding them back.

Like asyncio's introduction we'll probably see core stuff like http requests, file IO etc. all now have an entirely new permutation of libraries made to support non-GIL mode. This is going to get pretty spicy as stuff like http already has regular (blocking IO) and asyncio (non blocking IO) versions, so now do they need regular non-GIL and asyncio non-GIL versions too? Is the default for a library author going forward to be creating four permutations of your library with vastly different behavior in each of them? Yuck.

Re: Intent to approve PEP 703: making the GIL optional

#86

Post by dragonwriter »

slt2021 wrote: post #78
miraculixx wrote: post #74

Earlier quoted context omitted.

Fully agree. People put far too much emphasis & expectations on the "free" part in "free multithreading".

even if they get "free multithreading" with no-GIL, their system eventually will overgrow one beefy machine and will need to be deployed across a fleet of 10/100/1000 machines. at which point you lose benefit of no-GIL, because you now have to introduce redis and kafka into the system

> even if they get “free multithreading” with no-GIL, their system eventually will overgrow one beefy machine

Why?

Yeah, if you are building a system that is, say, serving web requests, and have an internet scale potential market, success might mean that.

Not every system works that way. A simulation system with defined parameters doesn’t grow in scale if it becomes more popular, you just have more people running isolated instances that don’t depend on each other. Plenty of other applications scale that way rather than the “SaaS that serves ever more clients” way.

Re: Intent to approve PEP 703: making the GIL optional

#87

Post by pama »

miraculixx wrote: post #65
slt2021 wrote: post #54

Naiive question: Who needs No-GIL when we have asyncio and multiprocessing packages ? never ever had a problem with GIL in python, always found a workaround just by spinning up ThreadPool or ProcessPool, and used async libraries when needed. is there any use case of No-GIL which is not solved by multiprocessing ? I thought Single threaded execution without overhead for concurrency primitives is the best way to high p…

Some of the use cases advocating for nogil come from the AI/ML group of library builders, stating a need for free threading concurrency.

Agreed. Feeding the GPUs with multiple forked memory-hogging processes is no fun and leads to annoying hacks. And, yes, as per your other post, there could have been other solutions to this problem, some of which might have been better.

Re: Intent to approve PEP 703: making the GIL optional

#88

Post by qbasic_forever »

jmount wrote: post #46

Why would you even want a no-GIL Python? Java and C showed how much more effort it takes to maintain slower thread safe code for no real benefit. Parallelize at the fork level or at the isolated numeric library level.

Yep I think over the next year a lot of python devs are going to learn threading isn't magic pixie dust that makes your code fast, and in reality is starts by making your code very unstable.

Re: Intent to approve PEP 703: making the GIL optional

#89

Post by qbasic_forever »

andrewstuart wrote: post #50

As a Python developer, what would be the benefit of no GIL?

Slightly faster performance when you write multithreading code correctly (much easier said than done). Very few python devs are actually running into this as a bottleneck day to day.
Post reply on HN