Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intent to approve PEP 703: making the GIL optional

discuss.python.org

Post reply on HN
Print view View original

221–230 of 513 posts

Re: Intent to approve PEP 703: making the GIL optional

#221

Post by mlyle »

asvitkine wrote: post #165
n2d4 wrote: post #25

Earlier quoted context omitted.

The GIL does very little to protect unexperienced users. It's still really easy to run into race conditions, for example, if your thread gets scheduled out in any multi-instruction operation (this is more common than you think [1]). In general, Python code still has to be thread-safe; you get the risks without the benefits. If you don't care about CPU performance, instead of threads you should go for an event-loop ap…

I don't think the blog you cite backs up your claim of "it's still really easy to run into race conditions". The author literally says: "This was actually pretty hard to discover. The first few experiments failed, because Python is pretty smart about when it runs each thread." But I think the main way the GIL protects inexperienced users is that its presence has the effect that Python code using the threading APIs is…

> I don't think the blog you cite backs up your claim of "it's still really easy to run into race conditions".

> The author literally says: "This was actually pretty hard to discover. The first few experiments failed, because Python is pretty smart about when it runs each thread."

I completely disagree here with your reading. Rare race conditions are much worse than races you trip over frequently.

Re: Intent to approve PEP 703: making the GIL optional

#222

Post by IshKebab »

jmount wrote: post #46

Why would you even want a no-GIL Python? Java and C showed how much more effort it takes to maintain slower thread safe code for no real benefit. Parallelize at the fork level or at the isolated numeric library level.

Well, multithreading in C or Java makes sense because it's the only way to increase performance if you've already optimised the single threaded case.

In Python there's a much better option if you care about performance - use a different language!

Re: Intent to approve PEP 703: making the GIL optional

#224

Post by miraculixx »

wokwokwok wrote: post #161
qbasic_forever wrote: post #157

Earlier quoted context omitted.

It's good if there's a benefit for everyone, like python 3 fixing the terrible Unicode story. It's not clear non-GIL will even be a net performance improvement for most people--you are effectively moving syncronization from the core runtime to each and every library and program at the edge. Writing safe code at that level doesn't come for free, your basic program will be slower (and likely buggier) if every call into…

I’m not gonna argue that point; but it seems massively disingenuous to down vote someone who complains “but now I have to rewrite my library because some people might use it in non-GIL mode”. That’s not whining; it’s just an observation that the committee making these decisions gives zero ducks about the impact this will have for anyone other than the handful of vested parties involved in making the decisions. Pypi h…

At the end of the day,this is the very point the SC and core devs are ignoring in their decision.

They do recognize the impact on the ecosystem will be huge, they recognize there will be at least 5 years of parallel gil/nogil versions(*), and they say they don't want a 2-3 story all over again.

Yet they have decided against their own best advise (to avoid such a situation).

I find it utterly confusing.

(*) not just one parallel version, there will be 2 for every release following the introduction. In reality organisations will have to maintain 4-6 different baselines of Python releases along with a matching (and likely differing) set of libraries. I don't mean to fearmonger, I just happen to maintain such environements and I know the effort that goes into this first hand.

Re: Intent to approve PEP 703: making the GIL optional

#225

Post by d0mine »

lanstin wrote: post #110
winter_blue wrote: post #102

Earlier quoted context omitted.

I wonder why so many library developers even chose to build native libraries on the shaky and poorly-architected foundation that Python is. Even writing a JVM native JNI library would have allowed to avoid a lot of that pain (and the library would have been useable from Clojure, Kotlin , Scala, JRuby[1], Jython[2], Java, etc) without any painful threading issues. [1] which I’m aware of having been used in production…

Because with the GIL it was dead simple to glue a C library into Python, and also was the canonical way to address hot inner loops in Python: rewrite into C. Nothing fancy but a little trial and error with module loading and you get 30% speed ups without being a great C programmer. I don’t think it’s false to say that the ease of moving hot spots into C is part of the reason Python has been so successful for thirty y…

btw, many C extensions release GIL already e.g., one Python thread can do regex work, another some number crunching, yet another waiting IO--all in parallel.

Re: Intent to approve PEP 703: making the GIL optional

#226

Post by almostnormal »

eru wrote: post #105
neilv wrote: post #61

Earlier quoted context omitted.

I remember scouring those C runtime docs, for every non-reentrant function. It might be what got me in the habit of checking docs when using some API that I know moderately well, just in case there's some important detail I missed before, or something had changed. Around that time, doing cross-platform C++, I got an early look at Java, with concurrency built in from the start, along with GC and various other nice fea…

> I got an early look at Java, with concurrency built in from the start, [...] and I "knew" it was going to be huge. And Java doesn't even have good (conceptual) support for concurrency. Compared with eg Erlang, Rust or even Haskell. But Java was still better at it than C or C++ at the time.

> But Java was still better at it than C or C++ at the time.

That's not even a contest. Java was released in 1995. At that time C did not have any kind of support for concurrency, all solutions were platform dependend and not part of the language.

Re: Intent to approve PEP 703: making the GIL optional

#227

Post by KeplerBoy »

n2d4 wrote: post #66
slt2021 wrote: post #54

Naiive question: Who needs No-GIL when we have asyncio and multiprocessing packages ? never ever had a problem with GIL in python, always found a workaround just by spinning up ThreadPool or ProcessPool, and used async libraries when needed. is there any use case of No-GIL which is not solved by multiprocessing ? I thought Single threaded execution without overhead for concurrency primitives is the best way to high p…

It's only about performance. asyncio is still inherently single-threaded, and hence also single core. multiprocessing is multi-core and hence better for performance, but each process is relatively heavy and there's additional overhead to shared memory. GIL multi-threading is both single-core and difficult to use correctly. No-GIL multi-threading is multi-core, though difficult to use. I don't know the Python implemen…

Even with multithreading Python will still be slow. It's not like singlethreaded Python can keep up with singlethreaded C or even JS.

This will just allow people to waste even more compute resources to get somewhat quick results from Python instead of doing the right thing.

From an ecological viewpoint this PEP will be disastrous and keep entire power plants online.

Re: Intent to approve PEP 703: making the GIL optional

#228

Post by geewee »

raminf wrote: post #152

Remember the transition of text to Unicode? 32 to 64-bit? Intel to ARM? Y2K? No-GIL is a much smaller shift. It can follow the same transition path without radically breaking things. And if some things do break, there would be a well-defined way to handle those cases. We all somehow survived those. Glad to see forward motion on this. It will open up a lot more terrain that has been marked off as untenable. One of the…

I mean going from text to unicode did pose a huge problem for python specifically.

Re: Intent to approve PEP 703: making the GIL optional

#229

Post by tjpnz »

KeplerBoy wrote: post #227
n2d4 wrote: post #66

Earlier quoted context omitted.

It's only about performance. asyncio is still inherently single-threaded, and hence also single core. multiprocessing is multi-core and hence better for performance, but each process is relatively heavy and there's additional overhead to shared memory. GIL multi-threading is both single-core and difficult to use correctly. No-GIL multi-threading is multi-core, though difficult to use. I don't know the Python implemen…

Even with multithreading Python will still be slow. It's not like singlethreaded Python can keep up with singlethreaded C or even JS. This will just allow people to waste even more compute resources to get somewhat quick results from Python instead of doing the right thing. From an ecological viewpoint this PEP will be disastrous and keep entire power plants online.

The text editors people use to write said code will keep even more power plants online.

Re: Intent to approve PEP 703: making the GIL optional

#230

Post by r0l1 »

jmount wrote: post #46

Why would you even want a no-GIL Python? Java and C showed how much more effort it takes to maintain slower thread safe code for no real benefit. Parallelize at the fork level or at the isolated numeric library level.

We are working with a huge Go and Python codebase and Python is just a pain in terms of using all system resources. We moved many parts to C++ which are called and handled by goroutines. The outcome was a big success. This proposal/change is a big step forward, especially for the deep learning community.
Post reply on HN