Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intent to approve PEP 703: making the GIL optional

discuss.python.org

Post reply on HN
Print view View original

201–210 of 513 posts

Re: Intent to approve PEP 703: making the GIL optional

#201

Post by cjfd »

neilv wrote: post #61
ggm wrote: post #33

Lots of C library code for decades carried man page warnings it was unstable for use in async, re-entrant and recursive contexts. We learned how to cope and incrementally re-entrant safe versions deployed without too much API instability. Maybe time has healed wounds and caused me memory loss of the pain of discovery you'd tripped over them. String parsing which tokenised in-place. DNS calls which used static buffers…

I remember scouring those C runtime docs, for every non-reentrant function. It might be what got me in the habit of checking docs when using some API that I know moderately well, just in case there's some important detail I missed before, or something had changed. Around that time, doing cross-platform C++, I got an early look at Java, with concurrency built in from the start, along with GC and various other nice fea…

What are 'MIS people'?

Re: Intent to approve PEP 703: making the GIL optional

#202

Post by _ZeD_ »

raverbashing wrote: post #187
paulddraper wrote: post #63

Earlier quoted context omitted.

No-GIL mode is optional, and libraries will be marked "no-GIL compatible" and the ecosystem will gradually support more and more of these. No one's flipping a switch and breaking mountains of sketchy C.

> marked "no-GIL compatible" Ugh, Nomenclature is hard Because non-GIL compatible doesn't mean GIL incompatible

"requires GIL" for all current projects?

Re: Intent to approve PEP 703: making the GIL optional

#204

Post by DougBTX »

TylerE wrote: post #168
qbasic_forever wrote: post #137

Earlier quoted context omitted.

Yep, and you'll have pip3-gil and pip3-nogil binaries because each permutation of python has separate and incompatible site-packages folders and libraries. It could get really ugly.

Is the situation in rust, where the answer is apparently to vendor the world, much better? Don't many of the big rust libraries still depend on nightly, too?

The major difference with rust is that the core language has strong backwards compatibility guarantees, and the package manager supports installing multiple versions of transitive dependencies so packages can be updated incrementally.

Re: Intent to approve PEP 703: making the GIL optional

#205

Post by smashed »

qbasic_forever wrote: post #107
paulddraper wrote: post #63

Earlier quoted context omitted.

No-GIL mode is optional, and libraries will be marked "no-GIL compatible" and the ecosystem will gradually support more and more of these. No one's flipping a switch and breaking mountains of sketchy C.

That's the exact attitude that lead to decades of pain with the 2 to 3 transition and libraries though. There was zero plan for how to help libraries migrate from python 2 to 3, or more importantly how does one library support _both_ python 2 and 3 from one codebase as their users take time to switch their python interpretor. The attitude of "just turn off GIL if you don't need it, just use libraries that support tur…

As I understand it, as a library author you either do absolutely nothing and your library will be marked as requiring GIL by default. Nothing to do, you keep on working with that good old GIL and nothing changes.

Or you make the extra effort of being thread safe and you can declare your library as not requiring the GIL.

Now if a user script mixes your GIL free lib with an older lib that has not been updated, well, too bad for them. Even with your hard work, the code will still operate like before, everything gets the GIL treatment.

Normal python devs will need to track down which pesky dependency of their script is causing the GIL slowdown. Kinda sucks but at least nothing breaks.

It's a sensitive, opt-in, and safe way forward. Hard to argue against it, really..

Re: Intent to approve PEP 703: making the GIL optional

#206

Post by Galanwe »

nyanpasu64 wrote: post #185
Galanwe wrote: post #156

Earlier quoted context omitted.

I don't quite see how Rust approach to multithreading is that amazing. Rust does nothing to magically prevent race conditions for you. Rust safety does not encompass race conditions, starvation, etc. What it does is defensively prevent you from sharing mutable variables without explicitly opting for it. It's good for catching careless sharing issues, but not much more.

In my experience with multi-threaded programming, C++ code with "careless sharing issues" is often filled with multiple threads accessing the same object and relying on convention to avoid calling the wrong thread's methods, pervasive data races and unsynchronized variable access, mistaken use of mutexes on only one side of shared memory, and logical race conditions that require adding mutexes (risking deadlock) or r…

> In my experience with multi-threaded programming, C++ code with "careless sharing issues" is often filled with multiple threads accessing the same object and relying on convention to avoid calling the wrong thread's methods

Right, I can see the kind of codebase you're referring to.

I don't see Rust as a magical weapon solving concurrency issues though. Namely because Rust (the compiler) has a very limited view of what happens in the lifetime of a multithreaded system, and no view at all of the lifetime of a multiprocess system.

Even when writing purely single threaded Rust, you quickly end up having to let go of the strictly static memory sharing checks and switch to dynamic ones.

I have yet to find a use case where Rust solves anything but the most blatant synchronization issues.

Re: Intent to approve PEP 703: making the GIL optional

#207

Post by pxeger1 »

slt2021 wrote: post #73
n2d4 wrote: post #66

Earlier quoted context omitted.

It's only about performance. asyncio is still inherently single-threaded, and hence also single core. multiprocessing is multi-core and hence better for performance, but each process is relatively heavy and there's additional overhead to shared memory. GIL multi-threading is both single-core and difficult to use correctly. No-GIL multi-threading is multi-core, though difficult to use. I don't know the Python implemen…

I would argue that if you have large concurrency and shared complex state - you better off use kafka and redis/memcached as a shared state - and design proper fan-out. This design scales much better for systems that will eventually overgrow one big machine. the No-GIL pytohn will be of no use, when you need to deploy your app across 100s machines. I understand people want to take advantage of all cores etc, but at la…

But many systems will never need to run at that kind of scale, but could still benefit from better threading performance, so it's good to have an "intermediate" choice, if that's what you consider it.

Re: Intent to approve PEP 703: making the GIL optional

#208

Post by TylerE »

DougBTX wrote: post #204
TylerE wrote: post #168

Earlier quoted context omitted.

Is the situation in rust, where the answer is apparently to vendor the world, much better? Don't many of the big rust libraries still depend on nightly, too?

The major difference with rust is that the core language has strong backwards compatibility guarantees, and the package manager supports installing multiple versions of transitive dependencies so packages can be updated incrementally.

Right. GP was complaint about having a gazillion versions of every dependency sitting around to satisfy various whims.

Re: Intent to approve PEP 703: making the GIL optional

#209

Post by vore »

cjfd wrote: post #201
neilv wrote: post #61

Earlier quoted context omitted.

I remember scouring those C runtime docs, for every non-reentrant function. It might be what got me in the habit of checking docs when using some API that I know moderately well, just in case there's some important detail I missed before, or something had changed. Around that time, doing cross-platform C++, I got an early look at Java, with concurrency built in from the start, along with GC and various other nice fea…

What are 'MIS people'?

Management Information Systems, I think.
Post reply on HN