Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intent to approve PEP 703: making the GIL optional

discuss.python.org

Post reply on HN
Print view View original

421–430 of 513 posts

Re: Intent to approve PEP 703: making the GIL optional

#421

Post by csmpltn »

usrbinbash wrote: post #364
miraculixx wrote: post #327

Earlier quoted context omitted.

Progress in the name if progress is rarely a good choice. The key question that remains unanswered is Why should Python even need a free threading model? There are no good answers afaik.

> There are no good answers afaik. I shall be more than happy to provide them: Supporting thread based parallelism is the norm among all mainstream PLs with the one inglorious exception of JS, which doesn't because it simply can't. And Python already does support it, it simply is limited by a legacy design decision. That was okay in a bygone age when fast single core machines were still the norm, Python was primarily…

> "It's about time Python rids itself of that needless limitation."

I want to correct one thing that I see plastered all over this thread.

The GIL isn't a programming language construct. It's an implementation detail. The GIL isn't a "Python limitation" in any way, because it has nothing to do with Python.

CPython (aka. Cython), probably the most popular and widely used VM for Python, was built around a GIL.

There are other VMs out there that don't have a GIL (ie. Jython, IronPython) and can be used out of the box.

Re: Intent to approve PEP 703: making the GIL optional

#422

Post by soulbadguy »

grumpyprole wrote: post #340
bitcharmer wrote: post #262

Earlier quoted context omitted.

This is not true. JSR-133 was one of the first and most prominent implementation of memory model among general purpose languages. You don't seem to know enough about the platform and it's history to make definite statements like that, imo.

JSR-133 was quite a few years after Java's release. But even after these changes, many problems remained. For example, standard library classes were often not thread safe out-of-the-box, even something seemingly innocuous like a date formatter class would malfunction in a threaded context. The rules around thread safety were also very complex, especially with regard to constructing objects. IMHO, it did not compare w…

+1

Re: Intent to approve PEP 703: making the GIL optional

#423

Post by MarkMarine »

sverhagen wrote: post #351
JonChesterfield wrote: post #241

Earlier quoted context omitted.

I can see system level python installs being abandoned. They seem to be getting progressively harder to use over time. I don't see the replacement being virtual env, it'll be a different language ecosystem. Whichever one looks like it has remembered "easy to do simple things in" is a feature.

While Python has never been my primary coding language, I've used it extensively for building scripts and tools, but I've pretty much given up on it. The language is so elegant, but the installation of it (with 2~3 compatibility issues being just a small part of that) just became such a turnoff. It's been super frustrating having to search the Internet every time I need to install something Python-related, to then fi…

Having different build tools, that’s one thing. I can live with a somewhat wonky development environment to work on my code, as long as I can set it up once and it works after that. I would prefer elegance, just one is used and that’s it, but I don’t know of a language with a large user base that has that.

My point lies more in the install experiences for say, a command line tool written in said language. For Java I need one Java install on my system, it’s the most recent one. I install it and then all the old code just works, all my Java command line tools just work, I never have to watch homebrew install some stupid version of Java@11 when I have Java@13 installed like I do python@3.9 when I know I already have python@3.10. All of this is BECAUSE people basically threw up their hands and distribute the core language at their version with their code and their chosen dependencies “installed” into it… so we’ve all got 15 bespoke versions of the same language kicking around on our hard drives. That is really stupid.

Re: Intent to approve PEP 703: making the GIL optional

#424

Post by MarkMarine »

yellowapple wrote: post #216
qbasic_forever wrote: post #137

Earlier quoted context omitted.

Yep, and you'll have pip3-gil and pip3-nogil binaries because each permutation of python has separate and incompatible site-packages folders and libraries. It could get really ugly.

Or it could motivate total abandonment of system-level Python installations in favor of per-app virtualenvs or whatever the new hotness is, and we'll finally achieve world peace.

I think this is standard and I find it a total and complete waste of my hard drive space.

Re: Intent to approve PEP 703: making the GIL optional

#425

Post by bigbillheck »

andybak wrote: post #373
vore wrote: post #209

Earlier quoted context omitted.

Management Information Systems, I think.

I do so love playing guessing games for every comment I read.

I don't think it's reasonable to expect people here to cater to a least common denominator of comprehension.

Re: Intent to approve PEP 703: making the GIL optional

#426

Post by [deleted] »

bjourne wrote: post #96

This can (and I think will) cause issues for C extensions because many are written without multi-threading in mind. Here is a small example which is unsafe if lst can be accessed from another thread: https://news.ycombinator.com/item?id=36649769 Note that the code may cause a context switch even today if the C code callbacks into Python bytecode (via a __del__ method) and the bytecode is long enough (100 instructions…

[deleted]

Re: Intent to approve PEP 703: making the GIL optional

#427

Post by csmpltn »

soulbadguy wrote: post #419
csmpltn wrote: post #410

Earlier quoted context omitted.

> "doesn't imply that other way is better" I never said that multiprocessing is "better" than multithreading. Both are mechanisms to achieve parallelism with different pros/cons, and there are legit cases where threads are a necessity that can't be satisfied with processes (ref my previous comment with examples). This conversation would've been much easier to have given specific constraints and examples (which nobody…

> I never said that multiprocessing is "better" than multithreading. Maybe not, but you seem to be implying that multiprocessing is a sufficient for the most case (or the average case ?) however ill define that average case is. > This conversation would've been much easier to have given specific constraints and examples (which nobody seems to give). Not quite, there is no need grand example here, processes vs threads…

> "there is no need [for a] grand example here"

> "you don't get to decide in the abstract what the right tool is for other people and context we don't know anything about"

So which one is it? Do I get to ask for concrete examples for why this work is worth all the trouble, or should we just pretend like it doesn't matter because you say so?

"I want threads, threads fast, others have threads" is not good enough.

> "somehow you say that there is no valid need for threading"

I've never said that. All I've been asking for are concrete examples that would justify removing the GIL and the impact it would have on the ecosystem. You won't provide any, despite me asking over and over again.

> "You have the magical ability to be plug into the python hivemind and decide which use case are valid"

CPython has had a GIL for the past 30 years - and things worked out just fine. There are other Python VMs that don't have a GIL - yet CPython remains the most widely used and popular VM. So how about you stop putting words in my mouth, and start actually justifying the asks beyond hand-waving my arguments away?

> "The fact that you ask for this level of qualification for a feature that pretty much every other language has is the problem."

Again, you're twisting and putting words in my mouth. There's a clear difference between building an ecosystem with multithreading in mind from day one - and suddenly introducing it out of nowhere, 30 years into it being used by millions of workloads globally. This is why I'm asking for "qualifications" (which you won't provide), and this is why I also deem your assertion that "the problem is not that deep" to be a proof beyond any that we should end this conversation now before things get too embarrassing for you :)

Re: Intent to approve PEP 703: making the GIL optional

#428

Post by [deleted] »

slt2021 wrote: post #54

Naiive question: Who needs No-GIL when we have asyncio and multiprocessing packages ? never ever had a problem with GIL in python, always found a workaround just by spinning up ThreadPool or ProcessPool, and used async libraries when needed. is there any use case of No-GIL which is not solved by multiprocessing ? I thought Single threaded execution without overhead for concurrency primitives is the best way to high p…

[deleted]

Re: Intent to approve PEP 703: making the GIL optional

#429

Post by thayne »

Galanwe wrote: post #140
miraculixx wrote: post #79

Earlier quoted context omitted.

Yes, used for decades - and for good reason and benefit. No, no the same thing. Sharing objects between processes is not easily achieved for various reasons (at least in Python). It would be easier to get it in multithreading with an arena based allocation model where objects live in a shared or non-shared area of memory. Also it's not my idea. I am just advocating it as the better model for Python to advance to. It…

> No, no the same thing. Well to me it is the same thing. Two threads sharing state through a common heap, or two processes sharing state through a shared memory is pretty much indistinguishable, at least on Linux. The question is not multithreading or multiprocessing anymore, the difference to me is more semantic than real. The question is then just how these threads/processes communicate. I would argue that shared…

> two processes sharing state through a shared memory

The point is that isn't very practical, especially in python. In practice, any data passed between processes must be serialized and deserialized over a pipe or in shared memory. With a thread per interpreter, native python objecta could potentially be shared across threads/interpreters as long as there was adequate synchronization. Technically, you could also do that with shared memory with multiprocessing, but it would be harder, and you would need to specify the object is intended to be shared from the beginning.

> pretty much indistinguishable, at least on Linux.

Linux isn't the only OS people use python on.

> I would argue that shared mutable state is rarely a good idea, and an equivalent message based system is often preferable.

I agree, but even for message passing, having to serialize and deserialize the object across process boundaries hurts performance.

Re: Intent to approve PEP 703: making the GIL optional

#430

Post by mrkeen »

slt2021 wrote: post #73
n2d4 wrote: post #66

Earlier quoted context omitted.

It's only about performance. asyncio is still inherently single-threaded, and hence also single core. multiprocessing is multi-core and hence better for performance, but each process is relatively heavy and there's additional overhead to shared memory. GIL multi-threading is both single-core and difficult to use correctly. No-GIL multi-threading is multi-core, though difficult to use. I don't know the Python implemen…

I would argue that if you have large concurrency and shared complex state - you better off use kafka and redis/memcached as a shared state - and design proper fan-out. This design scales much better for systems that will eventually overgrow one big machine. the No-GIL pytohn will be of no use, when you need to deploy your app across 100s machines. I understand people want to take advantage of all cores etc, but at la…

> you will eventually need to split computation across machines

More machines is a multiplier. Instead of underutilising 1 machine, you'll be underutilising 100s of machines. More incentive to remove locking.

Post reply on HN