Seems these guys never learn from the Python 2 -> 3 mess. GIL to NoGIL is even worse. Most changes from Python 2 to 3 are syntactic, but GIL to NoGIL could require a complete redesign and rewrite. Either it takes another 10+ years migrating OR nobody with a realistic production codebase with 10k dependencies can turn this No-GIL thing on.
Intent to approve PEP 703: making the GIL optional
261–270 of 513 posts
Re: Intent to approve PEP 703: making the GIL optional
#262Earlier quoted context omitted.
In your opinion. Some of us believe the exact opposite.
There are a lot of good things about java (and the jvm by some extend), but good concurency is not one of them.
This is not true. JSR-133 was one of the first and most prominent implementation of memory model among general purpose languages. You don't seem to know enough about the platform and it's history to make definite statements like that, imo.
Re: Intent to approve PEP 703: making the GIL optional
#263Earlier quoted context omitted.
Where can I read up on this planned way of working? This is not what the PEP says
It's in the "Py_mod_gil Slot" section [0] in the PEP: > In --disable-gil builds, when loading an extension, CPython will check for a new PEP 489-style Py_mod_gil slot. If the slot is set to Py_mod_gil_not_used, then extension loading proceeds as normal. If the slot is not set, the interpreter pauses all threads and enables the GIL before continuing. Additionally, the interpreter will issue a visible warning naming th…
Here's what's puzzling me about this:
Why then is there even a need for a seperate nogil build? If it is like this says, wouldn't it be easier to just make the standard build switch to gil or nogil automatically (or honor the users choice).
The fact that the SC thinks having two versions suggests there is more complexity involved than this section of the PEP leads readers to believe.
Re: Intent to approve PEP 703: making the GIL optional
#264Earlier quoted context omitted.
It's good if there's a benefit for everyone, like python 3 fixing the terrible Unicode story. It's not clear non-GIL will even be a net performance improvement for most people--you are effectively moving syncronization from the core runtime to each and every library and program at the edge. Writing safe code at that level doesn't come for free, your basic program will be slower (and likely buggier) if every call into…
I’m not gonna argue that point; but it seems massively disingenuous to down vote someone who complains “but now I have to rewrite my library because some people might use it in non-GIL mode”. That’s not whining; it’s just an observation that the committee making these decisions gives zero ducks about the impact this will have for anyone other than the handful of vested parties involved in making the decisions. Pypi h…
First of all, these changes are not being introduced because of a committee. They are being introduced because a way to get true thread-based parallelism in Python has been one of THE top priority demands of a huge part of the Python developer community for ages.
> “but now I have to rewrite my library because some people might use it in non-GIL mode”.
Yes, if library maintainers want their library to remain relevant, they will need to accomodate what the languages developer community uses. This is true for all languages. If they don't want to, that's okay, the community will come up with new libraries.
> the committee making these decisions gives zero ducks about the impact this will have
If they were giving zero ducks, they wouldn't make it backwards compatible, nor would there be a command line option to control the behavior.
>Pypi has what, 500k projects on it? Many abandoned. > >Whom exactly is going to update those?
Languages that base decisions on the update behavior, or lack thereof, of library maintainers, effectively freeze themselves.
And why exactly is the update behavior of abandoned packages a problem? They are abandoned anyway.
> Like, sure… it’s a good change for many people… once all the hard work is done by the community.
The people who want to get rid of the GIL are part of the Python development community. Many of them are library developers themselves.
Re: Intent to approve PEP 703: making the GIL optional
#265Unpopular opinion: This is a missed opportunity. What? Python could have been the one language with a sane multithreading model. Now it risks becoming a second version of Java. I fear this will make it a less attractive programming language, not least because it might lose its beginner friendlyness. For example, without the GIL a lot more care must be put into designing your programs. This can be true even though you…
> IMHO the far better and still my preferred approach would have been to favor a per-thread GIL with an explicit mode to share particular objects. You just described Ractors in Ruby, which didn’t turn out great. The setup cost for either freezing or copying memory to the target ractor to guarantee memory safety is often higher than the perf gains of the parallelism. Not that it can’t work or won’t be improved. But th…
Thanks for this. I'll read up on that. Any good links pointers to start?
Re: Intent to approve PEP 703: making the GIL optional
#266Naiive question: Who needs No-GIL when we have asyncio and multiprocessing packages ? never ever had a problem with GIL in python, always found a workaround just by spinning up ThreadPool or ProcessPool, and used async libraries when needed. is there any use case of No-GIL which is not solved by multiprocessing ? I thought Single threaded execution without overhead for concurrency primitives is the best way to high p…
> Naiive question: Who needs No-GIL when we have asyncio and multiprocessing packages ?
1. Because asyncio is completely useless when the problem is CPU bound, as the event loop still runs only on a single core. As the name implies, it is really only helpful when problems are IO bound.
2. Because sharing data between multiple processes is a giant PITA. Controlling data AND orchestrating processes is an even bigger pain.
3. Processes are expensive, and due to the aforementioned pain of sharing data, greenlets are not really a viable solutions.
Re: Intent to approve PEP 703: making the GIL optional
#267Lots of C library code for decades carried man page warnings it was unstable for use in async, re-entrant and recursive contexts. We learned how to cope and incrementally re-entrant safe versions deployed without too much API instability. Maybe time has healed wounds and caused me memory loss of the pain of discovery you'd tripped over them. String parsing which tokenised in-place. DNS calls which used static buffers…
I remember scouring those C runtime docs, for every non-reentrant function. It might be what got me in the habit of checking docs when using some API that I know moderately well, just in case there's some important detail I missed before, or something had changed. Around that time, doing cross-platform C++, I got an early look at Java, with concurrency built in from the start, along with GC and various other nice fea…
> originally was an embeddable extension language
Python was originally a teaching language, to replace BASIC and Pascal for kids and beginners.
(Yes, there once was a time when people took teaching programming seriously.)
Re: Intent to approve PEP 703: making the GIL optional
#268Earlier quoted context omitted.
I kept making this point as well as the other arguments above (and others did too) in the Core Dev discussion group. Unfortuately to no avail. To be sure I am not a core dev.
Well I don't agree that just because one needs >1 servers, no-gil is suddenly useless. Still lots of complexity and awkwardness that can be avoided if you can do threading instead of processes. Like Promotheus scraping from a non-webserver python app is a pita, as you need a new process and lots of communication, vs just plug and play as in other languages. Or just the insane resource usage. Had a java app serving mu…
It is not fair to compare CPython (which is on purpose not optimized, only a reference implementation of interpretable scripting language without any focus on performance) to OpenJDK - an arguably state of the art compiled bytecode VM with JIT and AOT compilers available, with decades and many $millions poured into runtime/JIT/GC/etc research and optimization
Re: Intent to approve PEP 703: making the GIL optional
#269Why would you even want a no-GIL Python? Java and C showed how much more effort it takes to maintain slower thread safe code for no real benefit. Parallelize at the fork level or at the isolated numeric library level.
Python gets an absolute kicking for being too slow relative to basically everything else and that performance characteristic is partially attributed to the interpreter lock. Misattributed in my opinion, but there we are.
Re: Intent to approve PEP 703: making the GIL optional
#270Earlier quoted context omitted.
> is there any use case of No-GIL which is not solved by multiprocessing ? Anything that benefits from both parallelism and replacing IPC overhead with shared data between parallel tasks.
but it would incur overhead of concurrency control: mutex, locks, semaphores. I dont believe python will ever have atomic operations, even if it had - they still incur significant overhead for concurrency control. sharing state between threads is such a narow niche use case, this pattern is practically solved by memcached/redis for larger scale python based systems
> sharing state between threads is such a narow niche use case,
Outside of python, no it really isn't, it's the norm.
And even within python itself: That redis connection IS shared state.