Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intent to approve PEP 703: making the GIL optional

discuss.python.org

Post reply on HN
Print view View original

351–360 of 513 posts

Re: Intent to approve PEP 703: making the GIL optional

#351

Post by sverhagen »

JonChesterfield wrote: post #241
yellowapple wrote: post #216

Earlier quoted context omitted.

Or it could motivate total abandonment of system-level Python installations in favor of per-app virtualenvs or whatever the new hotness is, and we'll finally achieve world peace.

I can see system level python installs being abandoned. They seem to be getting progressively harder to use over time. I don't see the replacement being virtual env, it'll be a different language ecosystem. Whichever one looks like it has remembered "easy to do simple things in" is a feature.

While Python has never been my primary coding language, I've used it extensively for building scripts and tools, but I've pretty much given up on it. The language is so elegant, but the installation of it (with 2~3 compatibility issues being just a small part of that) just became such a turnoff. It's been super frustrating having to search the Internet every time I need to install something Python-related, to then find all sorts of conflicting instructions, bleh. Virtualenv, venv, a different Pip in Ubuntu, and so on. Since there's a lot of references to Java in these discussions: there's "sadly" two ways to do it: Maven and Gradle. That's still one too many, but at least not a new flavor du jour every year, like seemingly with Python.

And I'm sure someone will drive by, now, and tell us we have to use xyz, "obviously". But if that's not "obviously" what I find in an Internet search, then the community has apparently not agreed on it consistently in a sustained way.

Re: Intent to approve PEP 703: making the GIL optional

#352

Post by jillesvangurp »

bjourne wrote: post #96

This can (and I think will) cause issues for C extensions because many are written without multi-threading in mind. Here is a small example which is unsafe if lst can be accessed from another thread: https://news.ycombinator.com/item?id=36649769 Note that the code may cause a context switch even today if the C code callbacks into Python bytecode (via a __del__ method) and the bytecode is long enough (100 instructions…

That's fine. We go from suspecting there are issues to knowing exactly where the issues are. The rest is just chipping away at that list and making the issues go away. Either by adding some kind of mutex around the code or by replacing the native code with something less likely to have issues.

The argument against this seems mainly that it's a lot of work; not that it's impossible work. It probably is a lot of work but if there are enough people doing the work, we should get some results.

Re: Intent to approve PEP 703: making the GIL optional

#353

Post by smcl »

paulddraper wrote: post #166
nequo wrote: post #141

Earlier quoted context omitted.

Without the last sentence, your comment would be better.

It's not nice but it is true. "Dur dur this is just like 2 -> 3." HN had always been susceptible to drive-by ignorant comments, but it's reaching new levels. There's literally nothing of substance to the suggestion. (And if you think there is, present an actual informed argument.)

I sympathise with you, I'm a little tired of these same recycled comments every time the Python comes up.

Re: Intent to approve PEP 703: making the GIL optional

#354

Post by matsemann »

slt2021 wrote: post #268
matsemann wrote: post #212

Earlier quoted context omitted.

Well I don't agree that just because one needs >1 servers, no-gil is suddenly useless. Still lots of complexity and awkwardness that can be avoided if you can do threading instead of processes. Like Promotheus scraping from a non-webserver python app is a pita, as you need a new process and lots of communication, vs just plug and play as in other languages. Or just the insane resource usage. Had a java app serving mu…

It is not fair to compare CPython (which is on purpose not optimized, only a reference implementation of interpretable scripting language without any focus on performance) to OpenJDK - an arguably state of the art compiled bytecode VM with JIT and AOT compilers available, with decades and many $millions poured into runtime/JIT/GC/etc research and optimization

Why is it not fair? I need to choose a tool, how it came to be is irrelevant for me.

Re: Intent to approve PEP 703: making the GIL optional

#355

Post by kzrdude »

qbasic_forever wrote: post #323
kzrdude wrote: post #252

Earlier quoted context omitted.

Flask is pure python, isn't it?

Even pure python code could have race conditions with the GIL disabled. Stuff like accessing and modifying a dictionary item in python code is assumed and currently guaranteed to be atomic because of the GIL. Remove the GIL and decades of assumptions break.

I think that's a misconception? At least the way how I understood this issue.

Things like `map[k] = v` would be atomic both before and after nogil, and things like `map[k] += 1` are not atomic even with GIL, the read and store can be split from each other.

Re: Intent to approve PEP 703: making the GIL optional

#356

Post by n2d4 »

usrbinbash wrote: post #294
n2d4 wrote: post #122

Earlier quoted context omitted.

In Python, asyncio and multiprocessing packages can get nearly the same or better performance for IO- and CPU-intensive tasks respectively as no-GIL multithreading (and are more performant than GIL multithreading), with only a tiny fraction of the pitfalls. For any use case where the last few percent matter, consider not using Python (which will be much much more significant). Regardless, we did somehow end up here,…

> asyncio ...is useless for CPU bound tasks. The event loop uses only one core. > multiprocessing ...relies on IPC and running actual system processes, both of which have alot more overhead than switching thread context and using shared memory. > For any use case where the last few percent matter, consider not using Python (which will be much much more significant). Here is an interesting question: If asyncio and mul…

If you're interested, I wrote a more elaborate comment on asyncio vs. multiprocessing vs. multithreading here: https://news.ycombinator.com/item?id=36915581 (fyi, Python's multiprocessing has support for interprocess shared memory, with overhead)

Those languages you mentioned are not only recommended to escape parallelism problems in Python. They are recommended because they are much, much faster, period. Four of those you listed compile down to machine code, the last one has a highly optimized JIT. Just two are garbage-collected, and none do reference counting. All of them are strongly typed. These differences sum up to orders of magnitudes. If well-written Python code were equally fast as well-written C code but the only way to do parallelism was using processes, then I promise you you wouldn't get (most of) those voices telling you to escape Python. Plenty of well-written C programs choose a multiprocessing approach for parallelism over multithreading.

Long story short, if you worry about the performance overhead between multithreading and multiprocessing, make sure you worry about plenty of more significant factors that differentiate Python from faster languages first.

Re: Intent to approve PEP 703: making the GIL optional

#357

Post by miraculixx »

catnibbler wrote: post #129
Jabrov wrote: post #113

Earlier quoted context omitted.

As opposed to?

Keep the GIL and avoid the problems its removal will cause. Allow parts of your program to run in a separate namespace with explicit passing of objects. No sharing means no contention, so no overhead.

Indeed!

Re: Intent to approve PEP 703: making the GIL optional

#359

Post by miraculixx »

frfl wrote: post #8

If I remember Guido van Rossum did mention the status of the GIL in one of the Lex Friedman episodes [1] he was on (it's been a while, so I may be misremembering). Surprised to see a big decision like this happen so quickly. Did Meta's announcement play a big role in this [2]? [1]: https://www.youtube.com/watch?v=-DVyjdw4t9I [2]: https://news.ycombinator.com/item?id=36643670

It sure did. Also the decision was pretty much rushed through, if you follow the latest discussion (which got shut down on the notion that everything has been said and trust the SC and coredevs).

https://discuss.python.org/t/a-fast-free-threading-python/27...

Re: Intent to approve PEP 703: making the GIL optional

#360

Post by miraculixx »

KRAKRISMOTT wrote: post #9
frfl wrote: post #8

If I remember Guido van Rossum did mention the status of the GIL in one of the Lex Friedman episodes [1] he was on (it's been a while, so I may be misremembering). Surprised to see a big decision like this happen so quickly. Did Meta's announcement play a big role in this [2]? [1]: https://www.youtube.com/watch?v=-DVyjdw4t9I [2]: https://news.ycombinator.com/item?id=36643670

Removing him as BDFL was probably the best thing to have happened to Python. He never prioritized performance as a top priority, at least not the same way Lua, JavaScript and Java did. Even Ruby has a JIT now.

His leadership & decision arbitration is dearly missing. Now we have decide by committee and as a result we see Python being pushed and pulled in all directions.
Post reply on HN