Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intent to approve PEP 703: making the GIL optional

discuss.python.org

Post reply on HN
Print view View original

481–490 of 513 posts

Re: Intent to approve PEP 703: making the GIL optional

#481

Post by weinzierl »

pjmlp wrote: post #309
weinzierl wrote: post #293

Earlier quoted context omitted.

That Java had concurrency built in from the start is a blessing mostly, but also a bit of a curse. Most of the Java ecosystem is still in the mindset that threads are cheap and firing up a couple more cannot hurt. So we end up with apps that run thousands of threads and this disease is hard to contain.

Like not everyone toy app isn't going to be the next FAANG, there are plenty of workloads where it hardly matters, while 30 years later it is still a mess in C and C++. And between C++ and Rust coroutines, still not sure which one I like less.

"And between C++ and Rust coroutines, still not sure which one I like less."

I, myself, am not a Go guy, but I feel it has to be mentioned here. Go's approach might not be as universal as C++'s or Rust's but I think for a large number of use cases it makes sense.

Re: Intent to approve PEP 703: making the GIL optional

#482

Post by commonlisp94 »

usrbinbash wrote: post #446
miraculixx wrote: post #332

Earlier quoted context omitted.

Yes, the same is true for free threading. Yet people assume free threading is free concurrency and that's the problem.

Show me someone who actually knows how threads work and what writing threading code entails who assumes that. I am perfectly aware that threads are not free. Just as I am perfectly aware that a context switch between threads is less expensive that switching a new process onto the core, and that IPC requires kernel involvement.

> context switch between threads is less expensive that switching a new process onto the core

But the overhead of a context switch for a thread and process is very similar. The main difference is whether memory is shared by default.

Re: Intent to approve PEP 703: making the GIL optional

#483

Post by paulddraper »

qbasic_forever wrote: post #323
kzrdude wrote: post #252

Earlier quoted context omitted.

Flask is pure python, isn't it?

Even pure python code could have race conditions with the GIL disabled. Stuff like accessing and modifying a dictionary item in python code is assumed and currently guaranteed to be atomic because of the GIL. Remove the GIL and decades of assumptions break.

In Python, race conditions with the GIL will be race conditions without the GIL, and vice versa.

....I believe so.

E.g. there's no plan to make lists, dicts, thread unsafe.

Re: Intent to approve PEP 703: making the GIL optional

#484

Post by sverhagen »

xmcqdpt2 wrote: post #368
sverhagen wrote: post #351

Earlier quoted context omitted.

While Python has never been my primary coding language, I've used it extensively for building scripts and tools, but I've pretty much given up on it. The language is so elegant, but the installation of it (with 2~3 compatibility issues being just a small part of that) just became such a turnoff. It's been super frustrating having to search the Internet every time I need to install something Python-related, to then fi…

The new official Scala build tool / compiler front end (scala-cli) is amazing, https://scala-cli.virtuslab.org/ The thing that really struck me after years of python is how it lets you out dependencies directly in a comment on top of a script and it will download and run with them automatically, without poisoning any system settings. It's so simple!

Just curious: what makes it "official"?

Re: Intent to approve PEP 703: making the GIL optional

#485

Post by lucidyan »

kzrdude wrote: post #344
slt2021 wrote: post #54

Naiive question: Who needs No-GIL when we have asyncio and multiprocessing packages ? never ever had a problem with GIL in python, always found a workaround just by spinning up ThreadPool or ProcessPool, and used async libraries when needed. is there any use case of No-GIL which is not solved by multiprocessing ? I thought Single threaded execution without overhead for concurrency primitives is the best way to high p…

GIL blocks parallelism for ThreadPool. But I have the same question as you have if we add another coming concurrency model: SubinterpreterThreadPool, which will be possible with the per-interpreter GIL in python 3.12 and later. That's another new model that is already confirmed to be coming: interpreters (isolated from each other) in the same process, that can run with each their own GIL.

Are we talking about "PEP 554 – Multiple Interpreters in the Stdlib"[1] proposal?

[1] https://peps.python.org/pep-0554/

Re: Intent to approve PEP 703: making the GIL optional

#486

Post by commonlisp94 »

soulbadguy wrote: post #420
commonlisp94 wrote: post #53

Earlier quoted context omitted.

Exactly. I think a lot of the negativity about GIL comes from a misunderstanding about forking processes. If python is being used as a scripting language, and spawning other tools, you're already getting free multi-core. A similar misunderstanding exists about SQLite and concurrency.. but that's a topic for another time.

> If python is being used as a scripting language And when it's not ?

When you write the entire system in python instead of using python to coordinate programs and libraries written in other languages.

Re: Intent to approve PEP 703: making the GIL optional

#487

Post by a_nop »

jborean93 wrote: post #254
a_nop wrote: post #104

Earlier quoted context omitted.

They kind of burned a breaking major version transition for no good reason with 2-to-3, now they are prefacing a major change with "it won't be like 2-to-3". It sounds like they may be maintaining two operating modes in CPython 3 instead of going forward with another major transition, just because of that history.

> They kind of burned a breaking major version transition for no good reason with 2-to-3 The unicode/text changes alone were a pretty good reason. Division producing floats are also a nice change IMO. I don’t want to discount the challenges with the transition but saying there was no good reason isn’t right to me.

Unicode was absolutely a good change, the burn was the execution

Re: Intent to approve PEP 703: making the GIL optional

#488

Post by usrbinbash »

commonlisp94 wrote: post #482
usrbinbash wrote: post #446

Earlier quoted context omitted.

Show me someone who actually knows how threads work and what writing threading code entails who assumes that. I am perfectly aware that threads are not free. Just as I am perfectly aware that a context switch between threads is less expensive that switching a new process onto the core, and that IPC requires kernel involvement.

> context switch between threads is less expensive that switching a new process onto the core But the overhead of a context switch for a thread and process is very similar. The main difference is whether memory is shared by default.

> is very similar.

Except that the threads share the exact same virtual address space, and processes do not, which makes the thread context switch faster.

And that is to say nothing about the setup and teardown process, which for a process involves copy-on-demand'ing the entire memory, but for a thread merely setting up its own stack.

Re: Intent to approve PEP 703: making the GIL optional

#489

Post by usrbinbash »

miraculixx wrote: post #473
usrbinbash wrote: post #370

Earlier quoted context omitted.

> No need to draw everyone else into this Please explain who exactly is "drawn into this"? If you want to use async, this change doesn't affect your code. If you want to use multiprocessing, this change doesn't affect your code. Even if you already use threading, and do it correctly, this change doesn't actually affect your code. So who is "drawn into this"? And please don't say library developers. a) Having to updat…

There will be 5+ years of parallel gil/nogil builds. Every library developer will have to deal with this at some point.

You do realize that this grace period is intended precisely to make it easier for libdevs to adapt, yes? 3rd item in the list in the linked article, quote:

"We also need to bring along the rest of the Python community as we gain those insights and make sure the changes we want to make, and the changes we want them to make, are palatable."

End quote.

Yes, library developers have to keep up with developments in the underlying language as well as changes in usage patterns by the community. That is true for all programming languages.

And as I have pointed out numerous times before, this change is on the wishlist of many libdevs in the python community.

Re: Intent to approve PEP 703: making the GIL optional

#490

Post by naclet »

cutler wrote: post #468
erinaceousjones wrote: post #462

Earlier quoted context omitted.

Sunk cost / laziness - I remember when Facebook wrote their own JIT VM to run PHP on top of (HHVM?) to speed up all that PHP code. Probably was easier to have one crack team of software developers write something new which could interpret all of the existing codebase, than it was to lead a widespread conversion of all of that code into faster languages. ie. not everyone's a senior dev. There's reams more junior devs…

Java isn't a scripting language. I don't get your point about junior devs either as FAANG companies such as Facebook can pick and choose from the highest caliber developers.

You are right, but Facebook practices ageism and wants young developers.

That's why many code bases are a mess because everything is dictated by almost teenagers (500 per code base ...) who rewrite everything daily.

Post reply on HN