Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's up, Python? The GIL removed, a new compiler, optparse deprecated

bitecode.dev

Post reply on HN
Print view View original

21–30 of 302 posts

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#21

Post by LexiMax »

fbdab103 wrote: post #11

Still not encouraged by the no-GIL, "We don't want another Python 2->3 situation", yet very little proffered on how to avoid that scenario. More documentation on writing thread-safe code, suggested tooling to lint for race conditions (for whatever it is worth), discussions with popular C libraries, dedicated support channels for top tier packages, what about the enormous long-tail of abandoned extensions which still…

In a past life I hacked on PHP for a living, and in the time it took Python 2 to ride off into the sunset, PHP got two major migrations under its belt in 5.2 to 5.3, and then again 5.6 to 7.0.

It was amazing to see the contrast between the two languages. PHP gave you plenty of reasons to upgrade, and the amount of incompatible breaking changes was kept to a minimum, often paired with a way to easily shim older code to continue working.

I really hope to see no-GIL make it into Python, but in the back of my mind I also worry about what lessons were learned from the 2 to 3 transition. Does the Python team have a more effective plan this time around?

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#22

Post by Spivak »

andrewstuart wrote: post #3

From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?

Right now multi-threading makes your Python code (that isn't really C) slower. The only real use of it is time slicing so you don't starve more important code like the web server or UI thread. You still have all the concurrency issues because your threads can still still be paused and resumed at arbitrary times. It does allow some operations in Python to be atomic but I, maybe naively, assume that those cases will be guarded by new, not whole interpreter, locks.

With no-gil your multithreading code can, with no change to your code, take advantage of multiple cores and actually speed up your program. If

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#23

Post by dpedu »

hanniabu wrote: post #16

Will writing multithreaded code become easier? Or will the developer UX remain the same?

The opposite, writing multithreaded code will get harder because you'll likely need to handle concurrency issues yourself that the GIL previously avoided. But, the tradeoff is that multithreaded programs could now actually achieve multithreaded performance gains.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#24

Post by actionfromafar »

csmpltn wrote: post #17
andrewstuart wrote: post #3

From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?

There are plenty of other Python VMs that don't have a GIL and can be used already today, out of the box (examples include Jython and IronPython). Despite that fact - CPython remains the most popular Python VM out there (it utilizes a GIL). Instead of waiting for the GIL to be removed out of CPython - take your fancy Python code and just run it using a different VM. It's literally as simple as that. If the GIL was su…

If you have a lot of code, there’s plenty of Internet drama to be had in moving to another runtime, too.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#26

Post by oivey »

csmpltn wrote: post #17
andrewstuart wrote: post #3

From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?

There are plenty of other Python VMs that don't have a GIL and can be used already today, out of the box (examples include Jython and IronPython). Despite that fact - CPython remains the most popular Python VM out there (it utilizes a GIL). Instead of waiting for the GIL to be removed out of CPython - take your fancy Python code and just run it using a different VM. It's literally as simple as that. If the GIL was su…

People stay on CPython due to the performance of C extensions and the vast ecosystem based on them. The fact that people have stuck with CPython isn't at all evidence that they like the GIL or that it doesn't lead to significant technical problems.

Besides the C extension issue, Jython is based on Python 2.7 and IronPython appears to be on 3.4. These aren't serious alternatives.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#27

Post by quickthrower2 »

csmpltn wrote: post #17
andrewstuart wrote: post #3

From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?

There are plenty of other Python VMs that don't have a GIL and can be used already today, out of the box (examples include Jython and IronPython). Despite that fact - CPython remains the most popular Python VM out there (it utilizes a GIL). Instead of waiting for the GIL to be removed out of CPython - take your fancy Python code and just run it using a different VM. It's literally as simple as that. If the GIL was su…

Would a large codebase seemlessly run on another interpreter?

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#29

Post by cpgxiii »

School-Cotton wrote: post #7
kukkamario wrote: post #6

Earlier quoted context omitted.

Well Python doesn't really do proper multi-threading currently thanks to GIL blocking any additional execution threads. So removing it would enable making Python code that is actually multi-threaded without resorting to extra processes and their overhead. So if you are writing small single process Python script then removing GIL shouldn't really change much. If you are doing some heavier computing or eg. running serv…

You don’t have to use separate processes to get the benefit of multithreading in Python today — you can also call into a library written in native code that drops the GIL (e.g. Numpy or Pytorch).

That only works in some cases, if the boundary between Python and native code is absolute. In many cases users want to extend/configure the behavior of that native code, e.g. through callbacks or subclassing, and the GIL makes the behavior prohibitively slow (needing to lock/unlock to serialize at any of these potential Pythonnative boundaries) or unsafe (deadlocks/corruption if the GIL isn't handled).

There's a lot of C++ code bound in python (e.g. via pybind11) where the GIL currently imposes a hard bound on how users can employ parallelism, even in "nominally" native code.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#30

Post by charrondev »

LexiMax wrote: post #21
fbdab103 wrote: post #11

Still not encouraged by the no-GIL, "We don't want another Python 2->3 situation", yet very little proffered on how to avoid that scenario. More documentation on writing thread-safe code, suggested tooling to lint for race conditions (for whatever it is worth), discussions with popular C libraries, dedicated support channels for top tier packages, what about the enormous long-tail of abandoned extensions which still…

In a past life I hacked on PHP for a living, and in the time it took Python 2 to ride off into the sunset, PHP got two major migrations under its belt in 5.2 to 5.3, and then again 5.6 to 7.0. It was amazing to see the contrast between the two languages. PHP gave you plenty of reasons to upgrade, and the amount of incompatible breaking changes was kept to a minimum, often paired with a way to easily shim older code t…

I’ve taken an application codebase from PHP 5.3 to 8.2 now and it was relatively easy the whole way.

The real key to minimize the pain was writing effective integration tests with high coverage. We didn’t have a good test suite to start but once we added some utilities to easily call our various endpoints (and internal API client if you will) and make assertions about the coverage came quickly.

Popular frameworks like Laravel offer such test utilities out of the box now.

That combined with static analysis tools like psalm make it so we can fearlessly move past major upgrades.

One thing I was surprised at was just how much crap PHP allowed with just a notice (not even a warning for a long time). A lot of that stuff still works (although over time some notices have progressed to warnings or errors gradually). We have our test suite convert any notices or warnings to exceptions and fail the test case.

Post reply on HN