Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intent to approve PEP 703: making the GIL optional

discuss.python.org

Post reply on HN
Print view View original

271–280 of 513 posts

Re: Intent to approve PEP 703: making the GIL optional

#271

Post by SkiFire13 »

pseudocomposer wrote: post #191
eru wrote: post #105

Earlier quoted context omitted.

> I got an early look at Java, with concurrency built in from the start, [...] and I "knew" it was going to be huge. And Java doesn't even have good (conceptual) support for concurrency. Compared with eg Erlang, Rust or even Haskell. But Java was still better at it than C or C++ at the time.

I agree that Rust handles concurrency about as well as Java (in addition to its memory safety). But could you provide docs on Erlang or Haskell concurrent data structures that were better than Java’s, say, 10-20 years ago? Java’s support for concurrent data structures was pretty unprecedented in its heyday.

Compared to those other languages, Java has no mechanism to enfore the use of concurrent data structures, it's entirely up to the programmer to use them.

Re: Intent to approve PEP 703: making the GIL optional

#272

Post by usrbinbash »

d0mine wrote: post #244
spamizbad wrote: post #98

Earlier quoted context omitted.

This probably isn't going to be that groundbreaking for your average web application. But for several of the niches where Python has a large footprint (AI, Data Science), being able to spin up a pile of cpu/gpu-bound threads and let them rip is a huge boon.

how likely, the corresponding code doesn't release GIL already? Pure Python is 100x slower than native code therefore the number crunching itself happens in C extensions where GIL can be released.

> therefore the number crunching itself happens in C extensions

The number crunching does, but distributing the workload, receiving results, storing and retreiving data, etc. doesn't. And these are huge losses in performance that could be avoided if we could parallelise them.

Re: Intent to approve PEP 703: making the GIL optional

#273

Post by abdulhaq »

8n4vidtmkvmk wrote: post #194
qbasic_forever wrote: post #170

Earlier quoted context omitted.

Yep I fully agree. It's going to ultimately mean 99% of people end up running in old GIL mode with deterministic behavior. Companies will get burned and have to have policies that absolutely under no circumstances will the GIL be disabled in their codebase. A very small handful of highly skilled and funded teams, probably at big companies only, will have the time and tenacity to make their code AND all their dependen…

I work at a big company. We had probably millions of lines of Python. We migrated to c++ instead of to Python 3.

I'm really struggling to think of a scenario where that makes sense

Re: Intent to approve PEP 703: making the GIL optional

#274

Post by grumpyprole »

pseudocomposer wrote: post #191
eru wrote: post #105

Earlier quoted context omitted.

> I got an early look at Java, with concurrency built in from the start, [...] and I "knew" it was going to be huge. And Java doesn't even have good (conceptual) support for concurrency. Compared with eg Erlang, Rust or even Haskell. But Java was still better at it than C or C++ at the time.

I agree that Rust handles concurrency about as well as Java (in addition to its memory safety). But could you provide docs on Erlang or Haskell concurrent data structures that were better than Java’s, say, 10-20 years ago? Java’s support for concurrent data structures was pretty unprecedented in its heyday.

Java still doesn't provide persistent data structures out-of-the-box. IMHO these are more important than lock-free mutable data structures.

When one really does need shared mutable state, Haskell supports transactional updates to mutable state (STM), with optimistic locking and rollbacks. It's awesome, but unfortunately just not practically possible in any language with rampant untracked side-effects.

Re: Intent to approve PEP 703: making the GIL optional

#275

Post by dgb23 »

hanselot wrote: post #247
KeplerBoy wrote: post #227

Earlier quoted context omitted.

Even with multithreading Python will still be slow. It's not like singlethreaded Python can keep up with singlethreaded C or even JS. This will just allow people to waste even more compute resources to get somewhat quick results from Python instead of doing the right thing. From an ecological viewpoint this PEP will be disastrous and keep entire power plants online.

Sometimes the right thing is writing code in a language that reaches market before runway runs out.

Ecosystem churn and exposing more issues to python contradicts this.

The whole point of using languages like python, js etc. is to not worry about whole classes of problems and intricacies.

There are modern languages with great tooling like Go, Rust or Clojure that have excellent support for safe concurrency and parallel execution etc.

And there are reliable, well established languages like Java/C# that have been doing this forever.

Re: Intent to approve PEP 703: making the GIL optional

#276

Post by nottorp »

superjared wrote: post #22
miraculixx wrote: post #16

Unpopular opinion: This is a missed opportunity. What? Python could have been the one language with a sane multithreading model. Now it risks becoming a second version of Java. I fear this will make it a less attractive programming language, not least because it might lose its beginner friendlyness. For example, without the GIL a lot more care must be put into designing your programs. This can be true even though you…

> Personally I can see no good will come from bringing free threading to the masses I’ve often thought that proper threading should be learned by the masses, not to be scared of it. This sort of statement does nothing but spread FUD.

> I’ve often thought that proper threading should be learned by the masses, not to be scared of it.

... or at least it should be public knowledge that anything prefixed with "async" is windows 3.1 multitasking from the 90s...

Why are we getting 64+ core servers to run single threaded cooperative multitasking on them? :)

Re: Intent to approve PEP 703: making the GIL optional

#277

Post by usrbinbash »

jmount wrote: post #46

Why would you even want a no-GIL Python? Java and C showed how much more effort it takes to maintain slower thread safe code for no real benefit. Parallelize at the fork level or at the isolated numeric library level.

> Parallelize at the fork level

IPC is a PITA, and orchestrating processes is even worse.

> or at the isolated numeric library level

Not everything I want to parallelise in python runs in numpy. Simple example: WebService Backends. I have a 64 core server running a Werkzeug/Gunicorn application. The Service is mostly doing CPU bound tasks (data aggregation and analysis), so asyncio is pointless.

What happens is, it runs 60 worker processes. Which puts hefty limitations on any crosstalk and data sharing, because these either require IPC, or using redis/sql. Which are nowhere near as performant as actually shared memory would be.

Re: Intent to approve PEP 703: making the GIL optional

#278

Post by KeplerBoy »

hanselot wrote: post #247
KeplerBoy wrote: post #227

Earlier quoted context omitted.

Even with multithreading Python will still be slow. It's not like singlethreaded Python can keep up with singlethreaded C or even JS. This will just allow people to waste even more compute resources to get somewhat quick results from Python instead of doing the right thing. From an ecological viewpoint this PEP will be disastrous and keep entire power plants online.

Sometimes the right thing is writing code in a language that reaches market before runway runs out.

That's fine for regular Python, but this doesn't convince me for multithreaded Python. Most Python modules which are optimized for performance (numpy, pytorch, pandas, and all others built on top of them) are already multithreaded and drop the GIL so you can parallelize your workload with the threading module.

If someone really needs several threads of pure python being interpreted, something is afoul imo.

Re: Intent to approve PEP 703: making the GIL optional

#279

Post by usrbinbash »

commonlisp94 wrote: post #53
jmount wrote: post #46

Why would you even want a no-GIL Python? Java and C showed how much more effort it takes to maintain slower thread safe code for no real benefit. Parallelize at the fork level or at the isolated numeric library level.

Exactly. I think a lot of the negativity about GIL comes from a misunderstanding about forking processes. If python is being used as a scripting language, and spawning other tools, you're already getting free multi-core. A similar misunderstanding exists about SQLite and concurrency.. but that's a topic for another time.

> you're already getting free multi-core.

Please explain: In what sense is the overhead of starting actual OS processes, and relying on IPC "free", compared to running threads or even greenlets, and using shared process memory?

Re: Intent to approve PEP 703: making the GIL optional

#280

Post by grumpyprole »

quickthrower2 wrote: post #214
pseudocomposer wrote: post #191

Earlier quoted context omitted.

I agree that Rust handles concurrency about as well as Java (in addition to its memory safety). But could you provide docs on Erlang or Haskell concurrent data structures that were better than Java’s, say, 10-20 years ago? Java’s support for concurrent data structures was pretty unprecedented in its heyday.

STM, 2006? https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.67... Still way after Java was released I guess

Java didn't initially ship with lock-free data structures, those came later thanks to Doug Lea.
Post reply on HN