Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

gh-116167: Allow disabling the GIL

github.com

Post reply on HN
Print view View original

201–210 of 259 posts

Re: gh-116167: Allow disabling the GIL

#201

Post by fireattack »

dathery wrote: post #161
tiffanyh wrote: post #130

ELI5 I get in concept what the GIL is. But what's the impact of this change? Packages will now break, for the hope of better overall performance?

Previously people basically didn't bother to write multithreaded Python at all due to the GIL. Threads were primarily used when you had multiple pieces of work to do which could end up blocked on independent I/O. Which is common and useful of course, but doesn't help with the performance of CPU-bound Python code. Even outside of high-intensity CPU work, this can be useful. A problem lately is that a lot of code is wr…

Would GIT affect how Python runs across multiple Python processes?

I'm asking because I encountered a weird phenomenon before.

I use a simple Python lib called "schedule" which is to run some tasks periodically (not precise). And I often run a script multiple times (with different arguments) to monitor something say, every 30 seconds. So they're in three separate Python Interpreter processes.

What I've noticed is that while when I initiated them, they were something like 5 seconds apart, they eventually will end up running in sync. Probably not related to GIL at all, but I guess do no harm to ask.

Re: gh-116167: Allow disabling the GIL

#202

Post by oblio »

fragmede wrote: post #115
tommiegannert wrote: post #50

First I read the news of tranched bread, and now this?! What a time! I was a bit disheartened when the Unladen Swallow project [1] fizzled out. Great to see Python back on the core optimization track. [1] https://en.wikipedia.org/wiki/CPython#Unladen_Swallow

tranched bread?

OP is a subprime mortgage seller.

Re: gh-116167: Allow disabling the GIL

#203

Post by Cacti »

purpleidea wrote: post #200

It's neat that this will eventually improve some python code, but at the end of the day, it's still a badly typed language, which will still be slower and less safe than more modern languages. Learn golang or rust instead. Impressive that they are managing this though!

but Python has my libraries, and go and rust do not.

>> 99% of my compute is offloaded to compiled BLAS or CUDA.

types are enough.

and memory and type safety are not terribly relevant for my use cases.

Re: gh-116167: Allow disabling the GIL

#204

Post by alfalfasprout »

purpleidea wrote: post #200

It's neat that this will eventually improve some python code, but at the end of the day, it's still a badly typed language, which will still be slower and less safe than more modern languages. Learn golang or rust instead. Impressive that they are managing this though!

Bad take. One doesn't just migrate massive ecosystems to "go or rust". Python, like it or not, is the lingua franca of ML/AI and science. Worse, as a user of Go/Rust it's very disingenuous to expect the same kind of iteration ability in those languages as you get in Python.

With tools like pyright now + the work on nogil everyone benefits from this using Python.

Re: gh-116167: Allow disabling the GIL

#205

Post by cozzyd »

sneed_chucker wrote: post #100
markhahn wrote: post #68

Earlier quoted context omitted.

You seem to be implying that there is something inherently slow to Python. What? This topic is an example: a detail of one particular implementation, since GIL is definitely not inherent to the language. Just the usual worry about looseness of types?

CPython is slow. That's not really something you can dispute. It is a non-optimizing bytecode interpreter and it makes no use of JIT compilation. JavaScript with V8 or any other modern JIT JS engine runs circles around it. Go, Java, and C# are an order of magnitude faster but they have type systems that make optimizing compilation much easier. There's no language-inherent reason why Python can't be at least as fast a…

JavaScript doesn't have to contend with a plethora of native extensions (which, to be fair, are generally a workaround for python slowness).

Re: gh-116167: Allow disabling the GIL

#206

Post by takeda »

purpleidea wrote: post #200

It's neat that this will eventually improve some python code, but at the end of the day, it's still a badly typed language, which will still be slower and less safe than more modern languages. Learn golang or rust instead. Impressive that they are managing this though!

Rust, sure, but typing in golang isn't superior to python's type annotations.

Also Rust became a quite popular tool for python extensions, where you can offload performance to rust and business logic to python.

Re: gh-116167: Allow disabling the GIL

#207

Post by [deleted] »

colatkinson wrote: post #162
ptx wrote: post #73

Earlier quoted context omitted.

Why is it a mess? What's wrong with it on Windows?

Adding on to the other comment, multiprocessing is also kinda broken on Linux/Mac. 1. Because global objects are refcounted, CoW effectively isn't a thing on Linux. They did add a way to avoid this [0], but you have to manually call it once your main imports are done. 2. On Mac, turns out a lot of the system libs aren't actually fork-safe [1]. Since these get imported inadvertently all the time, Python on Mac actuall…

[deleted]

Re: gh-116167: Allow disabling the GIL

#208

Post by fulafel »

colatkinson wrote: post #162
ptx wrote: post #73

Earlier quoted context omitted.

Why is it a mess? What's wrong with it on Windows?

Adding on to the other comment, multiprocessing is also kinda broken on Linux/Mac. 1. Because global objects are refcounted, CoW effectively isn't a thing on Linux. They did add a way to avoid this [0], but you have to manually call it once your main imports are done. 2. On Mac, turns out a lot of the system libs aren't actually fork-safe [1]. Since these get imported inadvertently all the time, Python on Mac actuall…

Re (1), are there publicly documented cases with numbers on observed slowdowns with it?

I see this mentioned from time to time, but intuitively you'd think this wouldn't pose a big slowdown since the system builtin objects would have been allocated at the same time (startup) and densely located on smaller nr of pages. I guess if you have a lot of global state in your app it could be more significant.

Would also be interesting to see a benchmark using hugepages, you'd think this could solve remaining perf problems if they were due to large number of independent CoW page faults.

Re: gh-116167: Allow disabling the GIL

#209

Post by dimitrios1 »

progmetaldev wrote: post #198
dimitrios1 wrote: post #151

Earlier quoted context omitted.

I was doing primarily go development since it was first released up until a few years ago when the pandemic allowed me the opportunity to move into a full time remote gig doing primarily Java development, so I can answer this as I hadn't done Java at that point for over 10 years, so I felt completely new (what Java I did before that, I was mostly trying to not use Java by using play framework or jruby on rails) As so…

I truly appreciate your answer, as someone that primarily does C#, but learned Java first back around 2003 to 2005. More so because my son is going to be learning Java next year in high school, and I know he'll be looking to me for help, and I'd like to at least be at the same language version as he's learning. I've also been looking at Go quite a bit, especially from a lot of commentary even recently on YouTube, as…

No problem. I am also a huge Dream Theater fan.

Re: gh-116167: Allow disabling the GIL

#210

Post by AdamJacobMuller »

purpleidea wrote: post #200

It's neat that this will eventually improve some python code, but at the end of the day, it's still a badly typed language, which will still be slower and less safe than more modern languages. Learn golang or rust instead. Impressive that they are managing this though!

> Learn golang or rust instead.

Bad take. Learn golang and rust and python.

You should use the language which is suited to the task, sometimes that's golang sometimes that's python and sometimes it's rust.

It's impressive that the python team as a whole continues to improve in such big ways after more than 30 years of development. It's more impressive that the python team managed to navigate 2to3 and come out stronger.

Post reply on HN