Is there a good overview of the bigger picture here?
gh-116167: Allow disabling the GIL
241–250 of 259 posts
Re: gh-116167: Allow disabling the GIL
#242Earlier quoted context omitted.
> Learn golang or rust instead. Bad take. Learn golang and rust and python. You should use the language which is suited to the task, sometimes that's golang sometimes that's python and sometimes it's rust. It's impressive that the python team as a whole continues to improve in such big ways after more than 30 years of development. It's more impressive that the python team managed to navigate 2to3 and come out stronge…
> Learn golang Having to "if err != nil" every single function call is a big put off - imagine having to "try catch" everything in a language like C#!
I love it.
It forces me to think about how my programs can fail and what I should do when they fail.
Re: gh-116167: Allow disabling the GIL
#243Earlier quoted context omitted.
> Learn golang or rust instead. Bad take. Learn golang and rust and python. You should use the language which is suited to the task, sometimes that's golang sometimes that's python and sometimes it's rust. It's impressive that the python team as a whole continues to improve in such big ways after more than 30 years of development. It's more impressive that the python team managed to navigate 2to3 and come out stronge…
Python is a bad programming language, but a great scripting one. You use it when you either have to do scripting or need a framework that is only offered in Python (the entirety of ML domain). Otherwise, the only reason to pick it is if you have no choice, you don't want to do a rewrite or you are being held at a gun point. Instead, pick C#/F#, Kotlin/Clojure or Rust depending on the use case.
Sure, I somewhat agree with what you're saying about python being good for "scripting" and not "programming" but to me, programming == code writing and scripting == code writing == programming.
I definitely agree that libraries can and should drive your language decisions. A 20,000 line golang program might be 10 lines of python because there is a library to do what you need. Similarly a complicated-to-reason-about python program may be made far simpler by using go channels/routines.
Re: gh-116167: Allow disabling the GIL
#244Earlier quoted context omitted.
Same exists with laws. And in most languages if I want to avoid the type system and hand you a goldfish instead of an int, I can. It just may take more effort. Other language will blow up too if you hand them strange things. Just like python those languages have ways to verify you're not passing goldfish, You just may need more or less effort to use them
Thats a lot of words for: yeah, you are right
A lot of words for "yes, but all languages have this problem." The amount of effort to break the type system is the only difference.
Re: gh-116167: Allow disabling the GIL
#245Earlier quoted context omitted.
> Learn golang Having to "if err != nil" every single function call is a big put off - imagine having to "try catch" everything in a language like C#!
I love it. It forces me to think about how my programs can fail and what I should do when they fail.
Being forced to think about every possible failure state is a sure way to spend a lot of time on not creating value for the end user
Re: gh-116167: Allow disabling the GIL
#246Earlier quoted context omitted.
Almost all packages depend on the GIL (at least at first). They assume they can "take the GIL", and then go and look at the various Python datastructures you passed them without worrying about them changing as they are being read. The later, when writing out their answer (which might involve editing something they were given), they can assume they are not changing. For example, you could write code which extends the…
This only applies to native extensions.
Sorry, you are or course correct, I should have said all extensions with native code.
Re: gh-116167: Allow disabling the GIL
#247Earlier quoted context omitted.
Would GIT affect how Python runs across multiple Python processes? I'm asking because I encountered a weird phenomenon before. I use a simple Python lib called "schedule" which is to run some tasks periodically (not precise). And I often run a script multiple times (with different arguments) to monitor something say, every 30 seconds. So they're in three separate Python Interpreter processes. What I've noticed is tha…
The GIL is per interpreter/process, so this wouldn't be related as far as I know. The GIL only really kicks in if you use threads in a single process. Then, the GIL will only let one single thread do actual work at a time, and will trade off which thread gets to do work. The other threads can wait on IO stuff (web requests, the file system, etc) but they can't do number crunching or data processing at the same time.…
I’ve seen this before! Indeed it has nothing to do with Python or the GIL.
The OS scheduler has to execute M threads on N cpu cores, while also balancing competing priorities like latency and power usage.
Because each separate process uses a naive timer, the timings will drift slowly due to imprecision and small process scheduling delays etc.
After enough drift the timers will sync up by chance (harmonics), at which point OS scheduler incentives can lead the timings to “stick” together seemingly.
This may be a bit tangential, but I find it fascinating that there seems to be a mechanical version of this phenomenon with ancient roots - lookup spontaneous synchronization of pendulums.
Re: gh-116167: Allow disabling the GIL
#248Earlier quoted context omitted.
> Python makes it incredibly tough. I disagree, Python makes it incredibly easy to work with threads in many different ways. It just doesn't make threads faster .
In what way? Threading, asyncio, tasks, event loops, multiprocessing, etc. are all complicated and interact poorly if at all. In other languages, these are effectively the same thing, lighter weight, and actually use multicore. If I launch 50 threads with run away while loops in Python, it takes minutes to laumch and barely works after. I can run hundreds of thousands and even millions of runaway processes in Elixir/…
> If I launch 50 threads with run away while loops in Python, it takes minutes to laumch and barely works after. I can run hundreds of thousands and even millions of runaway processes in Elixir/Erlang that launch very fast and processes keep chugging along just fine.
I'm not sure that argument helps your position on threading. I once saw a java program spin off 3000 threads doing god knows what. Debugging the fucking thing was impossible.
Re: gh-116167: Allow disabling the GIL
#249Earlier quoted context omitted.
In what way? Threading, asyncio, tasks, event loops, multiprocessing, etc. are all complicated and interact poorly if at all. In other languages, these are effectively the same thing, lighter weight, and actually use multicore. If I launch 50 threads with run away while loops in Python, it takes minutes to laumch and barely works after. I can run hundreds of thousands and even millions of runaway processes in Elixir/…
> If I launch 50 threads with run away while loops in Python, it takes minutes to laumch and barely works after. I can run hundreds of thousands and even millions of runaway processes in Elixir/Erlang that launch very fast and processes keep chugging along just fine. I'm not sure that argument helps your position on threading. I once saw a java program spin off 3000 threads doing god knows what. Debugging the fucking…
The point there is that processes in Elixir and Erlang are effectively like functions, in that you do not need to "manage" them in any sort of way. They are automatically distributed across all cores, pre-emptively scheduled, killable, have a built-in inbox, etc. One doesn't need to worry about what concurrency library to use nor manually create mailboxes using queues or whatever else. It just works, and you fire them off to do whatever you need. So there is no ceremony. Threads in many other languages and in Python in particular, require a huge amount of ceremony and management.
Re: gh-116167: Allow disabling the GIL
#250Earlier quoted context omitted.
The GIL is per interpreter/process, so this wouldn't be related as far as I know. The GIL only really kicks in if you use threads in a single process. Then, the GIL will only let one single thread do actual work at a time, and will trade off which thread gets to do work. The other threads can wait on IO stuff (web requests, the file system, etc) but they can't do number crunching or data processing at the same time.…
I’ve seen this before! Indeed it has nothing to do with Python or the GIL. The OS scheduler has to execute M threads on N cpu cores, while also balancing competing priorities like latency and power usage. Because each separate process uses a naive timer, the timings will drift slowly due to imprecision and small process scheduling delays etc. After enough drift the timers will sync up by chance (harmonics), at which…
Thank you for your insight!
By the way, I asked about it previously on their repo before, if you're interested.
No replies yet though, since the development of the lib isn't very active to begin with.