We are now one step closer to PythonOTP
gh-116167: Allow disabling the GIL
21–30 of 259 posts
Re: gh-116167: Allow disabling the GIL
#22Re: gh-116167: Allow disabling the GIL
#23While the title is correct, it is a bit misleading, because disabling the GIL breaks the asyncio tests. It's like saying the engine can be removed from my car. Sure, it can, but the car won't work.
Well this release will break any code that uses threads. The goal of this particular release is to work for thread-free programs.
This isn't correct. TFA said that small threaded programs had been run successfully, but that the test suite broke in asyncio.
Async I/O and threads are two different things, and either can be present in real code without the other.
Re: gh-116167: Allow disabling the GIL
#24Although this is nice, the problems with the GIL are often blown out of proportion: people stating that you couldn't do efficient (compute-bounded) multi-processing, which was never the case as the `multiprocessing` module works just fine.
On the other hand, this particular argument also gets overused. Not all compute-bounded parallel workloads are easily solved by dropping into multiprocessing. When you need to share non-trivial data structures between the processes you may quickly run into un/marshalling issues and inefficiency.
Re: gh-116167: Allow disabling the GIL
#25I've been programming in Python for over 6 years now and every week I learn something new. But recently I've been thinking about moving to a more capable language with proper concurrency for backend API requests (FastAPI sucked).
I also want types, so Elixir is not in the picture. I dabbled in Rust a bit. Although I was able to get the hang of things and build a CLI tool pretty quickly, I'm worried I'll have to deal with numerous quirks later if I keep using Rust (like numerous string types). Is that something to be worried about if all I want from Rust is Python+Types+Concurrency?
Re: gh-116167: Allow disabling the GIL
#26It will be very exciting to see how much faster they'll be able to make vanilla python. The value proposition is being challenged by the plethora of tools aiming to alleviate those issues. Speed improvers like Mojo, pytorch, triton, numba, taichi come to mind.
There are so many different attempts at solving this problem that that the last time I wanted to try one of them, I found myself overwhelmed with options. I chose taichi which is pretty fun and easy to use, although somewhat limited in scope.
Re: gh-116167: Allow disabling the GIL
#27Finally, looking forward to the benchmarks of many tools!
Re: gh-116167: Allow disabling the GIL
#28Although this is nice, the problems with the GIL are often blown out of proportion: people stating that you couldn't do efficient (compute-bounded) multi-processing, which was never the case as the `multiprocessing` module works just fine.
multiprocessing only works fine when you're working on problems that don't require 10+ GB of memory per process.
Once you have significant memory usage, you really need to find a way to share that memory across multiple CPU cores. For non-trivial data structures partly implemented in C++ (as optimization, because pure python would be too slow), that means messing with allocators and shared memory. Such GIL-workarounds have easily cost our company several man-years of engineer time, and we still have a bunch of embarrassingly parallel stuff that we still cannot parallelize due to GIL and not yet supporting shared memory allocation for that stuff.
Once the Python ecosystem supports either subinterpreters or nogil, we'll happily migrate to those and get rid of our hacky interprocess code.
Subinterpreters with independent GILs, released with 3.12, theoretically solve our problems but practically are not yet usable, as none of Cython/pybind11/nanobind support them yet. In comparison, nogil feels like it'll be easier to support.
Re: gh-116167: Allow disabling the GIL
#29I've been programming in Python for over 6 years now and every week I learn something new. But recently I've been thinking about moving to a more capable language with proper concurrency for backend API requests (FastAPI sucked). I also want types, so Elixir is not in the picture. I dabbled in Rust a bit. Although I was able to get the hang of things and build a CLI tool pretty quickly, I'm worried I'll have to deal…
Swift or Kotlin might be what you’re looking for but nobody uses Swift for backend really, and I’m not sure about Kotlin.
Re: gh-116167: Allow disabling the GIL
#30I've been programming in Python for over 6 years now and every week I learn something new. But recently I've been thinking about moving to a more capable language with proper concurrency for backend API requests (FastAPI sucked). I also want types, so Elixir is not in the picture. I dabbled in Rust a bit. Although I was able to get the hang of things and build a CLI tool pretty quickly, I'm worried I'll have to deal…
Sounds like you want Julia. It looks like Python, but also has, what you ask for.
You can even run Python from Julia, so that alleviates the problem with a lack of libraries somewhat.