What's up, Python? The GIL removed, a new compiler, optparse deprecated
1–10 of 302 posts
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#2Summary:
- Python without the GIL, for good
- LPython: a new Python Compiler
- Pydantic 2 is getting usable
- PEP 387 defines "Soft Deprecation", getopt and optparse soft deprecated
- Cython 3.0 released with better pure Python support
- PEP 722 – Dependency specification for single-file scripts
- Python VSCode support gets faster
- Paint in the terminal
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#3From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases.
Is that right?
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#4From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?
I would disagree with that.
The GIL means you can't use Python multithreading in order to take advantage of more CPU time by parallelism. Obviously getting rid of the GIL makes that a real option, just as it is in other languages.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#5From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?
Currently, yes that's kind of true. But it's really only considered obscure because the GIL makes it so you either have to do some weird non thread pattern or go with a different language, and people often go with a different language.
Kind of a Catch-22 of "Well no one uses it that way, so why should we make it possible to use it that way? Well, no one uses it that way because it's impossible to use it that way"
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#6From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?
Well Python doesn't really do proper multi-threading currently thanks to GIL blocking any additional execution threads. So removing it would enable making Python code that is actually multi-threaded without resorting to extra processes and their overhead.
So if you are writing small single process Python script then removing GIL shouldn't really change much. If you are doing some heavier computing or eg. running server back-end, then there are significant performance gains available with this change.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#7From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?
Well Python doesn't really do proper multi-threading currently thanks to GIL blocking any additional execution threads. So removing it would enable making Python code that is actually multi-threaded without resorting to extra processes and their overhead. So if you are writing small single process Python script then removing GIL shouldn't really change much. If you are doing some heavier computing or eg. running serv…
You don’t have to use separate processes to get the benefit of multithreading in Python today — you can also call into a library written in native code that drops the GIL (e.g. Numpy or Pytorch).
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#8From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?
> Maybe for somewhat obscure multithreading cases.
They're only "somewhat obscure" because currently you can't do it at all, so you don't do it and you do something else: it's of value for any case where you're multithreading for computational parallelism (as opposed to IO concurrency). The PEP also outlines a bunch of other situations where using process-based parallelism is problematic: https://peps.python.org/pep-0703/#motivation
With the proviso that while it will work for all pure-python code out of the box[0] loading any native package which has not opted into "no gil" mode will re-enable the GIL: https://peps.python.org/pep-0703/#py-mod-gil-slot
[0] modulo new race conditions where the code relied on the GIL for correctness, something which isn't strictly correct and can already break when the scheduler logic changes
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#9Earlier quoted context omitted.
Well Python doesn't really do proper multi-threading currently thanks to GIL blocking any additional execution threads. So removing it would enable making Python code that is actually multi-threaded without resorting to extra processes and their overhead. So if you are writing small single process Python script then removing GIL shouldn't really change much. If you are doing some heavier computing or eg. running serv…
You don’t have to use separate processes to get the benefit of multithreading in Python today — you can also call into a library written in native code that drops the GIL (e.g. Numpy or Pytorch).
Even then the GIL can cause issues, concerns of PyTorch are specifically one of the motivations of the PEP, and one of the reasons Meta / FB really really wants this:
> In PyTorch, Python is commonly used to orchestrate ~8 GPUs and ~64 CPU threads, growing to 4k GPUs and 32k CPU threads for big models. While the heavy lifting is done outside of Python, the speed of GPUs makes even just the orchestration in Python not scalable. We often end up with 72 processes in place of one because of the GIL. Logging, debugging, and performance tuning are orders-of-magnitude more difficult in this regime, continuously causing lower developer productivity.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#10From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?
That discussion was amusing. Removing the GIL opens up the possibility of actually getting a real performance benefit from multithreaded Python code. That's the value. Given every modern desktop and server is multicore (and increasingly getting to tens of cores if not hundreds), multithreading in Python unhampered by the GIL will be a useful thing. And no, multiprocessing is not a good alternative to multithreading. It's just an alternative, but it's slower, uses more memory, and coordination between processes is slower than between threads.