Still not encouraged by the no-GIL, "We don't want another Python 2->3 situation", yet very little proffered on how to avoid that scenario. More documentation on writing thread-safe code, suggested tooling to lint for race conditions (for whatever it is worth), discussions with popular C libraries, dedicated support channels for top tier packages, what about the enormous long-tail of abandoned extensions which still work today, etc.
What's up, Python? The GIL removed, a new compiler, optparse deprecated
11–20 of 302 posts
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#12From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?
That was the opinion of a handful of vocal posters. The overhead of using multiprocessing and/or some network service is extremely high for a lot of applications.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#13This is exactly what I have been looking forward to. Allow me to do no-gil, let me the developer make that choice. There are issues with that certainly, but I am conscious of this fact and given an analysis of no-gil benefits it is significantly more beneficial to have no-gil for certain use cases.
One of the most significant of these cases to me is threading outside an Operating system context. What if I want to use both of the cores to a Cortex M0? Multiprocessing can't help me, there are no processes. If I need locking, I will impliment it using the platform I have available to me.
The second is the fact that CPU's are increasingly scaling in core count. When I have to use multiprocessing the program becomes far more complex than one would expect. Why am I doing message passing and shared memory when the OS and CPU supplies better tools already? It also pollutes the process ID's. Imagine if we built every application in Python - there would be hundreds of thousands of individual processes all to use multiple cores. Because this is a problem mostly unique to python we often end up having to build applications in other languages that otherwise would have been better in Python.
I want a world where I can say "just use python" to almost anything. Telling new coders to drop their favorite language and use any other language to get the result they want immedietely kills the innovation they are working on. Instead of spending time creating the idea, they're spending time learning a languages I believe are unnecessary.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#14Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python but of course no-GIL looks interesting for this!
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#15Earlier quoted context omitted.
You don’t have to use separate processes to get the benefit of multithreading in Python today — you can also call into a library written in native code that drops the GIL (e.g. Numpy or Pytorch).
Even then the GIL can cause issues, concerns of PyTorch are specifically one of the motivations of the PEP, and one of the reasons Meta / FB really really wants this: > In PyTorch, Python is commonly used to orchestrate ~8 GPUs and ~64 CPU threads, growing to 4k GPUs and 32k CPU threads for big models. While the heavy lifting is done outside of Python, the speed of GPUs makes even just the orchestration in Python not…
I feel like orchestrating thousands of GPUs is such a niche use case that it’s fair to expect the people wanting to do it to learn a more suited language, rather than ruining Python for everyone else.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#16Will writing multithreaded code become easier? Or will the developer UX remain the same?
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#17From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?
There are plenty of other Python VMs that don't have a GIL and can be used already today, out of the box (examples include Jython and IronPython). Despite that fact - CPython remains the most popular Python VM out there (it utilizes a GIL).
Instead of waiting for the GIL to be removed out of CPython - take your fancy Python code and just run it using a different VM. It's literally as simple as that.
If the GIL was such a bottleneck as people make it out to be - people would move off of CPython a long time ago. But they won't, despite having the options. This only serves to prove that 95%+ of the workflows people build with Python can be satisfied regardless of GIL, often using some of the other parallelism mechanisms available in Python today (multiprocessing, asyncio, etc).
Most of the stuff people build with Python are CRUD apps, Jupyter notebooks, automations, tinkering, small hacks, etc. If you're okay with not utilizing all of your 64k CPUs at home - Python's multiprocessing and asyncio libraries should serve you just fine.
The whole GIL/No-GIL conversation is a complete waste of time and a distraction. People have all the options they need already here and now - but slinging crap at eachother over an issue tracker is so much fun that people can't help it.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#18Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…
> I may have missed something
You did not miss anything. The GIL prevents parallel multi threading.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#19Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…
Similar experience. Even with multi process and threads python is slow, very slow. Java, Go and .NET all provide a very performant out of box experience.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#20Python's been a little too happy to hard-deprecate things for my liking, so this soft-deprecation sounds pretty good.