71–80 of 513 posts
What are advantages of non-GIL Python?
No-GIL means it is possible to run things in parallel (without resorting to fancy C extensions).
Naiive question: Who needs No-GIL when we have asyncio and multiprocessing packages ? never ever had a problem with GIL in python, always found a workaround just by spinning up ThreadPool or ProcessPool, and used async libraries when needed. is there any use case of No-GIL which is not solved by multiprocessing ? I thought Single threaded execution without overhead for concurrency primitives is the best way to high p…
It's only about performance. asyncio is still inherently single-threaded, and hence also single core. multiprocessing is multi-core and hence better for performance, but each process is relatively heavy and there's additional overhead to shared memory. GIL multi-threading is both single-core and difficult to use correctly. No-GIL multi-threading is multi-core, though difficult to use. I don't know the Python implemen…
This design scales much better for systems that will eventually overgrow one big machine. the No-GIL pytohn will be of no use, when you need to deploy your app across 100s machines.
I understand people want to take advantage of all cores etc, but at large scale - you will eventually need to split computation across machines and will resort to in-memory cache/queue anyways - so better just architect your system since day0
Earlier quoted context omitted.
> is there any use case of No-GIL which is not solved by multiprocessing ? Anything that benefits from both parallelism and replacing IPC overhead with shared data between parallel tasks.
but it would incur overhead of concurrency control: mutex, locks, semaphores. I dont believe python will ever have atomic operations, even if it had - they still incur significant overhead for concurrency control. sharing state between threads is such a narow niche use case, this pattern is practically solved by memcached/redis for larger scale python based systems
Earlier quoted context omitted.
> is there any use case of No-GIL which is not solved by multiprocessing ? Anything that benefits from both parallelism and replacing IPC overhead with shared data between parallel tasks.
but it would incur overhead of concurrency control: mutex, locks, semaphores. I dont believe python will ever have atomic operations, even if it had - they still incur significant overhead for concurrency control. sharing state between threads is such a narow niche use case, this pattern is practically solved by memcached/redis for larger scale python based systems
Yes, but there are plenty of problems where that is more development overhead, but less runtime overhead than IPC.
Unpopular opinion: This is a missed opportunity. What? Python could have been the one language with a sane multithreading model. Now it risks becoming a second version of Java. I fear this will make it a less attractive programming language, not least because it might lose its beginner friendlyness. For example, without the GIL a lot more care must be put into designing your programs. This can be true even though you…
> Personally I can see no good will come from bringing free threading to the masses I’ve often thought that proper threading should be learned by the masses, not to be scared of it. This sort of statement does nothing but spread FUD.
It’s like saying that C is a safe language, just “get good” at it.
There are safe alternatives such as structured concurrency.
I hope this won't make Python's dependency hell even worse, but I'm not hopeful.
Earlier quoted context omitted.
but it would incur overhead of concurrency control: mutex, locks, semaphores. I dont believe python will ever have atomic operations, even if it had - they still incur significant overhead for concurrency control. sharing state between threads is such a narow niche use case, this pattern is practically solved by memcached/redis for larger scale python based systems
Fully agree. People put far too much emphasis & expectations on the "free" part in "free multithreading".
at which point you lose benefit of no-GIL, because you now have to introduce redis and kafka into the system
Unpopular opinion: This is a missed opportunity. What? Python could have been the one language with a sane multithreading model. Now it risks becoming a second version of Java. I fear this will make it a less attractive programming language, not least because it might lose its beginner friendlyness. For example, without the GIL a lot more care must be put into designing your programs. This can be true even though you…
I don't quite get your "unpopular opinion". First, I don't see how the GIL would have much to do with free multithreading. The GIL should not have much observable logical impact on multithreaded _Python_ code. It should not make it more or less susceptible to race conditions. It's only practical impact should be slowness. Second, your proposed "one GIL per thread" is pretty much the equivalent of the current state of…
No, no the same thing. Sharing objects between processes is not easily achieved for various reasons (at least in Python). It would be easier to get it in multithreading with an arena based allocation model where objects live in a shared or non-shared area of memory.
Also it's not my idea. I am just advocating it as the better model for Python to advance to. It would also not take years to implement and the risks are minimized as there is full compatibility with existing code.
Unpopular opinion: This is a missed opportunity. What? Python could have been the one language with a sane multithreading model. Now it risks becoming a second version of Java. I fear this will make it a less attractive programming language, not least because it might lose its beginner friendlyness. For example, without the GIL a lot more care must be put into designing your programs. This can be true even though you…
I don't quite get your "unpopular opinion". First, I don't see how the GIL would have much to do with free multithreading. The GIL should not have much observable logical impact on multithreaded _Python_ code. It should not make it more or less susceptible to race conditions. It's only practical impact should be slowness. Second, your proposed "one GIL per thread" is pretty much the equivalent of the current state of…
It has nothing to do with multithreading or GIL, though, and happily works without threads and with GIL in place.