Faster Parallel Python Without Python Multiprocessing
towardsdatascience.com
Faster Parallel Python Without Python Multiprocessing
1–10 of 30 posts
Re: Faster Parallel Python Without Python Multiprocessing
#2Every time I see a performance issue/solution of Python, I was wondering why there is no company maintaining a distribution with a high-performance Python JIT compiler with patched, GIL-free packages. Given the prevalence of Python and what have succeeded in JVM, it seems like a fruitful business.
Re: Faster Parallel Python Without Python Multiprocessing
#3Every time I see a performance issue/solution of Python, I was wondering why there is no company maintaining a distribution with a high-performance Python JIT compiler with patched, GIL-free packages. Given the prevalence of Python and what have succeeded in JVM, it seems like a fruitful business.
Most of the time, if performance is a top priority, you'll choose a different language to begin with.
It's nice to have the option to speed up python in certain cases though.
Re: Faster Parallel Python Without Python Multiprocessing
#4Every time I see a performance issue/solution of Python, I was wondering why there is no company maintaining a distribution with a high-performance Python JIT compiler with patched, GIL-free packages. Given the prevalence of Python and what have succeeded in JVM, it seems like a fruitful business.
Because Python is not typed?
Re: Faster Parallel Python Without Python Multiprocessing
#5Every time I see a performance issue/solution of Python, I was wondering why there is no company maintaining a distribution with a high-performance Python JIT compiler with patched, GIL-free packages. Given the prevalence of Python and what have succeeded in JVM, it seems like a fruitful business.
The thing is that there's really little benefit from this. Anything performance critical will be written as a native extension anyway, even if Python speeds up by 10x. If you're doing that, you can lift the GIL anyway and run as much in parallel as you want.
Re: Faster Parallel Python Without Python Multiprocessing
#6Every time I see a performance issue/solution of Python, I was wondering why there is no company maintaining a distribution with a high-performance Python JIT compiler with patched, GIL-free packages. Given the prevalence of Python and what have succeeded in JVM, it seems like a fruitful business.
The thing is that there's really little benefit from this. Anything performance critical will be written as a native extension anyway, even if Python speeds up by 10x. If you're doing that, you can lift the GIL anyway and run as much in parallel as you want.
Isn’t that a case of the tail wagging the dog? If Python code were 10x faster, the need to write performance sensitive code in a native extension would be a lot lower. There is a cognitive cost in having to switch between languages and knowing when to switch.
Re: Faster Parallel Python Without Python Multiprocessing
#7Every time I see a performance issue/solution of Python, I was wondering why there is no company maintaining a distribution with a high-performance Python JIT compiler with patched, GIL-free packages. Given the prevalence of Python and what have succeeded in JVM, it seems like a fruitful business.
One problem is that because of the GIL, many python libraries don't use locking on their data, etc., because they don't need to. That makes them thread-unsafe without the GIL.
Re: Faster Parallel Python Without Python Multiprocessing
#8Every time I see a performance issue/solution of Python, I was wondering why there is no company maintaining a distribution with a high-performance Python JIT compiler with patched, GIL-free packages. Given the prevalence of Python and what have succeeded in JVM, it seems like a fruitful business.
There was/is stackeless with a(n initial?) focus on game engines iirc:
https://github.com/stackless-dev/stackless/wiki
It got less attention than pypy, but might be more pragmatic wrt parallel perf. There's also some work in pypy to remove the GIL, but not sure if there's been any news on that lately.
Re: Faster Parallel Python Without Python Multiprocessing
#9Every time I see a performance issue/solution of Python, I was wondering why there is no company maintaining a distribution with a high-performance Python JIT compiler with patched, GIL-free packages. Given the prevalence of Python and what have succeeded in JVM, it seems like a fruitful business.
One problem is that because of the GIL, many python libraries don't use locking on their data, etc., because they don't need to. That makes them thread-unsafe without the GIL.
You mean libraries with c extensions
Re: Faster Parallel Python Without Python Multiprocessing
#10Earlier quoted context omitted.
One problem is that because of the GIL, many python libraries don't use locking on their data, etc., because they don't need to. That makes them thread-unsafe without the GIL.
You mean libraries with c extensions
Yes, getting rid of the GIL breaks most c extensions, many of which are very important.