A viable solution for Python concurrency
1–10 of 366 posts
Re: A viable solution for Python concurrency
#2> This "optimization" actually slows single-threaded accesses down slightly, according to the design document, but that penalty becomes worthwhile once multi-threaded execution becomes possible.
My understanding was that CPython viewed any single-threaded performance regression as a blocker to GIL-removal attempts, regardless of if other work by the developer has sped up the interpreter? This article seems to somewhat gloss over that with "it's only small". I'd be interested in knowing other estimations of what the "better-than-average chance" of this (promising sounding) attempt were.
Breaking C extensions (especially the less-conforming ones, which seem likely to be the least maintained) also seems like it would be a very hard pill to swallow, and the sort of thing that might make it a Python 3-to-4 breaking change, which I imagine would also be approached extremely carefully given there are still people to-this-day who believe that python 3 is a mistake and one day everyone will realise it and go back to python 2 (yes, really).
Re: A viable solution for Python concurrency
#3"The biggest source of problems might be multi-threaded programs with concurrency-related bugs that have been masked by the GIL until now."
Re: A viable solution for Python concurrency
#4Or you could just use PyPy, which uses a garbage collector, does more compile-time analysis, and runs much faster.
CPython is a naive interpreter, like original JavaScript. There's been progress since then.
Re: A viable solution for Python concurrency
#5"The biggest source of problems might be multi-threaded programs with concurrency-related bugs that have been masked by the GIL until now."
Yes. I once discovered that CPickle was not thread-safe. The response was that much of the library didn't really work in multi-threaded programs.
Re: A viable solution for Python concurrency
#6Why not using something trio or curio?
They are quite easy to learn, very powerful and have an approach similar to channel in golang.
Re: A viable solution for Python concurrency
#7This is some of the best news I read in a while!
Multiprocessing sort of works but it’s really sucky.
Re: A viable solution for Python concurrency
#8Or you could just use PyPy, which uses a garbage collector, does more compile-time analysis, and runs much faster. CPython is a naive interpreter, like original JavaScript. There's been progress since then.
PyPy being stuck on 3.7 hurts. If 3.8 support comes out soon, I'll be happy to switch for general-purpose work. 3.9 would be even nicer, to support the type annotation improvements. I donate every month, but I'm just an individual donating pocket change; it'd be great to see some corporate support for PyPy.
Re: A viable solution for Python concurrency
#9> This "optimization" actually slows single-threaded accesses down slightly, according to the design document, but that penalty becomes worthwhile once multi-threaded execution becomes possible. My understanding was that CPython viewed any single-threaded performance regression as a blocker to GIL-removal attempts, regardless of if other work by the developer has sped up the interpreter? This article seems to somewha…
It was Guido's requirement that GIL removal not degrade single threaded performance at all, but in the talk I attend at PyCon 2019, the speaker mentioned nothing about qualifications on that. Guido's restriction was presented, quite reasonably, as "no one should have to suffer because of removing the GIL". So a net break-even or performance improvement is fine.
And on top of that, Guido has retired now, and the steering committee may feel differently as long as the spirit of the restrictions is upheld.
Re: A viable solution for Python concurrency
#10Notably the dev proposing this (Sam Gross aka colesbury) is/was a major PyTorch developer, so someone quite familiar with high performance Python and extensions.