Live data from Hacker News

Notes from the Meeting on Python GIL Removal Between Python Core and Sam Gross

lukasz.langa.pl

11–20 of 105 posts

Re: Notes from the Meeting on Python GIL Removal Between Python Core and Sam Gross

#11
post #4

Just came in here briefly to opine that there is a very real risk of fork if the Python core community does not at least offer a viable alternative expediently. The economic pressures surrounding the benefits of gross’s changes will likely influence this more than any tears shed over subtle backwards incompatibility. I believe it was Dropbox that famously released their own private internal Python build a while back…

I mean, PyPy is over a decade old now, and micropython is a mere 7 years old. What's another fork? If anything, I strongly prefer languages that have more than one implementation.

I do not disagree with you.

However, I would speculate part of PSF's hesitancy is likely specifically around the perceived violence that gross' "GIL-less" changes may incur to the runtime semantics' backwards compatibility.

PSF in particular has a responsibility here as well I feel in that CPython is arguably the working spec or standard from which these other implementations work and are defined.

Do you also strongly prefer languages with different underlying concurrency semantics? While stackless and pypy etc. are around and available and this could suggest the answer could be "yes" we've been lucky that they haven't fundamentally changed the experience of writing Python.

The possibility that a ton of libraries might now be able to use efficient multi-threaded execution where they were previously constrained to multiprocessing will be a landslide of changes on its own, and likely reminiscent of python 2 -> 3 compatibility if we have to preserve "two ways of doing things."

Re: Notes from the Meeting on Python GIL Removal Between Python Core and Sam Gross

#12
post #2

I can't help but read all the interjections from the core developers as a strong indication of why all these sorts of things tend to fail on the vine. From Unladen Swallow onward there have been these groups off doing interesting and awesome experiments to try and make Python faster, and they never actually make it into something that Python end-users can use (yes, I know US had shortcomings). This is the work of one…

> there is a large number of “dark matter” Python (and C extension) code out there that isn’t open-source. We need to be careful not to break it since it might not be feasible for its users to make required changes, or to report problems back upstream to us. In particular, some C extensions protect their own internal state with the GIL. This is a big worry, and might be a big hindrance to adoption of a GIL-free Python.

Probably doesn't work across minor versions anyway, most stuff isn't built against the limited API.

Re: Notes from the Meeting on Python GIL Removal Between Python Core and Sam Gross

#14
post #9

Earlier quoted context omitted.

> Would you prefer to support some weird arbitrary nameless closed source extensions, or have a multicore Python? There's nothing wrong with staying on an older LTS version of Python. Let the people with the nameless closed-source stuff stick with that. The beauty of open source is that they can fork the older, GIL-ful version of Python and maintain it, if they like. Multicore would be a tremendous boon to the langua…

> Let the people with the nameless closed-source stuff stick with that. Let's hope it becomes a teachable moment.

Is there some particular reason to believe the Python team is even able to get good insight into regressions in the long tail of python packages?

Also it's important to remember that a lot of of material contributions to the community (either to the foundation, via jobs, or even open-sourcing part of their internal stack) might be coming from closed source in some way. It's not wise to ignore that & I think the core team is rightfully cognizant of needing to balance that (balance - not tip to one extreme or the other).

Re: Notes from the Meeting on Python GIL Removal Between Python Core and Sam Gross

#15
post #5
post #4

Just came in here briefly to opine that there is a very real risk of fork if the Python core community does not at least offer a viable alternative expediently. The economic pressures surrounding the benefits of gross’s changes will likely influence this more than any tears shed over subtle backwards incompatibility. I believe it was Dropbox that famously released their own private internal Python build a while back…

> I believe it was Dropbox that famously released their own private internal Python build a while back and included some concurrency patches. Google also had their Unladen Swallow version, but it seems they lost interest at some point.

Unladen Swallow is a 20% project. Besides, LLVM's JIT turns out to be an disappointment and I believe that no modern interpreted language uses LLVM's JIT at this moment.

Re: Notes from the Meeting on Python GIL Removal Between Python Core and Sam Gross

#16
post #5
post #4

Just came in here briefly to opine that there is a very real risk of fork if the Python core community does not at least offer a viable alternative expediently. The economic pressures surrounding the benefits of gross’s changes will likely influence this more than any tears shed over subtle backwards incompatibility. I believe it was Dropbox that famously released their own private internal Python build a while back…

> I believe it was Dropbox that famously released their own private internal Python build a while back and included some concurrency patches. Google also had their Unladen Swallow version, but it seems they lost interest at some point.

"PEP 3146 -- Merging Unladen Swallow into CPython" > Future Work (2010) https://www.python.org/dev/peps/pep-3146/#future-work

Perhaps Google/Grumpy could be updated to compile Python 3.x+ to Go with e.g. the RustPython version of the CPython Python Standard Library modules?

"Inside cpyext: Why emulating CPython C API is so Hard" (2018) https://news.ycombinator.com/item?id=18040664

Today, conda-forge compiles CPython to relocatable platform+architecture-specific binaries with LLVM. https://github.com/conda-forge/python-feedstock/blob/master/...

conda-forge also compiles PyPy Python to relocatable platform+architecture-specific binaries with LLVM. conda-forge/pypy3.6-feedstock (3.7) https://github.com/conda-forge/pypy3.6-feedstock/blob/master...

https://github.com/conda-forge/pypy-meta-feedstock/blob/mast... :

> summary: Metapackage to select pypy as python implementation

Pyodide (JupyterLite) compiles CPython to WASM (or LLVM IR?) with LLVM/emscripten IIRC. Hopefully there's a clear way to implement the new GIL-less multithreading support with Web Workers in WASM, too?

The https://rapids.ai/ org has a bunch a fast Python for HPC and Cloud; with Dask and pick a scheduler. Less process overhead and less need for interprocess locking of memory handles that transgress contexts due to a new GIL removal approach would be even faster than debuggable one process per core Python.

Re: Notes from the Meeting on Python GIL Removal Between Python Core and Sam Gross

#17
post #4

Just came in here briefly to opine that there is a very real risk of fork if the Python core community does not at least offer a viable alternative expediently. The economic pressures surrounding the benefits of gross’s changes will likely influence this more than any tears shed over subtle backwards incompatibility. I believe it was Dropbox that famously released their own private internal Python build a while back…

I mean, PyPy is over a decade old now, and micropython is a mere 7 years old. What's another fork? If anything, I strongly prefer languages that have more than one implementation.

MicroPython isn't really an alternative implementation of Python so much as it is an embedded scripting language that looks pretty much identical to normal Python.

Re: Notes from the Meeting on Python GIL Removal Between Python Core and Sam Gross

#19
post #2

I can't help but read all the interjections from the core developers as a strong indication of why all these sorts of things tend to fail on the vine. From Unladen Swallow onward there have been these groups off doing interesting and awesome experiments to try and make Python faster, and they never actually make it into something that Python end-users can use (yes, I know US had shortcomings). This is the work of one…

If this proof of concept is accepted, Facebook will put more resources to move the project forward.

Re: Notes from the Meeting on Python GIL Removal Between Python Core and Sam Gross

#20
post #9

Earlier quoted context omitted.

> Let the people with the nameless closed-source stuff stick with that. Let's hope it becomes a teachable moment.

Is there some particular reason to believe the Python team is even able to get good insight into regressions in the long tail of python packages? Also it's important to remember that a lot of of material contributions to the community (either to the foundation, via jobs, or even open-sourcing part of their internal stack) might be coming from closed source in some way. It's not wise to ignore that & I think the core…

Concurrency is impossible to prove sound without language-level guarantees.

https://www2.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-...

I'm astounded that a change which will release untold heisenbugs into the wild is being considered. It changes my view of Python. In terms of inducing subtle, silent breakage in existing code, it reminds me of this horrifying change from PHP 8:

https://www.php.net/manual/en/migration80.incompatible.php#m...

Post reply on HN