Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wheels for free-threaded Python now available for psutil

gmpy.dev

Post reply on HN
Print view View original

11–20 of 21 posts

Re: Wheels for free-threaded Python now available for psutil

#11

Post by rogerbinns »

LtWorf wrote: post #10
rogerbinns wrote: post #8

Earlier quoted context omitted.

Note that free threaded compatible doesn't necessarily mean the package supports free threading (concurrent execution), just that it can be loaded into a free threaded interpreter. This is the case with my own package which is on the hugovk list (apsw) which will cause the GIL to be re-enabled if you load it into a free threaded Python. The reason I provide a binary wheel is so that you don't have to keep separate GI…

Why do you provide it at all then if it's not working as intended yet?

As I stated:

> so that you don't have to keep separate GIL full and free threaded interpreters around

It means the user doesn't have to keep two Pythons around, install packages in both of them, etc.

It is also possible with the free threaded Python to keep the GIL disabled even if a package such as mine says it needs the GIL. And my package will indeed work just fine, until you supply it with mutable data and concurrently modify it in another thread.

Re: Wheels for free-threaded Python now available for psutil

#12

Post by cyanydeez »

rogerbinns wrote: post #11
LtWorf wrote: post #10

Earlier quoted context omitted.

Why do you provide it at all then if it's not working as intended yet?

As I stated: > so that you don't have to keep separate GIL full and free threaded interpreters around It means the user doesn't have to keep two Pythons around, install packages in both of them, etc. It is also possible with the free threaded Python to keep the GIL disabled even if a package such as mine says it needs the GIL. And my package will indeed work just fine, until you supply it with mutable data and concur…

Do you atleast warn users? This sounds like madness.

Re: Wheels for free-threaded Python now available for psutil

#13

Post by zahlman »

woodruffw wrote: post #5
semiinfinitely wrote: post #4

It has never been clear to me what the term "wheels" means in software releases. seems like something along the lines of "thing that works" – does it mean anything more specific than that?

It's a Python packaging specific term. A "wheel" is a built distribution, which in the most basic sense just means that it can be unarchived directly into a Python import prefix instead of needing a build step (historically `setup.py`) to prepare it for installation. In practice, many built distributions contain binary artifacts (e.g. builds of CPython extensions). This differentiates them from source distributions,…

> In practice, many built distributions contain binary artifacts (e.g. builds of CPython extensions). This differentiates them from source distributions

This seems like a good opportunity to mention that that the wheel isn't required to contain any non-Python content, and indeed it's strongly preferred to provide one even (I would say "especially") when there is no meaningful "build step" (https://pradyunsg.me/blog/2022/12/31/wheels-are-faster-pure-...). There are of course abandoned legacy projects around from before the option was available, that may never get fixed. But developers of new code in pure Python should ensure they understand the system.

(I would strongly prefer if we could have a single type of package artifact with an optional post-install hook for building any needed non-Python content. Alas....)

Re: Wheels for free-threaded Python now available for psutil

#14

Post by woodruffw »

zahlman wrote: post #13
woodruffw wrote: post #5

Earlier quoted context omitted.

It's a Python packaging specific term. A "wheel" is a built distribution, which in the most basic sense just means that it can be unarchived directly into a Python import prefix instead of needing a build step (historically `setup.py`) to prepare it for installation. In practice, many built distributions contain binary artifacts (e.g. builds of CPython extensions). This differentiates them from source distributions,…

> In practice, many built distributions contain binary artifacts (e.g. builds of CPython extensions). This differentiates them from source distributions This seems like a good opportunity to mention that that the wheel isn't required to contain any non-Python content, and indeed it's strongly preferred to provide one even (I would say "especially") when there is no meaningful "build step" ( https://pradyunsg.me/blog/…

Yeah, I was simplifying. Wheels are in many ways a strictly better alternative to source distributions even in the “pure” case!

Re: Wheels for free-threaded Python now available for psutil

#15

Post by rogerbinns »

cyanydeez wrote: post #12
rogerbinns wrote: post #11

Earlier quoted context omitted.

As I stated: > so that you don't have to keep separate GIL full and free threaded interpreters around It means the user doesn't have to keep two Pythons around, install packages in both of them, etc. It is also possible with the free threaded Python to keep the GIL disabled even if a package such as mine says it needs the GIL. And my package will indeed work just fine, until you supply it with mutable data and concur…

Do you atleast warn users? This sounds like madness.

Yes. The interpreter warns by default, and requires steps to disable the warning. My release notes say that the GIL will be enabled when the package is loaded.

Is it madness that other packages claim they support running without the GIL, yet it is possible to cause corruption and crashes just by writing concurrent Python code? That is the case with the standard library. Compiler thread sanitizers don't work with free threaded Python. Diligent code inspection by humans is the only way to update C code so far.

Free threading is at the beginning of the project. It works. You get slow downs in single threaded performance due to extra locking, and speedups in concurrent performance due to threading. But it is possible to cause corruption and crashes via Python code. Don't expose it to untrusted data and code.

But do investigate it and see what works well and what doesn't. See what code patterns are now possible. Help improve the tools and documentation. It had to start somewhere, and the current state is somewhere.

Re: Wheels for free-threaded Python now available for psutil

#16

Post by LtWorf »

rogerbinns wrote: post #11
LtWorf wrote: post #10

Earlier quoted context omitted.

Why do you provide it at all then if it's not working as intended yet?

As I stated: > so that you don't have to keep separate GIL full and free threaded interpreters around It means the user doesn't have to keep two Pythons around, install packages in both of them, etc. It is also possible with the free threaded Python to keep the GIL disabled even if a package such as mine says it needs the GIL. And my package will indeed work just fine, until you supply it with mutable data and concur…

But the users install the free threaded python to do free threaded stuff. The second they use your package they have a GIL again, which entirely defeats the point.

Wouldn't it be much better to just not support it if it's not supported?

Re: Wheels for free-threaded Python now available for psutil

#17

Post by LtWorf »

rogerbinns wrote: post #15
cyanydeez wrote: post #12

Earlier quoted context omitted.

Do you atleast warn users? This sounds like madness.

Yes. The interpreter warns by default, and requires steps to disable the warning. My release notes say that the GIL will be enabled when the package is loaded. Is it madness that other packages claim they support running without the GIL, yet it is possible to cause corruption and crashes just by writing concurrent Python code? That is the case with the standard library. Compiler thread sanitizers don't work with free…

I think what you are doing is hiding problems. I think crashes and bugs are preferable to find the issues at this point. People who want the safe option will run the regular python.

Re: Wheels for free-threaded Python now available for psutil

#18

Post by rogerbinns »

LtWorf wrote: post #16
rogerbinns wrote: post #11

Earlier quoted context omitted.

As I stated: > so that you don't have to keep separate GIL full and free threaded interpreters around It means the user doesn't have to keep two Pythons around, install packages in both of them, etc. It is also possible with the free threaded Python to keep the GIL disabled even if a package such as mine says it needs the GIL. And my package will indeed work just fine, until you supply it with mutable data and concur…

But the users install the free threaded python to do free threaded stuff. The second they use your package they have a GIL again, which entirely defeats the point. Wouldn't it be much better to just not support it if it's not supported?

What is better:

1) Saying your package supports free threading, but it isn't safe - ie concurrent mutation can result in corruption and crashes

2) Allowing the package to be loaded into a free threaded Python, which immediately enables the GIL. Concurrent mutation does not result in corruption and crashes because of the GIL. The user doesn't have to maintain two Python installations. They can set the environment variable PYTHON_GIL=0 or start Python with -Xgil=0 which will keep the GIL disabled, and they will be fine if they avoid concurrent mutation.

I chose 2. The stdlib json package (along with many others) picked 1. Heck I'll guarantee that most that picked 1 aren't 100% safe either, because doing the changes is hard work, *every* case has to be covered, and tools like thread sanitizers don't work.

The reason I chose 2 is because I care about data integrity. I will eventually reach 1, but only once I can be certain the code is correct.

Re: Wheels for free-threaded Python now available for psutil

#19

Post by rogerbinns »

LtWorf wrote: post #17
rogerbinns wrote: post #15

Earlier quoted context omitted.

Yes. The interpreter warns by default, and requires steps to disable the warning. My release notes say that the GIL will be enabled when the package is loaded. Is it madness that other packages claim they support running without the GIL, yet it is possible to cause corruption and crashes just by writing concurrent Python code? That is the case with the standard library. Compiler thread sanitizers don't work with free…

I think what you are doing is hiding problems. I think crashes and bugs are preferable to find the issues at this point. People who want the safe option will run the regular python.

I provide 3 options:

1) Use regular GIL Python and you get the highest levels of integrity and correctness of operation of my package

2) Use a free threaded Python, the GIL will be enabled at load time, and you get the highest levels of integrity and correctness

3) Use a free threaded Python, and set $PYTHON_GIL=0 or start with -Xgil=0 to keep the GIL disabled, and providing you do not do concurrent mutation of data provided to my package, you get the highest levels of integrity and correctness

BTW I did not randomly choose to provide the free threaded builds. I specifically asked the setuptools maintainers (under the Python Packaging Authority) how to prevent free threaded builds for PyPI. They encouraged me to do the free threaded builds so that a user doesn't to have maintain parallel regular and free threaded Python installations. And it allows option 3 above.

Re: Wheels for free-threaded Python now available for psutil

#20

Post by LtWorf »

rogerbinns wrote: post #18
LtWorf wrote: post #16

Earlier quoted context omitted.

But the users install the free threaded python to do free threaded stuff. The second they use your package they have a GIL again, which entirely defeats the point. Wouldn't it be much better to just not support it if it's not supported?

What is better: 1) Saying your package supports free threading, but it isn't safe - ie concurrent mutation can result in corruption and crashes 2) Allowing the package to be loaded into a free threaded Python, which immediately enables the GIL. Concurrent mutation does not result in corruption and crashes because of the GIL. The user doesn't have to maintain two Python installations. They can set the environment vari…

3) Saying your package doesn't support free threading instead of adding a GIL and force the users to stick to regular python.
Post reply on HN