Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

500 Python Interpreters

izzys.casa

Post reply on HN
Print view View original

31–35 of 35 posts

Re: 500 Python Interpreters

#31

Post by SAHChandler »

wrlanv wrote: post #23
elnatro wrote: post #22

Does this mean that even the Python 3.13 version is around the corner we have to expect some issues when running threads on a program? Maybe I’m slow today, but are the author complaining (rightfully so) about the buggy no-gil implementation? Should we wait until the feature is more stable before implementing new libraries? I was thinking on making use of it in new libraries I plan to develop. What are the most blata…

The article says that embedding (calling Python from C/C++) is currently buggy. It is not clear whether the Python interpreter was compiled with --disable-gil. I did not check the claim, since, as the article rightly points out, there are general issues in the Python space and other languages like Lua are far better for embedding.

It might not have been obvious, but I used the currently available Python 3.12.5 release. However upon filing a ticket, and being asked for more testing, we found this issue persists into 3.13 and 3.14 alpha with and without the GIL disabled :(

Re: 500 Python Interpreters

#32

Post by elnatro »

cclac109 wrote: post #28
elnatro wrote: post #25

Earlier quoted context omitted.

Thanks, I’m not an embedded developer and was losing the train of thought of the author.

These threading issues may also be present when not using embedding. Embedding the interpreter frequently shows general threading bugs much quicker. In other words, when "torturing" the interpreter with embedding a threading bug might show up after 30 min. The same threading bug might show up once every 6 months in a web server (which is a nasty security risk, since it is hard to detect and to reproduce).

Ouch. So we (app developers) are doomed with this feature?

Re: 500 Python Interpreters

#33

Post by SAHChandler »

elnatro wrote: post #32
cclac109 wrote: post #28

Earlier quoted context omitted.

These threading issues may also be present when not using embedding. Embedding the interpreter frequently shows general threading bugs much quicker. In other words, when "torturing" the interpreter with embedding a threading bug might show up after 30 min. The same threading bug might show up once every 6 months in a web server (which is a nasty security risk, since it is hard to detect and to reproduce).

Ouch. So we (app developers) are doomed with this feature?

I did open an issue on GitHub with the CPython implementation and it's being looked into. The crash happens for sure in the debug version of Python. However we're seeing valgrind and address sanitizer fire off for the upcoming 3.13 release in release mode when using the C API as I showed in my example. However, this behavior only appears to occur when embedding python directly, not via extending it, into an application. Because of how the (hidden because it is experimental) `_xxsubinterpreters` module works internally, it's extra internal bookkeeping seems to give enough time for the race condition to disappear.

Re: 500 Python Interpreters

#34

Post by staticautomatic »

curious_cat_163 wrote: post #2

> "No matter what happens there is going to be a lock occurring. No two PyThreadStates can execute Python bytecode at the same time. However, they can execute multiple C calls at the same time which is why for long running pure C operations extension and embedding developers are encouraged to release the GIL temporarily." Very exciting! I wonder what the first set of motivating applications are and what kind of perfo…

Lxml frees the GIL internally and iirc the motivation was something like concurrent reads of different portions of large documents.

Re: 500 Python Interpreters

#35

Post by elnatro »

SAHChandler wrote: post #31
wrlanv wrote: post #23

Earlier quoted context omitted.

The article says that embedding (calling Python from C/C++) is currently buggy. It is not clear whether the Python interpreter was compiled with --disable-gil. I did not check the claim, since, as the article rightly points out, there are general issues in the Python space and other languages like Lua are far better for embedding.

It might not have been obvious, but I used the currently available Python 3.12.5 release. However upon filing a ticket, and being asked for more testing, we found this issue persists into 3.13 and 3.14 alpha with and without the GIL disabled :(

Thanks for your work and for bringing up this issue to them!
Post reply on HN