Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CPython Internals Explained

github.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: CPython Internals Explained

#41

Post by shakna »

westurner wrote: post #20
westurner wrote: post #16

vstinner's Python docs; "Unofficial Python Development (Victor's notes) documentation" > Garbage Collector > "Implement the GC protocol in a type": https://pythondev.readthedocs.io/garbage_collector.html#impl... Python Developer's Guide > "CPython's internals": https://devguide.python.org/internals/index.html Python/cpython//InternalDocs/README.md > "CPython Internals Documentation": https://github.com/python/cpython…

IDK why /InternalDocs/ instead of /Doc/internals/ ? ( `ln -s` works with Mac/Lin/ WSL . ) Ideally what's in InternalDocs/ would be built into the docs.python.org docs . Is it just that markdown support in sphinx is not understood to exist? Sphinx has native markdown support. Sphinx does not have native MyST Markdown support. To support MyST Markdown in a sphinx-doc project, you must e.g. `pip install myst_parser` and…

> Ideally what's in InternalDocs/ would be built into the docs.python.org docs .

If you expose internals in documentation, then people depend on internals.

And when you break it, because it isn't meant to be tracked by any kind of API, there are wonderful groups who will sue you (usually under "devaluation").

Re: CPython Internals Explained

#42

Post by shakna »

tonymet wrote: post #17
rich_sasha wrote: post #15

Earlier quoted context omitted.

This seems to be literally looking at the details of the C implementation of a Python interpreter. Exactly specifying the implementation makes sense here. You wouldn't say "how does the C++ compiler work" then look only at gcc.

c++ / g++ is not comparable because the original c++ reference compilers are not commercially popular today. No one is using Strouvestroups compilers. CPython is Python. Every time your buddy says “just download python” you are using CPython . There’s no reason to be pedantic.

g++ and clang are comparable. You need to specify the implementation.

Re: CPython Internals Explained

#43

Post by throwaway81523 »

squeedles wrote: post #21

Had to write a fairly substantial native extension to Python a couple years ago and one of the things I enjoyed was that the details were not easily "Googleable" because implementation results were swamped by language level results. It took me back to the old days of source diving and accumulated knowledge that you carried around in your head. https://www.dave.org/posts/20220806_python/

There's a file on docs.python.org explaining the C api. The rest is pretty straightforward, at least until recently when free threading was introduced (IDK about now). Main hassle is manually having to track reference borrowing etc. Understandable in Python 2, but another tragedy in Python 3.

Re: CPython Internals Explained

#44

Post by westurner »

shakna wrote: post #41
westurner wrote: post #20

Earlier quoted context omitted.

IDK why /InternalDocs/ instead of /Doc/internals/ ? ( `ln -s` works with Mac/Lin/ WSL . ) Ideally what's in InternalDocs/ would be built into the docs.python.org docs . Is it just that markdown support in sphinx is not understood to exist? Sphinx has native markdown support. Sphinx does not have native MyST Markdown support. To support MyST Markdown in a sphinx-doc project, you must e.g. `pip install myst_parser` and…

> Ideally what's in InternalDocs/ would be built into the docs.python.org docs . If you expose internals in documentation, then people depend on internals. And when you break it, because it isn't meant to be tracked by any kind of API, there are wonderful groups who will sue you (usually under "devaluation").

That's why three different procedures for docs?

Python docs procedures: (0) Devguide, (1) PEPs w/ front matter in RST, (2) RST in /Doc with Sphinx, (3) MD and TXT in /InternalDocs without a toctree

The .. warning: or even admonition directives could be used for indicating that docs under /internals are not public API and can change with or without a PEP; though that should also or at least be indicated in the source unless that's a given expectation that not marked public APIs are not to be considered stable

Re: CPython Internals Explained

#45

Post by stonecharioteer »

elcapitan wrote: post #38

This looks quite nice. I always wished there was something like "Ruby Under a Microscope" for Python (and other languages). It was quite instrumental for my deeper understanding of the language.

There is.

https://realpython.com/products/cpython-internals-book/ But it's for 3.9 and doesn't cover the massive changes regarding delayed annotations and the GIL updates

The Ruby under a Microscope guy is updating it.

Re: CPython Internals Explained

#46

Post by shakna »

westurner wrote: post #44
shakna wrote: post #41

Earlier quoted context omitted.

> Ideally what's in InternalDocs/ would be built into the docs.python.org docs . If you expose internals in documentation, then people depend on internals. And when you break it, because it isn't meant to be tracked by any kind of API, there are wonderful groups who will sue you (usually under "devaluation").

That's why three different procedures for docs? Python docs procedures: (0) Devguide, (1) PEPs w/ front matter in RST, (2) RST in /Doc with Sphinx, (3) MD and TXT in /InternalDocs without a toctree The .. warning: or even admonition directives could be used for indicating that docs under /internals are not public API and can change with or without a PEP; though that should also or at least be indicated in the source…

How many, many times has a project said, "Don't use, internal only", only for it to become an industry-wide common "trick"?

Saying "here be dragons" is not enough to discourage people whose job it is to be creative.

Re: CPython Internals Explained

#47

Post by wk_end »

kccqzy wrote: post #37
wk_end wrote: post #8

Earlier quoted context omitted.

V8 itself might not, but, say, Node does and that doesn't torpedo performance. Was Node-API just better designed than Python's FFI?

My understanding is that Node still doesn’t give you low-level C APIs into the language itself. It gives you JavaScript APIs that call into I/O libraries (libuv basically). Python it’s not hard to write a module in pure C that manipulates other Python objects. This means the representation of Python objects has to be stable enough for the C code. V8 does not allow that.

I haven’t tried it myself but I don’t think that’s the case. See the documentation here:

https://nodejs.org/api/n-api.html

I’ve only skimmed this, but it sure sounds like it lets you write C code that operates on JS objects. In fact, it explicitly says “APIs exposed by Node-API are generally used to create and manipulate JavaScript values.”

Re: CPython Internals Explained

#48

Post by rich_sasha »

tonymet wrote: post #17
rich_sasha wrote: post #15

Earlier quoted context omitted.

This seems to be literally looking at the details of the C implementation of a Python interpreter. Exactly specifying the implementation makes sense here. You wouldn't say "how does the C++ compiler work" then look only at gcc.

c++ / g++ is not comparable because the original c++ reference compilers are not commercially popular today. No one is using Strouvestroups compilers. CPython is Python. Every time your buddy says “just download python” you are using CPython . There’s no reason to be pedantic.

If you know enough about Python to look at how the dict is implemented, you also know the difference between Python and CPython. It's not a beginners intro.

Re: CPython Internals Explained

#49

Post by elcapitan »

stonecharioteer wrote: post #45
elcapitan wrote: post #38

This looks quite nice. I always wished there was something like "Ruby Under a Microscope" for Python (and other languages). It was quite instrumental for my deeper understanding of the language.

There is. https://realpython.com/products/cpython-internals-book/ But it's for 3.9 and doesn't cover the massive changes regarding delayed annotations and the GIL updates The Ruby under a Microscope guy is updating it.

That's nice too, but it seems to be more a tour of the code base, and doesn't have the detailled diagrams of memory layout that the Ruby book and the one posted here have.

Re: CPython Internals Explained

#50

Post by simonw »

brcmthrowaway wrote: post #18
hmry wrote: post #7

Earlier quoted context omitted.

> Edit: Python has no JIT In 3.14 and up you can enable JIT by setting the env var PYTHON_JIT=1

Who made this JIT? FAcebook?

Facebook engineers (most notably Sam Gross) contributed a lot of the no-GIL work: https://lwn.net/Articles/939981/

Much of the initial work on the JIT came from Microsoft's Faster CPython team: https://lwn.net/Articles/1029307/

Post reply on HN