Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python extensions should be lazy

gauge.sh

Post reply on HN
Print view View original

11–20 of 65 posts

Re: Python extensions should be lazy

#11

Post by 0x63_Problems »

raymondh wrote: post #8

This is an impressive post showing some nice investigative work that isolates a pain point and produces a performant work-around. However, the conclusion is debatable. Not everyone has this problem. Not everyone would benefit from the same solution. Sure, if your data can be loaded, manipulated, and summarized outside of Python land, then lazy object creation is a good way to go. But then you're giving up all of the…

First off, thank you for all your contributions to Python!

I completely take your point that there are many places where this approach won't fit. It was a surprise for me to trace the performance issue to allocations and GC, specifically because it is rare.

WRT boxing and unboxing, I'd imagine it depends on access patterns primarily - given I was extracting a small portion of data from the AST only once each, it was a good fit. But I can imagine that the boxing and unboxing could be a net loss for more read-heavy use cases.

Re: Python extensions should be lazy

#12

Post by 0x63_Problems »

formerly_proven wrote: post #10

> In the case of ASTs, one could imagine a kind of ‘query language’ API for Python that operates on data that is owned by the extension - analogous to SQL over the highly specialized binary representations that a database would use. This would let the extension own the memory, and would lazily create Python objects when necessary. You could make the API transparently lazy, i.e. ast.parse creates only one AstNode obje…

This seems like it could be implemented without much trouble for consumers, but I actually think for the common case of full AST traversal you'd still want to avoid building objects for the nodes while traversing.

That is to say, ast.NodeVisitor living in Python is part of the problem for use cases like mine. I need the extension to own the traversal as well so that I can avoid building objects except for the result set (which is typically a very small subset). That was what led me to imagine a query-like interface instead, so that Python can give concise traversal instructions.

Re: Python extensions should be lazy

#13

Post by jph00 »

raymondh wrote: post #9
tomjakubowski wrote: post #6

Earlier quoted context omitted.

re: 3, Python has a native numeric array type https://docs.python.org/3/library/array.html

We should probably get rid of that. It is old (predating numpy) and has limited functionality. In almost every case I can think of, you would be better off with numpy.

If you don't want to add a dep on numpy (which is a big complex module) then it's nice to have a stdlib option. So there are certainly at least some cases where you're not better off with numpy.

Re: Python extensions should be lazy

#14

Post by mkoubaa »

stabbles wrote: post #5
lalaland1125 wrote: post #2

Optimizing Python extensions is becoming increasingly important as Python is used in more and more compute intensive environments. The key for optimizing a Python extension is to minimize the number of times you have to interact with Python. A couple of other tips in addition to what this article provides: 1. Object pooling is quite useful as it can significantly cut down on the number of allocations. 2. Be very care…

Why not go all the way and limit the times you have to interact with Python to zero ;)

Because if your users want python, you have to convince them they don't want it. If you fail, you'll have made optimized code that nobody uses.

Another strategy is to actually serve your users

Re: Python extensions should be lazy

#15

Post by coldtea »

raymondh wrote: post #8

This is an impressive post showing some nice investigative work that isolates a pain point and produces a performant work-around. However, the conclusion is debatable. Not everyone has this problem. Not everyone would benefit from the same solution. Sure, if your data can be loaded, manipulated, and summarized outside of Python land, then lazy object creation is a good way to go. But then you're giving up all of the…

>However, the conclusion is debatable. Not everyone has this problem. Not everyone would benefit from the same solution.

Everyone would benefit from developers being more performance minded and not doing uneccesarry work though! Especially Python who has long suffered with performance issues.

Love your work btw!

Re: Python extensions should be lazy

#16

Post by coldtea »

stabbles wrote: post #5
lalaland1125 wrote: post #2

Optimizing Python extensions is becoming increasingly important as Python is used in more and more compute intensive environments. The key for optimizing a Python extension is to minimize the number of times you have to interact with Python. A couple of other tips in addition to what this article provides: 1. Object pooling is quite useful as it can significantly cut down on the number of allocations. 2. Be very care…

Why not go all the way and limit the times you have to interact with Python to zero ;)

Because then you have the warts of the new language, and the pain of migrating code, which could be 100s or millions of lines, to worry about...

Re: Python extensions should be lazy

#17

Post by coldtea »

jph00 wrote: post #13
raymondh wrote: post #9

Earlier quoted context omitted.

We should probably get rid of that. It is old (predating numpy) and has limited functionality. In almost every case I can think of, you would be better off with numpy.

If you don't want to add a dep on numpy (which is a big complex module) then it's nice to have a stdlib option. So there are certainly at least some cases where you're not better off with numpy.

Even better if Python adds a mainline pandas/numpy like C-based table structure, with a very small subset of the pandas/numpy functionality, that's also convertable to pandas/numpy/etc.

Re: Python extensions should be lazy

#18

Post by jhylton »

0x63_Problems wrote: post #11
raymondh wrote: post #8

This is an impressive post showing some nice investigative work that isolates a pain point and produces a performant work-around. However, the conclusion is debatable. Not everyone has this problem. Not everyone would benefit from the same solution. Sure, if your data can be loaded, manipulated, and summarized outside of Python land, then lazy object creation is a good way to go. But then you're giving up all of the…

First off, thank you for all your contributions to Python! I completely take your point that there are many places where this approach won't fit. It was a surprise for me to trace the performance issue to allocations and GC, specifically because it is rare. WRT boxing and unboxing, I'd imagine it depends on access patterns primarily - given I was extracting a small portion of data from the AST only once each, it was…

You could create a custom C type that wrapped an arbitrary AST node and dynamically created values for attributes when you accessed them. The values would also be wrappers around the next AST node, and they could generate new AST nodes on writes. Python objects would be created on traversal, but each one would be smaller. It wouldn’t use Python lists to handle repeated fields It seems like a non-trivial implementation, but not fundamentally hard.

The analogy with numpy doesn’t seem quite right, as Raymond observes, because numpy depends on lots of builtin operations that operate on the underlying data representation. We don’t have any such code for the AST. You’ll still want to write Python code to traverse, inspect, and modify the AST.

Re: Python extensions should be lazy

#19

Post by jay-barronville »

Evan, just a tip…

When linking to code on GitHub in an article like this, for posterity, it’s a good idea to link based on a specific commit instead of a branch.

It might be a good idea to change your link to the `Py_CompileStringObject()` function in CPython’s `Python/pythonrun.c` [0] to a commit-based link [1].

[0]: https://github.com/python/cpython/blob/main/Python/pythonrun...

[1]: https://github.com/python/cpython/blob/967a4f1d180d4cd669d5c...

Re: Python extensions should be lazy

#20

Post by jay-barronville »

alkh wrote: post #7
lalaland1125 wrote: post #2

Optimizing Python extensions is becoming increasingly important as Python is used in more and more compute intensive environments. The key for optimizing a Python extension is to minimize the number of times you have to interact with Python. A couple of other tips in addition to what this article provides: 1. Object pooling is quite useful as it can significantly cut down on the number of allocations. 2. Be very care…

Re: 2, is there any good repo with raw C Python API that can be used as a reference for someone who is not too proficient in C? I took a look at numpy but it seems too complicated for me

You mind elaborating on what exactly you’re looking for? Maybe I can help point you in the right direction, but right now, it’s not clear given your description.
Post reply on HN