21–30 of 65 posts
Evan, just a tip… When linking to code on GitHub in an article like this, for posterity, it’s a good idea to link based on a specific commit instead of a branch. It might be a good idea to change your link to the `Py_CompileStringObject()` function in CPython’s `Python/pythonrun.c` [0] to a commit-based link [1]. [0]: https://github.com/python/cpython/blob/main/Python/pythonrun... [1]: https://github.com/python/cpyth…
Optimizing Python extensions is becoming increasingly important as Python is used in more and more compute intensive environments. The key for optimizing a Python extension is to minimize the number of times you have to interact with Python. A couple of other tips in addition to what this article provides: 1. Object pooling is quite useful as it can significantly cut down on the number of allocations. 2. Be very care…
I have always loved how the trick to making Python better eventually comes down to not writing Python.
Evan, just a tip… When linking to code on GitHub in an article like this, for posterity, it’s a good idea to link based on a specific commit instead of a branch. It might be a good idea to change your link to the `Py_CompileStringObject()` function in CPython’s `Python/pythonrun.c` [0] to a commit-based link [1]. [0]: https://github.com/python/cpython/blob/main/Python/pythonrun... [1]: https://github.com/python/cpyth…
Thanks for pointing this out, the link is now updated!
Side note: Your tool, Tach, seems interesting…you might want to ask @dang [0] via email [1] if he’d be willing to add your submission [2] to the second-chance pool [3] (maybe also provide a clearer and more technical explanation of the tool and its key features).
[0]: https://news.ycombinator.com/user?id=dang
[1]: mailto:hn@ycombinator.com
Optimizing Python extensions is becoming increasingly important as Python is used in more and more compute intensive environments. The key for optimizing a Python extension is to minimize the number of times you have to interact with Python. A couple of other tips in addition to what this article provides: 1. Object pooling is quite useful as it can significantly cut down on the number of allocations. 2. Be very care…
re: 3, Python has a native numeric array type https://docs.python.org/3/library/array.html
Earlier quoted context omitted.
First off, thank you for all your contributions to Python! I completely take your point that there are many places where this approach won't fit. It was a surprise for me to trace the performance issue to allocations and GC, specifically because it is rare. WRT boxing and unboxing, I'd imagine it depends on access patterns primarily - given I was extracting a small portion of data from the AST only once each, it was…
You could create a custom C type that wrapped an arbitrary AST node and dynamically created values for attributes when you accessed them. The values would also be wrappers around the next AST node, and they could generate new AST nodes on writes. Python objects would be created on traversal, but each one would be smaller. It wouldn’t use Python lists to handle repeated fields It seems like a non-trivial implementatio…
When I referenced numpy, I was thinking of a query layer which could push traversal into the extension as well. Something that could have given me “.select(ast.Import).all()”, which in my head is kind of like doing a filtered sum in numpy.
Very cool to get your thoughts on this, thanks for making an account :)
Optimizing Python extensions is becoming increasingly important as Python is used in more and more compute intensive environments. The key for optimizing a Python extension is to minimize the number of times you have to interact with Python. A couple of other tips in addition to what this article provides: 1. Object pooling is quite useful as it can significantly cut down on the number of allocations. 2. Be very care…
Re: 2, is there any good repo with raw C Python API that can be used as a reference for someone who is not too proficient in C? I took a look at numpy but it seems too complicated for me
Earlier quoted context omitted.
Totally agree, keeping the interface with the extension as thin as possible makes sense. I hadn't considered object pooling in this context, it might be more involved since each node has distinct data but for my use case it might still be a performance win. Have you ever used pyo3 for rust bindings? I haven't measured the overhead but I have been assuming that it's worth the tradeoff vs. rolling my own. (I'm the auth…
My last workplace used pyo3 for a project. It was slower than vanilla Python, and you picked up all the normal compiled-language problems like slow builds and cross-compilation toolchains. I wouldn't take away from that observation that pyo3 is slow (it was just a poor fit; FFI for miniscule amounts of work), but the fact that the binding costs were higher than vanilla Python computations suggests that the overhead i…
Earlier quoted context omitted.
If you don't want to add a dep on numpy (which is a big complex module) then it's nice to have a stdlib option. So there are certainly at least some cases where you're not better off with numpy.
Even better if Python adds a mainline pandas/numpy like C-based table structure, with a very small subset of the pandas/numpy functionality, that's also convertable to pandas/numpy/etc.
"Providing a quality implementation of matrix multiplication is highly non-trivial. Naive nested loop implementations are very slow and shipping such an implementation in CPython would just create a trap for users. But the alternative – providing a modern, competitive matrix multiply – would require that CPython link to a BLAS library, which brings a set of new complications. In particular, several popular BLAS libraries (including the one that ships by default on OS X) currently break the use of multiprocessing."
Optimizing Python extensions is becoming increasingly important as Python is used in more and more compute intensive environments. The key for optimizing a Python extension is to minimize the number of times you have to interact with Python. A couple of other tips in addition to what this article provides: 1. Object pooling is quite useful as it can significantly cut down on the number of allocations. 2. Be very care…
Earlier quoted context omitted.
If you don't want to add a dep on numpy (which is a big complex module) then it's nice to have a stdlib option. So there are certainly at least some cases where you're not better off with numpy.
Even better if Python adds a mainline pandas/numpy like C-based table structure, with a very small subset of the pandas/numpy functionality, that's also convertable to pandas/numpy/etc.