Extending Python with Rust
maxwellrules.com
Extending Python with Rust
1–10 of 88 posts
Re: Extending Python with Rust
#2I implemented a well known curve simplification algorithm in Rust, and was pleasantly surprised how easy the interaction with Python was. For packaging, setuptools_rust was great, and I too used PyO3 for the bindings. I haven't tried Rumpy yet, but it looks interesting.
Re: Extending Python with Rust
#3> What's more interesting is that the Rust implementation is just a factor of 1.23 slower (for large arrays) than just using Numpy
I suppose what is meant is faster (also follows from the diagram?). But it is still not a dramatic gain for many use cases. This shows how non-trivial the python performance calculus: pure python, versus numpy python, versus compiled c/c++ or rust. People who want to speed up python should really look whether numpy helps before complicating their codebase more.
But there are more benefits to those bindings besides performance so its really nice to see the expanding options
Re: Extending Python with Rust
#4> What's more interesting is that the Rust implementation is just a factor of 1.23 slower (for large arrays) than just using Numpy I suppose what is meant is faster (also follows from the diagram?). But it is still not a dramatic gain for many use cases. This shows how non-trivial the python performance calculus: pure python, versus numpy python, versus compiled c/c++ or rust. People who want to speed up python shoul…
The diagram shows Numpy (orange line) below "Rumpy" (blue line). Since the y axis is time, less is more, so Numpy is faster indeed.
Re: Extending Python with Rust
#5> What's more interesting is that the Rust implementation is just a factor of 1.23 slower (for large arrays) than just using Numpy I suppose what is meant is faster (also follows from the diagram?). But it is still not a dramatic gain for many use cases. This shows how non-trivial the python performance calculus: pure python, versus numpy python, versus compiled c/c++ or rust. People who want to speed up python shoul…
Numpy is often faster because it’s often using highly optimized simd, or makes use of BLAS/Fortran/LAPACK/MKL/CuBLAS implementations.
A pure rust implementation will likely always be slower by virtue of not using the same tightly designed optimized code.
Side note: A fun implementation detail of numpy is that after you install it from pypi, it does a user side compile of some of the modules on first import. Which means you need to be somewhat careful if you ever relocate an install of it to a new machine
Re: Extending Python with Rust
#6> What's more interesting is that the Rust implementation is just a factor of 1.23 slower (for large arrays) than just using Numpy I suppose what is meant is faster (also follows from the diagram?). But it is still not a dramatic gain for many use cases. This shows how non-trivial the python performance calculus: pure python, versus numpy python, versus compiled c/c++ or rust. People who want to speed up python shoul…
Numpy is often faster because it’s often using highly optimized simd, or makes use of BLAS/Fortran/LAPACK/MKL/CuBLAS implementations. A pure rust implementation will likely always be slower by virtue of not using the same tightly designed optimized code. Side note: A fun implementation detail of numpy is that after you install it from pypi, it does a user side compile of some of the modules on first import. Which mea…
And note that for example on Apple M1 it's essentially impossible to beat an implementation that uses Apple's Accelerate library for things like matrix multiplication, because Apple uses undocumented instructions unavailable to the public in that library.
Re: Extending Python with Rust
#7> What's more interesting is that the Rust implementation is just a factor of 1.23 slower (for large arrays) than just using Numpy I suppose what is meant is faster (also follows from the diagram?). But it is still not a dramatic gain for many use cases. This shows how non-trivial the python performance calculus: pure python, versus numpy python, versus compiled c/c++ or rust. People who want to speed up python shoul…
Numpy is often faster because it’s often using highly optimized simd, or makes use of BLAS/Fortran/LAPACK/MKL/CuBLAS implementations. A pure rust implementation will likely always be slower by virtue of not using the same tightly designed optimized code. Side note: A fun implementation detail of numpy is that after you install it from pypi, it does a user side compile of some of the modules on first import. Which mea…
Interesting! Using what compiler?
Re: Extending Python with Rust
#8Meta comment: “… with Rust” feels like a HN meme to me.
I know nothing about Rust, but I’ve noticed that there’s a top post on HN every day that followed this format.
Re: Extending Python with Rust
#9Earlier quoted context omitted.
Numpy is often faster because it’s often using highly optimized simd, or makes use of BLAS/Fortran/LAPACK/MKL/CuBLAS implementations. A pure rust implementation will likely always be slower by virtue of not using the same tightly designed optimized code. Side note: A fun implementation detail of numpy is that after you install it from pypi, it does a user side compile of some of the modules on first import. Which mea…
Interesting! Using what compiler?
I believe it looks to see what’s available and otherwise falls back to less efficient implementations.
Re: Extending Python with Rust
#10This is great but you should be aware of the cost. Distributing binary wheels for every available platform (including ARM, MacOS, ...) and implementation (PyPy...) is not easy, and not doing it causes really abysmal user experience (doing `pip install requests` and being told you need to install a Rust toolchain to build `cryptography`). Sometimes the performance might not be worth it.
Thankfully there are GitHub Actions and similar tools that help with this.