It's much easier and more accurate to time your python scripts with `time python script.py`. Cool write up.
Making Python faster with Rust
31–40 of 223 posts
Re: Making Python faster with Rust
#32This is a great article but there's still a core problem there - why should developers have to choose between accessibility and performance? So much scientific computing code suffers between core packages being split away from their core language - at what point do we stop and abandon python for languages which actually make sense? Obviously julia is the big example here, but its interest, development and ecosystem d…
Today, there is a Python package for everything. The ecosystem is possibly best in class for having a library available that will do X. You cannot separate the language from the ecosystem. Being better, faster, and stronger means little if I have to write all of my own supporting libraries.
Also, few scientific programmers have any notion of what C or Fortran is under the hood. Most are happy to stand on the shoulders of giants and do work with their specialized datasets. Which for the vast majority of researchers are not big data. If the one-time calculation takes 12 seconds instead of 0.1 seconds is not a problem worth optimizing.
Re: Making Python faster with Rust
#33Earlier quoted context omitted.
OP fully rewrote the example program in rust, by also moving the entire data structures there. This would mean that any interaction with these ndarrays could be possible only on the rust side, hence any other code that uses them must be rewritten, unless there’s some porting of rust ndarrays to python numpy ndarrays
Yes, what did you expect? That he shares his internal code base with the world just to silence people who can’t generalize?
No? In fact, if you understand what what I wrote, by generalizing this small piece of code it would mean that most of the codebase must be also partially rewritten to rust to be able to interoperate with the new data structures moved in rust. Thus these “less than 100 lines of code” refer to just this simple example program, which was fully rewritten in rust, hence, by generalizing, the premise was pointless
Re: Making Python faster with Rust
#34Earlier quoted context omitted.
Yeah what's wrong with that? I think this sounds amazing. It gives you all the fast prototyping and simplicity of Python, but once you hit that bottleneck all you have to do is bring in a ringer to replace key components with a faster language. No need to use Golang or Rust from the start, no need for those resources until you absolutely need the speed improvement. Sounds like a dream to a lot of people who find it m…
Python is a rough language to be productive in. It's a great scratchpad, but dynamic typing, exceptions/poor error handling, and a horrifying deployment and dependency system make me reach for something like Go in any case where I need something to be even vaguely reliable. The more ML I do, the more disappointed I get.
To some degree, it's about familiarity with your tools. And different tools are optimized for different tasks.
Besides, aren't you deploying Docker containers, anyway?
Re: Making Python faster with Rust
#35The most important part of the article seems to be that this Python code is taking "an avg of 293.41ms per iteration": def find_close_polygons( polygon_subset: List[Polygon], point: np.array, max_dist: float ) -> List[Polygon]: close_polygons = [] for poly in polygon_subset: if np.linalg.norm(poly.center - point) And after replacing it with this Rust code, it is taking "an avg of 23.44ms per iteration": use pyo3::pre…
One carries the entire feature set of the python runtime, the other is compiled.
Re: Making Python faster with Rust
#36Earlier quoted context omitted.
Python is a rough language to be productive in. It's a great scratchpad, but dynamic typing, exceptions/poor error handling, and a horrifying deployment and dependency system make me reach for something like Go in any case where I need something to be even vaguely reliable. The more ML I do, the more disappointed I get.
To some degree, it's about familiarity with your tools. And different tools are optimized for different tasks. Besides, aren't you deploying Docker containers, anyway?
I do, absolutely. But it seems like an exceptionally rare practice for most Python code, at least in the ML space, which puts me back at the start.
Re: Making Python faster with Rust
#37>Also, using any JIT-based tricks (PyPy / numba) results in very small gains (as we will measure, just to make sure).
i wasnt able to see the numba comparison. anyone know how much worse it was ?
Re: Making Python faster with Rust
#38The most important part of the article seems to be that this Python code is taking "an avg of 293.41ms per iteration": def find_close_polygons( polygon_subset: List[Polygon], point: np.array, max_dist: float ) -> List[Polygon]: close_polygons = [] for poly in polygon_subset: if np.linalg.norm(poly.center - point) And after replacing it with this Rust code, it is taking "an avg of 23.44ms per iteration": use pyo3::pre…
Python's for loop implementation is slow, also. You can use built in utils like map() which are "native" and can be a lot faster than a for loop with a push:
https://levelup.gitconnected.com/python-performance-showdown...
Re: Making Python faster with Rust
#39I had a similar problem, when I was working as a PhD student a few years ago, where I needed to match the voxel representation of a 3D printer with the tetrahedral mesh of our rendering application.
My first attempt in Python was both prohibitively slow and more complicated than necessary, because I tried to use vectorized numpy, where possible.
Since this was only a small standalone script, I rewrote it in Julia in a day. The end result was ca. 100x faster and the code a lot cleaner, because I could just implement the core logic for one tetrahedron and then use Julia's broadcast to apply it to the array of tetrahedrons.
Anyway, Julia's long startup time often prohibits it from being used inside other languages (even though the Python/Julia interoperability is good). On the contrary the Rust/Python interop presented here seems to be pretty great. Another reason I should finally invest the time to learn Rust.
Re: Making Python faster with Rust
#40The most important part of the article seems to be that this Python code is taking "an avg of 293.41ms per iteration": def find_close_polygons( polygon_subset: List[Polygon], point: np.array, max_dist: float ) -> List[Polygon]: close_polygons = [] for poly in polygon_subset: if np.linalg.norm(poly.center - point) And after replacing it with this Rust code, it is taking "an avg of 23.44ms per iteration": use pyo3::pre…
One carries the entire feature set of the python runtime, the other is compiled.
The time is spent in this 3-line loop:
for poly in polygon_subset:
if np.linalg.norm(poly.center - point)