Why do I need to rewrite in Rust, when I can just use Polars[1] that will cover most usecases [1] https://www.pola.rs/
What is the relation? Polars replaces pandas, not numpy.
Making Python faster with Rust
91–100 of 223 posts
Re: Making Python faster with Rust
#92It's much easier and more accurate to time your python scripts with `time python script.py`. Cool write up.
`time` is absolutely awful, with the minor exception of bsd’s time maybe.
If you’re going to benchmark scripts or executables, use hyperfine.
Re: Making Python faster with Rust
#93I wonder why GraalVM is not more often used for these speed critical cases: https://www.graalvm.org/python/ (Same for ruby https://www.graalvm.org/ruby/ ) Is the problem the Oracle involvement? Or is it not that fast as advertised or problems with the ecosystem (C libraries)?
I would not use anything made by oracle if I have the choice. My team sees it similarly.
Re: Making Python faster with Rust
#94Earlier quoted context omitted.
>Today, there is a Python package for everything . The same could be said about CPAN and NPM. Yet Perl is basically dead and JavaScript isn't used for any machine learning tasks as far as I'm aware. WebAssembly did help bring a niche array of audio and video codecs to the ecosystem[1][2], something I'm yet to see from Python. I don't use Python, but with what little exposure I've had to it at work, its overall sluggi…
That’s a lot of opinions for so little exposure. There are a lot uses that don’t involve docker or a dozen virtual envs.
Honestly, I use python everyday in the ML/AI space. If we're talking in that context they're pretty spot on about python, virtualenvs, and docker.
Re: Making Python faster with Rust
#95Earlier quoted context omitted.
Python's for loop implementation is slow, also. You can use built in utils like map() which are "native" and can be a lot faster than a for loop with a push: https://levelup.gitconnected.com/python-performance-showdown...
Its not the looping itself that is slow in the article you linked, its that every element is appended to the list. If you use a list comprehension its even faster and it still loops over all elements of the list.
Here is the decompilation of the listcomp
[x for x in range(5)]
RESUME 0
BUILD_LIST
LOAD_FAST
FOR_ITER 4
STORE_FAST (x)
LOAD_FAST (x)
LIST_APPEND
JUMP_BACKWARDS 5
RETURN_VALUE
Re: Making Python faster with Rust
#96I think a big mistake in the article, in a context where performance is the main objective, is that the author uses an array of structs (AoS), rather than a struct of arrays (SoA). An SoA makes it so that the data is ordered contiguously, which is easy to read for the CPU, while an AoS structure interleaves different data (namely the x and y in this case), which is very annoying for the CPU. A CPU likes to read chunks of data (for example 128 bits of data/read) and to process these with SIMD instructions, executing a multiple of calculations with one CPU cycle. This is completely broken when using an array of structs.
He uses the same data structure in both the Python and Rust code, so I imagine that he can get an extra 4x speedup at least if he rewrites his code with memory layout in mind.
Re: Making Python faster with Rust
#97A vectorized implementation of find_close_polygons wouldn't be very complex or hard to maintain at all, but the authors would also have to ditch their OOP class based design, and that's the real issue here. The object model doesn't lend itself to performant, vectorized numpy code.
What's a good guide to learn how to make (and see) vectorized code? It's a mindshift and not one I find easy.
Re: Making Python faster with Rust
#98One of the others understated pros of rewriting some parts in Rust, it's that you can parallelize easily with Rayon[0]
Re: Making Python faster with Rust
#99A vectorized implementation of find_close_polygons wouldn't be very complex or hard to maintain at all, but the authors would also have to ditch their OOP class based design, and that's the real issue here. The object model doesn't lend itself to performant, vectorized numpy code.
What's a good guide to learn how to make (and see) vectorized code? It's a mindshift and not one I find easy.
I think a great start is to make arrays of similar data. Instead of an array of (x,y,z) use an array for x, an array for y and another one for z. If you then square these and sum them for example, the compiler might figure out good optimizations for it if you write it as a simple loop.
Also read about SIMD instructions like AVX2. They are often used under the hood when possible, but just knowing what they require can help "triggering" them, depending on which language you use. In C++ for example, the compiler really looks for opportunities to use those instructions. You can tell the compiler did it, by looking in the assembly code if any XMM or YMM registers are being used (these are the names of the SIMD registers).
Re: Making Python faster with Rust
#100I had a similar problem, when I was working as a PhD student a few years ago, where I needed to match the voxel representation of a 3D printer with the tetrahedral mesh of our rendering application. My first attempt in Python was both prohibitively slow and more complicated than necessary, because I tried to use vectorized numpy, where possible. Since this was only a small standalone script, I rewrote it in Julia in…
Julia 1.9 is fast. And you can use https://github.com/Suzhou-Tongyuan/jnumpy to write python extension in Julia now. So I think after 1.9 release julia would be much more usable.