Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extending Python with Rust

maxwellrules.com

Post reply on HN
Print view View original

51–60 of 88 posts

Re: Extending Python with Rust

#51

Post by zozbot234 »

kzrdude wrote: post #36
jacquesm wrote: post #21

Earlier quoted context omitted.

NumPy is insanely fast for most use cases, I've (grudgingly) come to the conclusion that if NumPy doesn't hack it then I should re-think the problem or my way of solving it rather than to try to optimize that particular bit of code if it isn't meant for something that is going to be run in production on a large number of machines. Likely there are better uses of my time. It's interesting how what is nominally a scrip…

How do you handle it in numpy if you want something like ((x - y)/2 + z) * w all elementwise over the arrays? Naively that's 3 intermediate unused arrays being created.

Isn't that what Theano/Aesara is for? Still based on a NumPy interface AIUI, but automagically compiling from NumPy-based to efficient code on the CPU or GPU.

Re: Extending Python with Rust

#52

Post by vlmutolo »

rpep wrote: post #12

I've been through the journey with compiling extensions to Python in lower languages (primarily C and C++) and it's really worth evaluating the benefit. In general, the control that it gives you over memory layout and chaining of operations make it very difficult to say from toy examples what benefit you'll get. If you test a single array operation with NumPy then it will compare favourably to a low level implementat…

> but where it generally will lose out is when you’re chaining together multiple operations which a compiler will vectorise more efficiently

I think the bigger issue is remaining cache-friendly in the face of multiple operations. If you do the operations one at a time, you’re doing multiple passes over the data.

In any case, the approach that Polars[0] took seems like a pretty good solution, if difficult to implement well.

Polars is a dataframe library, and has a “lazy” API. This builds up a graph of operations, runs it through a query optimizer, and then executes them all at once. This allows for parallelization, optimizing expressions, and fewer passes over the data. The downside is that it’s nontrivial to implement a query optimizer.

[0]: https://pola.rs/

Re: Extending Python with Rust

#53

Post by warangal »

Compiling extensions to python from another language also provide opportunities to fuse two or more operations, resulting in speed up even for cases where fast but independent implementation of operations are already available in python. And may be really worth it if that functionality is needed frequently. I have been writing python extensions[0] using Nim language for a while now and it has been a very smooth experience every time.

https://ramanlabs.in/static/blog/Generate_Python_extensions_...

Re: Extending Python with Rust

#54

Post by the__alchemist »

ChrisRackauckas wrote: post #49
the__alchemist wrote: post #47

Earlier quoted context omitted.

I've used it in the past, but not recently. I remember that Diffeq lib, and it being best-in-class! Does it work for 3D (PDEs?) I remember having trouble applying it to that domain earlier, although it was outstanding for ODEs. I think the big issue with Julia here is (other than the improving(?) JIT slowness (IIRC it was faster to compile and run a Rust program than JIT a Julia script), is I'm not sure how I'd inter…

There are some nice tools for 3D PDEs which connect to DiffEq like GridAP ( https://docs.sciml.ai/Gridap/stable/ ) and Ferrite ( https://docs.sciml.ai/Ferrite/stable/ ). PDE tooling is where focus has been moving to as things evolve. As for JIT, just today there was a PR that was merged that makes Julia cache and reuse binaries of packages ( https://github.com/JuliaLang/julia/pull/47184 ). It won't be out until the n…

Great news, and TY for all your passion and work on the diffeq lib and Julia in general!

Re: Extending Python with Rust

#55

Post by ciupicri »

the_mitsuhiko wrote: post #20
remram wrote: post #10

This is great but you should be aware of the cost. Distributing binary wheels for every available platform (including ARM, MacOS, ...) and implementation (PyPy...) is not easy, and not doing it causes really abysmal user experience (doing `pip install requests` and being told you need to install a Rust toolchain to build `cryptography`). Sometimes the performance might not be worth it. Thankfully there are GitHub Act…

I would however make the argument that if you do have a native extension but you have no binary wheel, it's better for that code to be written in Rust than C. Usually for a rust binary extension to build you just need a recent rust compiler on your machine whereas making C/C++ things to work can be a nightmare from you needing to have the right version of cmake, Cons, header files, dependency libraries and more to be…

I tried once building a very simple Rust extension on Windows using the Anaconda Python distribution and had no luck. If I remember correctly it got stuck trying to find the VC++ compiler.

Cython extensions on the other hand work fine with the MinGW GCC compiler provided by Anaconda, although it's pretty old (version 5).

Re: Extending Python with Rust

#56

Post by whooie »

the__alchemist wrote: post #41

I think the trouble comes when moving beyond numpy. I'm working on understanding chemistry more, and how to approximate wavefunctions for multi-particle systems. I wrote a Python script a few weeks ago that compares measured vs calculated psi'', to assess how good a trial WF is. Plotting 2/3 of the dimensions using a Matplotlib surface plot. The surface plot brings my computer to a crawl. And, I'm not sure how to mak…

What did you use for your plotter? I'm in science as well and I'm shopping around for a good Rust library to use for just throwing together a quick plot of some simulation data. Currently I generate my data in Rust, write it to a numpy format, and look at it later with matplotlib, but as you've said, 3D plots usually end up looking like a slideshow.

Re: Extending Python with Rust

#57

Post by [deleted] »

notemaker wrote: post #42
rpep wrote: post #12

I've been through the journey with compiling extensions to Python in lower languages (primarily C and C++) and it's really worth evaluating the benefit. In general, the control that it gives you over memory layout and chaining of operations make it very difficult to say from toy examples what benefit you'll get. If you test a single array operation with NumPy then it will compare favourably to a low level implementat…

I've written a couple of C-extensions for a Python service that needed to run in real-time. We started out 100x slower than real-time and managed to make it slightly faster than real-time in a hectic couple of weeks. The main bottleneck that C-extensions solved for us then was marshaling. Instead of doing something like Numpy --> custom algorithm --> Numpy --> custom algorithm in Python, we moved it all to C. This wa…

[deleted]

Re: Extending Python with Rust

#58

Post by mynameisnull »

ciupicri wrote: post #55
the_mitsuhiko wrote: post #20

Earlier quoted context omitted.

I would however make the argument that if you do have a native extension but you have no binary wheel, it's better for that code to be written in Rust than C. Usually for a rust binary extension to build you just need a recent rust compiler on your machine whereas making C/C++ things to work can be a nightmare from you needing to have the right version of cmake, Cons, header files, dependency libraries and more to be…

I tried once building a very simple Rust extension on Windows using the Anaconda Python distribution and had no luck. If I remember correctly it got stuck trying to find the VC++ compiler. Cython extensions on the other hand work fine with the MinGW GCC compiler provided by Anaconda, although it's pretty old (version 5).

In my experience that is just the silly amount of variables you have to export to use the VC++ compiler and this is why they provide a dedicated terminal prompt for the VC++ compiler with all the right env vars loaded. I've struggled as well automating VC++ compilation in rust as a result.

If you're so inclined to revisit this using mingw with rust you can see how here https://rust-lang.github.io/rustup/installation/windows.html

This is far easier to automate imo

Re: Extending Python with Rust

#59

Post by pastage »

remram wrote: post #10

This is great but you should be aware of the cost. Distributing binary wheels for every available platform (including ARM, MacOS, ...) and implementation (PyPy...) is not easy, and not doing it causes really abysmal user experience (doing `pip install requests` and being told you need to install a Rust toolchain to build `cryptography`). Sometimes the performance might not be worth it. Thankfully there are GitHub Act…

I have patched too many python programs to remove unsupported cryptography imports. That rust recompile/install is not fun.

Re: Extending Python with Rust

#60

Post by the8472 »

dagmx wrote: post #5
college_physics wrote: post #3

> What's more interesting is that the Rust implementation is just a factor of 1.23 slower (for large arrays) than just using Numpy I suppose what is meant is faster (also follows from the diagram?). But it is still not a dramatic gain for many use cases. This shows how non-trivial the python performance calculus: pure python, versus numpy python, versus compiled c/c++ or rust. People who want to speed up python shoul…

Numpy is often faster because it’s often using highly optimized simd, or makes use of BLAS/Fortran/LAPACK/MKL/CuBLAS implementations. A pure rust implementation will likely always be slower by virtue of not using the same tightly designed optimized code. Side note: A fun implementation detail of numpy is that after you install it from pypi, it does a user side compile of some of the modules on first import. Which mea…

If numpy uses runtime detection of available SIMD instructions while rust is only compiled with the x86-64 baseline (which only includes SSE2) then compiling the module with `RUSTFLAGS=-Ctarget-cpu=native` might provide some additional performance gains on number-crunchy code.
Post reply on HN