Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extending Python with Rust

maxwellrules.com

Post reply on HN
Print view View original

41–50 of 88 posts

Re: Extending Python with Rust

#41

Post by the__alchemist »

I think the trouble comes when moving beyond numpy. I'm working on understanding chemistry more, and how to approximate wavefunctions for multi-particle systems. I wrote a Python script a few weeks ago that compares measured vs calculated psi'', to assess how good a trial WF is. Plotting 2/3 of the dimensions using a Matplotlib surface plot.

The surface plot brings my computer to a crawl. And, I'm not sure how to make it interactive. I gave up on it a few days ago.

Could have tried a Rust module, but ended up throwing together a custom Rust plotter in a few hours, with translated Python code. (I already have a basic WGPU-based rendering engine, with EGUI for UI). It runs much faster. They're both imperitive langs, so you can copy and paste, replacing `**` with `pow()`, `np.exp(x)` with `x.exp()` etc. And indexing with x[i][j][k] instead of vectorized numpy ops.

Ie the program compiles and runs in release mode faster than Python/numpy could perform the calcs. And the 3D graphics are smooth instead of a slideshow. And, I can make it real-time interactive since I'm using a low-level lang that integrates with GPU APIs directly. I'm not sure how feasible that would be in Python.

Re: Extending Python with Rust

#42

Post by notemaker »

rpep wrote: post #12

I've been through the journey with compiling extensions to Python in lower languages (primarily C and C++) and it's really worth evaluating the benefit. In general, the control that it gives you over memory layout and chaining of operations make it very difficult to say from toy examples what benefit you'll get. If you test a single array operation with NumPy then it will compare favourably to a low level implementat…

I've written a couple of C-extensions for a Python service that needed to run in real-time. We started out 100x slower than real-time and managed to make it slightly faster than real-time in a hectic couple of weeks.

The main bottleneck that C-extensions solved for us then was marshaling. Instead of doing something like Numpy --> custom algorithm --> Numpy --> custom algorithm in Python, we moved it all to C. This way, we decreased the number of times we had to marshal data & we reduced the amount of data that needed to be marshaled.

IIRC we gained 10-100x improved performance in all scenarios where we used C-extensions. Before doing so we tried to use Numba, as it seemed to offer a "free lunch", but we never got it to work.

The final cherry on top that let us surpass real-time speed was replacing a graph algorithm with another that had a better time complexity for larger graphs. Before doing that switch, we'd still be faster than real-time for _most_ inputs but would lag behind severely whenever we had a dense input graph come in.

Re: Extending Python with Rust

#43

Post by europeanguy »

the__alchemist wrote: post #41

I think the trouble comes when moving beyond numpy. I'm working on understanding chemistry more, and how to approximate wavefunctions for multi-particle systems. I wrote a Python script a few weeks ago that compares measured vs calculated psi'', to assess how good a trial WF is. Plotting 2/3 of the dimensions using a Matplotlib surface plot. The surface plot brings my computer to a crawl. And, I'm not sure how to mak…

Have a look at Julia. Julia is fast, and in addition its differential equations lib is possibly the best in the world.

Re: Extending Python with Rust

#44

Post by korijn »

kzrdude wrote: post #36
jacquesm wrote: post #21

Earlier quoted context omitted.

NumPy is insanely fast for most use cases, I've (grudgingly) come to the conclusion that if NumPy doesn't hack it then I should re-think the problem or my way of solving it rather than to try to optimize that particular bit of code if it isn't meant for something that is going to be run in production on a large number of machines. Likely there are better uses of my time. It's interesting how what is nominally a scrip…

How do you handle it in numpy if you want something like ((x - y)/2 + z) * w all elementwise over the arrays? Naively that's 3 intermediate unused arrays being created.

Inplace operators?

Re: Extending Python with Rust

#45

Post by europeanguy »

kzrdude wrote: post #36
jacquesm wrote: post #21

Earlier quoted context omitted.

NumPy is insanely fast for most use cases, I've (grudgingly) come to the conclusion that if NumPy doesn't hack it then I should re-think the problem or my way of solving it rather than to try to optimize that particular bit of code if it isn't meant for something that is going to be run in production on a large number of machines. Likely there are better uses of my time. It's interesting how what is nominally a scrip…

How do you handle it in numpy if you want something like ((x - y)/2 + z) * w all elementwise over the arrays? Naively that's 3 intermediate unused arrays being created.

You use Julia and let loop fusion handle it. You get 1 allocation if you want to store the result in a new array, or zero allocations if you already have the array allocated.

https://julialang.org/blog/2017/01/moredots/

Re: Extending Python with Rust

#46

Post by smcl »

ryantgtg wrote: post #27
smcl wrote: post #22

Earlier quoted context omitted.

I'm not a Rust developer, but it seems to me that there are very real benefits to using the language. So I think the reason there are so many "I did $task in Rust!" - where $task is something usually accomplished in C, the current lingua franca of lower-level systems programming - is that people are genuinely excited to share that it really is looking possible that Rust can be used in place of C. So it's a bit of a m…

Yeah, without knowing much about Rust, I’ve decided from these types of posts that I should gravitate toward “app I use but now it’s in Rust.” I use plenty of Rust programs simply because people seem excited about it.

Yeah that's one genre of these posts - someone sharing their experience (re)writing *nix utils like cd, ls, grep etc in Rust. But as a project I think it's unrealistic to expect they'll supplant those utilities with an identical Rust-based one, there's often not much to be gained for the risk of potentially breaking a bunch of important stuff in your system by introducing a bit of an unknown quantity, and the effort to do so with any level of quality (and then maintain it) would be pretty massive.

However another genre is more interesting, to me at least - utilising existing interfaces to extend software people already use without having to make them throw out what currently works and take the risk of replacing it with a recently implemented version. In this case someone implemented a Python module in Rust, in others entire Linux device drivers have been implemented in Rust. Lower-level programming in embedded systems seems like a good application too, but I don't know how many architectures Rust can target and is officially supported on.

So to summarise, the two genres[0] I've identified are, roughly:

- I did a RIIR[1] of 50% of grep's functionality for fun

- Here is how you can accomplish a common task in C using Rust instead

As I said, not a Rust dev and frankly I'm quite intimidated by all the rants people had about fighting with "The Borrow Checker" which sounds like a ferocious Elden Ring boss. But I am tempted by the way it allows safe (or safer) software to be written.

[0] - both are valid, and more exist but these are two common ones that came up

[1] - RIIR = "Rewrite It In Rust", which was a bit of a meme for a while as a bunch of devs got excited about Rust and launched projects of varying completeness reimplementing things.

Re: Extending Python with Rust

#47

Post by the__alchemist »

europeanguy wrote: post #43
the__alchemist wrote: post #41

I think the trouble comes when moving beyond numpy. I'm working on understanding chemistry more, and how to approximate wavefunctions for multi-particle systems. I wrote a Python script a few weeks ago that compares measured vs calculated psi'', to assess how good a trial WF is. Plotting 2/3 of the dimensions using a Matplotlib surface plot. The surface plot brings my computer to a crawl. And, I'm not sure how to mak…

Have a look at Julia. Julia is fast, and in addition its differential equations lib is possibly the best in the world.

I've used it in the past, but not recently. I remember that Diffeq lib, and it being best-in-class! Does it work for 3D (PDEs?) I remember having trouble applying it to that domain earlier, although it was outstanding for ODEs.

I think the big issue with Julia here is (other than the improving(?) JIT slowness (IIRC it was faster to compile and run a Rust program than JIT a Julia script), is I'm not sure how I'd interface with the GPU for graphics, compute shaders, and a GUI.

Julia's mathematical syntax is best-in-class; I wish more langs used something like that.

Re: Extending Python with Rust

#49

Post by ChrisRackauckas »

the__alchemist wrote: post #47
europeanguy wrote: post #43

Earlier quoted context omitted.

Have a look at Julia. Julia is fast, and in addition its differential equations lib is possibly the best in the world.

I've used it in the past, but not recently. I remember that Diffeq lib, and it being best-in-class! Does it work for 3D (PDEs?) I remember having trouble applying it to that domain earlier, although it was outstanding for ODEs. I think the big issue with Julia here is (other than the improving(?) JIT slowness (IIRC it was faster to compile and run a Rust program than JIT a Julia script), is I'm not sure how I'd inter…

There are some nice tools for 3D PDEs which connect to DiffEq like GridAP (https://docs.sciml.ai/Gridap/stable/) and Ferrite (https://docs.sciml.ai/Ferrite/stable/). PDE tooling is where focus has been moving to as things evolve.

As for JIT, just today there was a PR that was merged that makes Julia cache and reuse binaries of packages (https://github.com/JuliaLang/julia/pull/47184). It won't be out until the next release of Julia, but it's a pretty major improvement to not JIT fully inferred package calls.

Post reply on HN