Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extending Python with Rust

maxwellrules.com

Post reply on HN
Print view View original

71–80 of 88 posts

Re: Extending Python with Rust

#71

Post by andrewshadura »

smcl wrote: post #46
ryantgtg wrote: post #27

Earlier quoted context omitted.

Yeah, without knowing much about Rust, I’ve decided from these types of posts that I should gravitate toward “app I use but now it’s in Rust.” I use plenty of Rust programs simply because people seem excited about it.

Yeah that's one genre of these posts - someone sharing their experience (re)writing *nix utils like cd, ls, grep etc in Rust. But as a project I think it's unrealistic to expect they'll supplant those utilities with an identical Rust-based one, there's often not much to be gained for the risk of potentially breaking a bunch of important stuff in your system by introducing a bit of an unknown quantity, and the effort…

You cannot write cd in Rust.

Re: Extending Python with Rust

#72

Post by notemaker »

packetlost wrote: post #66
notemaker wrote: post #42

Earlier quoted context omitted.

I've written a couple of C-extensions for a Python service that needed to run in real-time. We started out 100x slower than real-time and managed to make it slightly faster than real-time in a hectic couple of weeks. The main bottleneck that C-extensions solved for us then was marshaling. Instead of doing something like Numpy --> custom algorithm --> Numpy --> custom algorithm in Python, we moved it all to C. This wa…

It's not really meaningful to say "100x slower than real-time" or "slightly faster than real-time", "real-time" isn't some threshold of sub-second precision. It usually refers to relative (though maybe absolute wall-clock) time of execution for instructions that are typically controlling hardware that needs precise timings (ie. the time between turning this thing on and turning it back off needs to be exactly 150us).

It is meaningful, but maybe I didn't give enough context.

The service ingests data from a timespan T spanning M minutes. Before the next timespan T+1 is ingested, the service has to process the data from timespan T in <M minutes otherwise we fall behind. What I mean with "faster than real-time" is that it has to process the data faster than it is ingested.

Re: Extending Python with Rust

#73

Post by steveklabnik »

andrewshadura wrote: post #71
smcl wrote: post #46

Earlier quoted context omitted.

Yeah that's one genre of these posts - someone sharing their experience (re)writing *nix utils like cd, ls, grep etc in Rust. But as a project I think it's unrealistic to expect they'll supplant those utilities with an identical Rust-based one, there's often not much to be gained for the risk of potentially breaking a bunch of important stuff in your system by introducing a bit of an unknown quantity, and the effort…

You cannot write cd in Rust.

I mean, if your shell is written in Rust...

https://github.com/nushell/nushell/blob/768ff47d28ec587b689a...

Re: Extending Python with Rust

#74

Post by the__alchemist »

whooie wrote: post #56
the__alchemist wrote: post #41

I think the trouble comes when moving beyond numpy. I'm working on understanding chemistry more, and how to approximate wavefunctions for multi-particle systems. I wrote a Python script a few weeks ago that compares measured vs calculated psi'', to assess how good a trial WF is. Plotting 2/3 of the dimensions using a Matplotlib surface plot. The surface plot brings my computer to a crawl. And, I'm not sure how to mak…

What did you use for your plotter? I'm in science as well and I'm shopping around for a good Rust library to use for just throwing together a quick plot of some simulation data. Currently I generate my data in Rust, write it to a numpy format, and look at it later with matplotlib, but as you've said, 3D plots usually end up looking like a slideshow.

I made my own, using WGPU and this basic graphics engine: https://github.com/David-OConnor/graphics_wgpu

Note that there's currently no docs or template/examples, and I'm rapidly breaking the API.

The surface plots are just meshes of a grid divided into triangles. When I need to manipulate them, I re-gen the meshes from a nested 3D array.

So, not a plotting lib at all; a flexible 3d and UI lib. Could probably made usable by others with a basic example of how you interact with the render and UI.

Have you tried the Plotter lib? I haven't used it, but looks nice from a skim. https://plotters-rs.github.io/book/basic/draw_3d_plots.html

Re: Extending Python with Rust

#75

Post by tmtvl »

jeroenhd wrote: post #68
smcl wrote: post #46

Earlier quoted context omitted.

Yeah that's one genre of these posts - someone sharing their experience (re)writing *nix utils like cd, ls, grep etc in Rust. But as a project I think it's unrealistic to expect they'll supplant those utilities with an identical Rust-based one, there's often not much to be gained for the risk of potentially breaking a bunch of important stuff in your system by introducing a bit of an unknown quantity, and the effort…

> As I said, not a Rust dev and frankly I'm quite intimidated by all the rants people had about fighting with "The Borrow Checker" which sounds like a ferocious Elden Ring boss. But I am tempted by the way it allows safe (or safer) software to be written. In my experience, those "fighting" the borrow checker are often novices misunderstanding the language model or lacking the experience necessary to effectively use i…

Chapter 17 (fearless conspiracy)

That sounds like a very different kind of book.

OT: I quite like the concept of the borrow checker saving me from shooting myself in the foot, it's just a shame that Rust has traits, without those it might have been a nice language to use.

Re: Extending Python with Rust

#76

Post by Matumio »

amelius wrote: post #40

To be honest, that Rust syntax looks pretty horrible for something as simple as multiplying a vector by a scalar.

Part of that is the ndarray crate, which IMO is too generic often, making syntax complicated. The nalgebra crate is a bit nicer IMO, example:

https://github.com/martinxyz/progenitor/blob/85260/crates/pr...

Re: Extending Python with Rust

#77

Post by estebank »

tmtvl wrote: post #75
jeroenhd wrote: post #68

Earlier quoted context omitted.

> As I said, not a Rust dev and frankly I'm quite intimidated by all the rants people had about fighting with "The Borrow Checker" which sounds like a ferocious Elden Ring boss. But I am tempted by the way it allows safe (or safer) software to be written. In my experience, those "fighting" the borrow checker are often novices misunderstanding the language model or lacking the experience necessary to effectively use i…

Chapter 17 (fearless conspiracy) That sounds like a very different kind of book. OT: I quite like the concept of the borrow checker saving me from shooting myself in the foot, it's just a shame that Rust has traits, without those it might have been a nice language to use.

What do you dislike of Rust's traits?

Re: Extending Python with Rust

#78

Post by VWWHFSfQ »

orangepurple wrote: post #33
waynehn wrote: post #31

This is truly a fantastic combination -- implement the logic in Rust and use it in Python. GreptimeDB also implements a similar functionality that allows writing Python script to do post-process of SQL query results, with the help of RustPython and Arrow. Maybe this combination can bring a sweet point between performance and efficiency. docs: https://docs.greptime.com/user-guide/coprocessor-and-scripti... code: https…

Python 3 can be executed server side by Postgres as well CREATE FUNCTION pymax (a integer, b integer) RETURNS integer AS $$ if a > b: return a return b $$ LANGUAGE plpython3u; https://www.postgresql.org/docs/current/plpython.html

Python running in Postgres sounds like it would be unusably slow. What's the performance like?

Re: Extending Python with Rust

#79

Post by tmtvl »

estebank wrote: post #77
tmtvl wrote: post #75

Earlier quoted context omitted.

Chapter 17 (fearless conspiracy) That sounds like a very different kind of book. OT: I quite like the concept of the borrow checker saving me from shooting myself in the foot, it's just a shame that Rust has traits, without those it might have been a nice language to use.

What do you dislike of Rust's traits?

That they enable code looking like:

  users.remove(expired_user);
Which gives me flashbacks to Java and makes me want to saw open my skull and pour bleach over my brains. users is data, it should not contain logic.

Re: Extending Python with Rust

#80

Post by pmeira »

kzrdude wrote: post #36
jacquesm wrote: post #21

Earlier quoted context omitted.

NumPy is insanely fast for most use cases, I've (grudgingly) come to the conclusion that if NumPy doesn't hack it then I should re-think the problem or my way of solving it rather than to try to optimize that particular bit of code if it isn't meant for something that is going to be run in production on a large number of machines. Likely there are better uses of my time. It's interesting how what is nominally a scrip…

How do you handle it in numpy if you want something like ((x - y)/2 + z) * w all elementwise over the arrays? Naively that's 3 intermediate unused arrays being created.

You could use NumExpr: https://github.com/pydata/numexpr#what-is-numexpr
Post reply on HN