Calling Rust from Python using PyO3

Check out apache arrow in rust for transferring large amounts data between rust and python. With it you can get zero copy transfers between the languages. Afaik there is some performance penalty involved when converting python types to native rust types.

It used to be hard to do such transfers between rust and python using arrow but from v3 or v4, arrow implementation in rust started supporting the c data interface. Creating the array in rust and sending it to python might involve leaking a boxed object though

Check this out if you want to see how https://github.com/jhoekx/python-rust-arrow-interop-example/...

I've been playing around with Rust, PyO3 and Blender the last few days and it has been really nice to use.

Not disagreeing that cargo-raze in bazel is awkward, but just wanted to add:

Cargo raze should let you consume libraries you've imported in a rust_binary target, if that's what you're talking about. It's also possible, but slightly more annoying, to vendor the dependency tree if you need to rely on pulling from a local mirror.

I've also used it to import cargo binaries (i.e wasm-bindgen-cli) into a Bazel workspace -- it just makes the resulting target something like @raze_some_bin//:bin

We've been using PyO3 and Maturin at Spring for a while now, and happily. The smooth Python interop means we can call out to Rust without much pain for performance-critical codepaths. But the other side-effect is that we can use Rust across the org more broadly, even when Python interop isn't a consideration -- e.g., for isolated services, or applications that need to compile to Windows, or whatever else -- since we're building up the cultural knowledge and shared libraries to do so.

Other people have pushed back about safety, but I want to push back on performance.

Here's the thing, presumably you agree that "It's fast except that it never works" isn't actually fast. So in practice your high performance C ends up compromised by the reality that it must work, at least often, which constrains what you are confident to actually write because you already know you can't write over-complicated code correctly.

Languages like Rust (and C++) enable you to write much more complicated software that you still understand well enough to debug it. An efficient filter that might be a single line of Rust or C++ may take dozens of lines of C, or else several macro invocations that add invisible overhead of lines that are constructed by the pre-processor unseen by the programmer, yet still taint the shared namespace and semantics. As a result, while the Rust or C++ programmer feels free to chain say, six filters the equivalent C looks monstrous and you recoil from it, even though that's the efficient way to solve the problem you had.

The intuition that if it looks simpler it's faster is often wrong, the reason Godbolt (Compiler Explorer) exists is that Matt Godbolt was concerned about whether the C++ for-each loop results in the same fast machine code as a manual loop, or whether you might pay a price for the nicer syntax. You don't, and Matt's continued exploration of this sort of issue, plus his generosity in sharing the result with the world is why the site is there now.

But of course the nicer iterator loop (and there are a lot of examples like this) encourages you to choose more complicated solutions which are faster, because total cognitive load isn't so great as it would be in a less expressive language. As a result even though you could in principle write an equally high performance C program, actual C programmers would not do that.

Now, machines don't fear complexity, so one of the interesting results of WUFFS is that they produce C code which no human would ever write, but which has all the properties they guaranteed (e.g. memory safety but lots more) in their small interest domain, by "simply" using tremendous amounts of complexity. The unexpected effect of this is that WUFFS-the-library is very fast: Since the only possible mistakes are programs that either don't do what is required or don't compile, WUFFS programmers are freed to focus on very fast tight code for the library. Again, you could in theory have written that code in C, but you wouldn't because you're human and you can't handle that.