Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extending Python with Rust

maxwellrules.com

Post reply on HN
Print view View original

21–30 of 88 posts

Re: Extending Python with Rust

#21

Post by jacquesm »

rpep wrote: post #12

I've been through the journey with compiling extensions to Python in lower languages (primarily C and C++) and it's really worth evaluating the benefit. In general, the control that it gives you over memory layout and chaining of operations make it very difficult to say from toy examples what benefit you'll get. If you test a single array operation with NumPy then it will compare favourably to a low level implementat…

NumPy is insanely fast for most use cases, I've (grudgingly) come to the conclusion that if NumPy doesn't hack it then I should re-think the problem or my way of solving it rather than to try to optimize that particular bit of code if it isn't meant for something that is going to be run in production on a large number of machines. Likely there are better uses of my time. It's interesting how what is nominally a scripting language can perform so well for compute intensive tasks. Of course, technically you are just stringing together highly optimized low level functions together using the Python language but the advantages of doing it that way have more often than not surprised me.

Re: Extending Python with Rust

#22

Post by smcl »

ryantgtg wrote: post #8

Meta comment: “ … with Rust” feels like a HN meme to me. I know nothing about Rust, but I’ve noticed that there’s a top post on HN every day that followed this format.

I'm not a Rust developer, but it seems to me that there are very real benefits to using the language. So I think the reason there are so many "I did $task in Rust!" - where $task is something usually accomplished in C, the current lingua franca of lower-level systems programming - is that people are genuinely excited to share that it really is looking possible that Rust can be used in place of C.

So it's a bit of a meme, sure, but there is a good reason for it :)

Re: Extending Python with Rust

#23

Post by tmtvl »

ryantgtg wrote: post #8

Meta comment: “ … with Rust” feels like a HN meme to me. I know nothing about Rust, but I’ve noticed that there’s a top post on HN every day that followed this format.

To be fair though, there are also posts every so often showing "xyz library in a single C header." It gets some of the love it deserves.

Re: Extending Python with Rust

#24

Post by ziml77 »

remram wrote: post #10

This is great but you should be aware of the cost. Distributing binary wheels for every available platform (including ARM, MacOS, ...) and implementation (PyPy...) is not easy, and not doing it causes really abysmal user experience (doing `pip install requests` and being told you need to install a Rust toolchain to build `cryptography`). Sometimes the performance might not be worth it. Thankfully there are GitHub Act…

Having to install a Rust toolchain is still easier than trying to get C extensions to build. Before binary wheels were common on PyPi, I would either give up as soon as I encountered a C extension or switch to something like Conda to get their binaries.

Re: Extending Python with Rust

#25

Post by ryantgtg »

timeon wrote: post #19
ryantgtg wrote: post #8

Meta comment: “ … with Rust” feels like a HN meme to me. I know nothing about Rust, but I’ve noticed that there’s a top post on HN every day that followed this format.

> followed this format And comments like yours are following one as well.

As is yours!

Re: Extending Python with Rust

#26

Post by [deleted] »

jacquesm wrote: post #21
rpep wrote: post #12

I've been through the journey with compiling extensions to Python in lower languages (primarily C and C++) and it's really worth evaluating the benefit. In general, the control that it gives you over memory layout and chaining of operations make it very difficult to say from toy examples what benefit you'll get. If you test a single array operation with NumPy then it will compare favourably to a low level implementat…

NumPy is insanely fast for most use cases, I've (grudgingly) come to the conclusion that if NumPy doesn't hack it then I should re-think the problem or my way of solving it rather than to try to optimize that particular bit of code if it isn't meant for something that is going to be run in production on a large number of machines. Likely there are better uses of my time. It's interesting how what is nominally a scrip…

[deleted]

Re: Extending Python with Rust

#27

Post by ryantgtg »

smcl wrote: post #22
ryantgtg wrote: post #8

Meta comment: “ … with Rust” feels like a HN meme to me. I know nothing about Rust, but I’ve noticed that there’s a top post on HN every day that followed this format.

I'm not a Rust developer, but it seems to me that there are very real benefits to using the language. So I think the reason there are so many "I did $task in Rust!" - where $task is something usually accomplished in C, the current lingua franca of lower-level systems programming - is that people are genuinely excited to share that it really is looking possible that Rust can be used in place of C. So it's a bit of a m…

Yeah, without knowing much about Rust, I’ve decided from these types of posts that I should gravitate toward “app I use but now it’s in Rust.” I use plenty of Rust programs simply because people seem excited about it.

Re: Extending Python with Rust

#29

Post by rapatel0 »

jacquesm wrote: post #21
rpep wrote: post #12

I've been through the journey with compiling extensions to Python in lower languages (primarily C and C++) and it's really worth evaluating the benefit. In general, the control that it gives you over memory layout and chaining of operations make it very difficult to say from toy examples what benefit you'll get. If you test a single array operation with NumPy then it will compare favourably to a low level implementat…

NumPy is insanely fast for most use cases, I've (grudgingly) come to the conclusion that if NumPy doesn't hack it then I should re-think the problem or my way of solving it rather than to try to optimize that particular bit of code if it isn't meant for something that is going to be run in production on a large number of machines. Likely there are better uses of my time. It's interesting how what is nominally a scrip…

This

Op is better off recompiling numpy to target his specific hardware thank trying to speed it up using rust. This would be to try different underlying math implementations that are linked with numpy ( blas, mkl, eigen). Each of these haha several internal simulation frameworks to optimize not just the actual instructions but memory layout for various math kernels.

Re: Extending Python with Rust

#30

Post by orangepurple »

college_physics wrote: post #3

> What's more interesting is that the Rust implementation is just a factor of 1.23 slower (for large arrays) than just using Numpy I suppose what is meant is faster (also follows from the diagram?). But it is still not a dramatic gain for many use cases. This shows how non-trivial the python performance calculus: pure python, versus numpy python, versus compiled c/c++ or rust. People who want to speed up python shoul…

I had to do a double take. The Rust implementation is slower and harder to maintain. I recommend adding a cythonized function and a numba jitted function to the benchmark for completeness.
Post reply on HN