Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Python faster with Rust

ohadravid.github.io

Post reply on HN
Print view View original

11–20 of 223 posts

Re: Making Python faster with Rust

#11

Post by INTPenis »

ssivark wrote: post #2

Making Python (near infinitely) faster by using it as a glue language, and running all the computation outside Python :-P

Yeah what's wrong with that? I think this sounds amazing. It gives you all the fast prototyping and simplicity of Python, but once you hit that bottleneck all you have to do is bring in a ringer to replace key components with a faster language.

No need to use Golang or Rust from the start, no need for those resources until you absolutely need the speed improvement. Sounds like a dream to a lot of people who find it much easier to develop in Python.

Re: Making Python faster with Rust

#13

Post by anewhnaccount2 »

yjftsjthsd-h wrote: post #4

> The library was already using numpy for a lot of its calculations, so why should we expect Rust to be better? I literally clicked in to read the article to see if they'd mention this:) But... unless I missed it, there wasn't really an answer? I thought numpy does do the heavy lifting in native code, so why is this faster? Does this version just push more of the logic into native code than numpy did?

The article mentioned that there are some gains from using vectorised numpy methods, i.e. spending more time in numpy code. I would be interested in whether the List[Polygon] could be converted into two long arrays of all xs and all ys with indices into the starts (essentially a dense representation of a sparse array) and then the core function rewritten for Numba since it could now not use any Python objects. This would break the interface of course, but may get within striking distance of the Rust implementation.

Re: Making Python faster with Rust

#14

Post by okaleniuk »

INTPenis wrote: post #11
ssivark wrote: post #2

Making Python (near infinitely) faster by using it as a glue language, and running all the computation outside Python :-P

Yeah what's wrong with that? I think this sounds amazing. It gives you all the fast prototyping and simplicity of Python, but once you hit that bottleneck all you have to do is bring in a ringer to replace key components with a faster language. No need to use Golang or Rust from the start, no need for those resources until you absolutely need the speed improvement. Sounds like a dream to a lot of people who find it m…

Also, there is a faster Python which is also Python. And the author considered it as well (both PyPy and Numba), it's just in this particular scenario they were not the best way to go.

Re: Making Python faster with Rust

#15

Post by internetter »

INTPenis wrote: post #11
ssivark wrote: post #2

Making Python (near infinitely) faster by using it as a glue language, and running all the computation outside Python :-P

Yeah what's wrong with that? I think this sounds amazing. It gives you all the fast prototyping and simplicity of Python, but once you hit that bottleneck all you have to do is bring in a ringer to replace key components with a faster language. No need to use Golang or Rust from the start, no need for those resources until you absolutely need the speed improvement. Sounds like a dream to a lot of people who find it m…

Right? I mean, it's not like we haven't been doing this already. All the computationally intensive python libraries are just a convenient wrapper for C anyway, the only reason python can be used for ML.

Re: Making Python faster with Rust

#16

Post by jacquesm »

It's not really Python that was sped up though, it was an application written in python augmented with a bit of optimized rust code.

This sort of hybrid is super common, you typically spend 90%+ of your time in computationally intensive problems in a very small subset of your code, typically the innermost loops. Optimizing those will have very good pay-off.

Traditionally we'd do this with high level stuff in one language and then assembly for the performance critical parts, these days it is more likely a combination of a scripting language for the high level part and a compiled language for the low level parts (C, rust, whatever). Java and such as less suitable for such optimization purposes, both because they come with a huge runtime and because they are hard to interface to other languages unless they happen to use the same underlying VM, but then there usually isn't much performance gain.

Another nice way to optimize computationally intensive code is by finding out if the code is suitable for adaptation to the GPU, which can give you many orders of magnitude speed improvement in some cases.

Re: Making Python faster with Rust

#17

Post by b0b10101 »

This is a great article but there's still a core problem there - why should developers have to choose between accessibility and performance?

So much scientific computing code suffers between core packages being split away from their core language - at what point do we stop and abandon python for languages which actually make sense? Obviously julia is the big example here, but its interest, development and ecosystem doesn't seem to be growing at a serious pace. Given that the syntax is moderately similar and the performance benefits are often 10x what's stopping people from switching???

Re: Making Python faster with Rust

#19

Post by est »

ssivark wrote: post #2

Making Python (near infinitely) faster by using it as a glue language, and running all the computation outside Python :-P

> and running all the computation outside Python

You don't have to move all the computation, just `for` loops will help alot.

Re: Making Python faster with Rust

#20

Post by ThouYS »

b0b10101 wrote: post #17

This is a great article but there's still a core problem there - why should developers have to choose between accessibility and performance? So much scientific computing code suffers between core packages being split away from their core language - at what point do we stop and abandon python for languages which actually make sense? Obviously julia is the big example here, but its interest, development and ecosystem d…

everything. why are there still cobol programmers? why is c++ still the defacto native language (also in research)?

but also I don't see any problem there, I think the python + c++/rust idiom is actually pretty nice. I have a billion libs to choose from on either side. Great usability on the py side, and unbeatable performance on the c++ side

Post reply on HN