Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Python faster with Rust

ohadravid.github.io

Post reply on HN
Print view View original

131–140 of 223 posts

Re: Making Python faster with Rust

#131

Post by el_oni »

Barrin92 wrote: post #80
bakuninsbart wrote: post #76

Earlier quoted context omitted.

Because professional software developers with a background in CS are a minority of people who program today. The learning curve of pointers, memory-allocation, binary operations, programming paradigms, O-Notation and other things you need to understand to efficiently code in something like C is a lot to ask of someone who is for example primarily a sociologist or biologist. The use case btw. is often also very differ…

tbf you don't need to go to C. You could write Common Lisp or Ocaml, both academic high level languages and very performant. Hell SBCL can get you to C range performance wise while you're writing dynamic, GCed code. Sure it's a little bit more involved than learning Python but not that much if you get 50x performance for free. Prevalence of Python is really baffling to me because compute resources cost money.

academic for CS maybe. Not so much for chemistry, biology etc. If you work in computation areas of those subjects you are more likely to know matlab, R, python and maybe Julia.

I didn't know of anybody who had done any Lisp or Ocaml in my time in academia (in chemistry, chemical engineering and biology departments), but that's just 3 universities, and i certainly didn't know everybody.

Re: Making Python faster with Rust

#132

Post by faitswulff »

thanatropism wrote: post #122

I feel a lot of the "Python perf" thing is an inferiority complex. cPython is getting faster all the time, and (obviously using libraries like numpy and others that hook into compiled code) I don't think I've ever seen it become a business bottleneck. If it's ever a scaling problem, then you hire lower-level language devs, it's a good problem to have. Python is much, much easier to learn, and Rust is notoriously diff…

I think you might be surprised. I would say Rust isn't notoriously difficult per se, it's just harder to please the compiler. But that's still viewing the compiler as an adversary when it's more like an assistant, so the analogy breaks down. You don't have to be a 100x engineer to use Rust. In fact, quite the opposite. Rust gives engineers a lot more guardrails to prevent what would be runtime errors in other languages.

Re: Making Python faster with Rust

#133

Post by [deleted] »

_glass wrote: post #119
brahbrah wrote: post #83

This was a silly and unnecessary optimization. He’s just using numpy wrong. Instead of: for p in ps: norm(p.center - point) You should do: centers = np.array([p.center for p in ps]) norm(centers - point, axis=1) You’ll get your same speed up in 2 lines without introducing a new dependency

Isn't this the version of refenced on the github repo [0] which speeds up 6x instead of 101x? There's also a "v1.5" version which is 6x faster, and uses "vectorizing" (doing more of the work directly in numpy). This version is much harder to optimize further. [0] https://github.com/ohadravid/poly-match

[deleted]

Re: Making Python faster with Rust

#134

Post by brahbrah »

_glass wrote: post #119
brahbrah wrote: post #83

This was a silly and unnecessary optimization. He’s just using numpy wrong. Instead of: for p in ps: norm(p.center - point) You should do: centers = np.array([p.center for p in ps]) norm(centers - point, axis=1) You’ll get your same speed up in 2 lines without introducing a new dependency

Isn't this the version of refenced on the github repo [0] which speeds up 6x instead of 101x? There's also a "v1.5" version which is 6x faster, and uses "vectorizing" (doing more of the work directly in numpy). This version is much harder to optimize further. [0] https://github.com/ohadravid/poly-match

No, their v1.5 is still calling norm on every polygon. They’re still using it wrong

On Google colab

    import numpy as np
    import time


    vals = np.random.randn(1000000, 2)
    point = np.array([.2, .3])
    s = time.time()
    for x in vals:
        np.linalg.norm(x - point)
~296x faster, significantly faster than the solution in the article.

Re: Making Python faster with Rust

#135

Post by flohofwoe »

wcrossbow wrote: post #128

Rust is great but isn’t the core problem here using the wrong algorithm? It looks like this is ideally suited for a quad tree instead of a naive for loop. I would expect that to pulverise any current benchmark.

I guess you also need to take the time into account to create the quad tree from an unsorted 'polygon soup' first, and in terms of coding effort, a brute force conversion from python to a compiled tight loop over unsorted arrays provides a lot of bang for the buck (and a speedup of 100x for relatively little effort might be 'good enough' for quite a while until the input data grows big enough to require the next optimization effort).

(e.g. it doesn't need to be "as fast as possible", just fast enough to no longer be a workflow bottleneck)

Re: Making Python faster with Rust

#136

Post by JodieBenitez »

thanatropism wrote: post #122

I feel a lot of the "Python perf" thing is an inferiority complex. cPython is getting faster all the time, and (obviously using libraries like numpy and others that hook into compiled code) I don't think I've ever seen it become a business bottleneck. If it's ever a scaling problem, then you hire lower-level language devs, it's a good problem to have. Python is much, much easier to learn, and Rust is notoriously diff…

> Also: does it ever make sense to write something like a CRUD+ backend in Rust?

Well... given 2 frameworks equally pleasant to work with, why not using the one with the best performance ? (Whatever performance means... less cpu ? less memory ? better i/o ?).

As a Django user, working on problems where Django shines, I have yet to see such a solution in Rust, but that doesn't mean it won't happen one day.

Re: Making Python faster with Rust

#137

Post by rhodysurf »

JodieBenitez wrote: post #136
thanatropism wrote: post #122

I feel a lot of the "Python perf" thing is an inferiority complex. cPython is getting faster all the time, and (obviously using libraries like numpy and others that hook into compiled code) I don't think I've ever seen it become a business bottleneck. If it's ever a scaling problem, then you hire lower-level language devs, it's a good problem to have. Python is much, much easier to learn, and Rust is notoriously diff…

> Also: does it ever make sense to write something like a CRUD+ backend in Rust? Well... given 2 frameworks equally pleasant to work with, why not using the one with the best performance ? (Whatever performance means... less cpu ? less memory ? better i/o ?). As a Django user, working on problems where Django shines, I have yet to see such a solution in Rust, but that doesn't mean it won't happen one day.

I use it for backends that do a lot of math because the python deployment story sucks, rust is faster anyways, and I love rust (do python and js by day). It’s been so easy to bring up rust containers on serverless and they just don’t break. The cold start times are also insanely good compared to python

Re: Making Python faster with Rust

#138

Post by hn92726819 »

rwalle wrote: post #114
korijn wrote: post #60

A vectorized implementation of find_close_polygons wouldn't be very complex or hard to maintain at all, but the authors would also have to ditch their OOP class based design, and that's the real issue here. The object model doesn't lend itself to performant, vectorized numpy code.

Exactly, that is the real issue, vectorization might be good enough in terms of performance. It doesn't seem to be mentioned in the article at all.

It might have been added later, but the author mentions vectorization in the beginning:

> It’s worth noting that converting parts of / everything to vectorized numpy might be possible for this toy library, but will be nearly impossible for the real library while making the code much less readable and modifiable, and the gains are going to be limited (here’s a partially vertorized version, which is faster but far from the results we are going to achieve).

Re: Making Python faster with Rust

#139

Post by ZeroCool2u »

okaleniuk wrote: post #10

Good for you! You did everything right: measure always, fix the bottleneck if possible, rewrite if necessary. A little tip, you don't have to compare actual distances, you can compare squared distances just as well. Then in `norm I once rewrote a GDI+ point transformation routine in pure C# and got 200x speedup just because the routine was riddled with needless virtual constructors, copying type conversions, and some…

The ignore the square root while computing/comparing distances trick is a great one. That's how I got to the top of the performance leaderboard in my first algorithms class.

Re: Making Python faster with Rust

#140

Post by akasakahakada »

brahbrah wrote: post #126
masklinn wrote: post #90

Earlier quoted context omitted.

You made this assertion multiple times, but so far it’s been entirely unsupported in fact, despite TFA having made the entire code set available for you to test your hypothesis on.

On Google colab import numpy as np import time vals = np.random.randn(1000000, 2) point = np.array([.2, .3]) s = time.time() for x in vals: np.linalg.norm(x - point) ~296x faster, significantly faster than the solution in the article. And my assertion was supported by nearly 20 years of numpy being a leading tool in various quantitative fields. It’s not hard to imagine that a ubiquitous tool that’s been used and opti…

Finally some dude knows how to use numpy properly. I wish I can upvote 5 times.

I basically raise the same question somewhere below and got downvoted LOL.

Post reply on HN