71–80 of 223 posts
> Rust (with the help of pyo3) unlocks true native performance for everyday Python code, with minimal compromises. Hasn't he heard of ctypes? You can wrap C structs add Python objects since forever.
I assume, haven’t really messed with numpy for anything but I can’t imagine it wouldn’t work that way.
> The library was already using numpy for a lot of its calculations, so why should we expect Rust to be better? I literally clicked in to read the article to see if they'd mention this:) But... unless I missed it, there wasn't really an answer? I thought numpy does do the heavy lifting in native code, so why is this faster? Does this version just push more of the logic into native code than numpy did?
Numpy is fast when the code is vectorized. The code they are benchmarking against was not vectorized. They wanted to calculated the distances of n points against a given point and find out which points are closer than a threshold (max_dist). Instead of vectorizing the whole operation, the python code was just calling numpy in a loop to find the distance of two points. Just that small change already gives 10x faster p…
Scipy should have already implemented such thing. Scikit-Learn also. Because KNN clustering is exactly doing this kind of work.
``` centers = get_centers(polgons) # M x 3 array close_idx = np.where( np.linalg.norm(centers - point, axis=1) That's one reason I prefer for to use arrays for polygons, rather then abstract it into a Python object. Fundamentally geometries are sequences of points, and with some zero-padding to account for irregular point counts, you can still keep them in a nice, efficient array representation.
Earlier quoted context omitted.
The final code takes just 2.90ms per iteration.
The rest is not a fair comparison, because it rewrites the used libraries, not the application code. You can always speed up an application if you rewrite the used libraries to match your specific use case.
"That Rust library will be faster than the corresponding python library" is a useful thing to know here.
I wonder if being able to quickly retrieve a numpy array of the polygon centers would make an equivalent difference. Since then you could at least retrieve the centers from the polygon as an array you could just use numpy operations for the closest polygon operation: ``` centers = get_centers(polgons) # M x 3 array close_idx = np.where( np.linalg.norm(centers - point, axis=1) That's one reason I prefer for to use arr…
poly = Polygon(vertices)
This is a great article but there's still a core problem there - why should developers have to choose between accessibility and performance? So much scientific computing code suffers between core packages being split away from their core language - at what point do we stop and abandon python for languages which actually make sense? Obviously julia is the big example here, but its interest, development and ecosystem d…
The use case btw. is often also very different. In most of academia, writing code is basically just a fancy mode of documentation for what is basically a glorified calculator. Readability trumps efficiency by a large margin every time.
Earlier quoted context omitted.
The rest is not a fair comparison, because it rewrites the used libraries, not the application code. You can always speed up an application if you rewrite the used libraries to match your specific use case.
Usually not by 10x though, unless the original implementation involved some really bad decisions.
So much faster that the allocations for the result will probably be the bottleneck.
The most important part of the article seems to be that this Python code is taking "an avg of 293.41ms per iteration": def find_close_polygons( polygon_subset: List[Polygon], point: np.array, max_dist: float ) -> List[Polygon]: close_polygons = [] for poly in polygon_subset: if np.linalg.norm(poly.center - point) And after replacing it with this Rust code, it is taking "an avg of 23.44ms per iteration": use pyo3::pre…
Python's for loop implementation is slow, also. You can use built in utils like map() which are "native" and can be a lot faster than a for loop with a push: https://levelup.gitconnected.com/python-performance-showdown...
Benchmarking methodology in the link is not good. Author should use timeit() or cProfiler or so. 0.01s of difference is mostly due to fluctuation. The order of execution also matters. Say you want to test A and B function, you need actually to run A, B, B, A to see if the ordering brings the different.
Is the problem the Oracle involvement? Or is it not that fast as advertised or problems with the ecosystem (C libraries)?
This is a great article but there's still a core problem there - why should developers have to choose between accessibility and performance? So much scientific computing code suffers between core packages being split away from their core language - at what point do we stop and abandon python for languages which actually make sense? Obviously julia is the big example here, but its interest, development and ecosystem d…
Because professional software developers with a background in CS are a minority of people who program today. The learning curve of pointers, memory-allocation, binary operations, programming paradigms, O-Notation and other things you need to understand to efficiently code in something like C is a lot to ask of someone who is for example primarily a sociologist or biologist. The use case btw. is often also very differ…