Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Python faster with Rust

ohadravid.github.io

Post reply on HN
Print view View original

81–90 of 223 posts

Re: Making Python faster with Rust

#81

Post by ben-schaaf »

FreeHugs wrote: post #40
nickstinemates wrote: post #35

Earlier quoted context omitted.

One carries the entire feature set of the python runtime, the other is compiled.

The time is spent in this 3-line loop: for poly in polygon_subset: if np.linalg.norm(poly.center - point) I don't think the entire feature set of the Python runtime is involved in this.

Without using every feature you still have to conform to the complexity of the runtime. Every variable in that loop is a hash map lookup into the locals. `np.linalg.norm` is two field accesses, necessitating more hash map lookups on the module objects. `-` and `<` are attribute lookups as well as full function calls.

Re: Making Python faster with Rust

#82

Post by visarga »

b0b10101 wrote: post #17

This is a great article but there's still a core problem there - why should developers have to choose between accessibility and performance? So much scientific computing code suffers between core packages being split away from their core language - at what point do we stop and abandon python for languages which actually make sense? Obviously julia is the big example here, but its interest, development and ecosystem d…

Julia doesn't get the latest models first, or have as big of a community.

Re: Making Python faster with Rust

#83

Post by brahbrah »

This was a silly and unnecessary optimization. He’s just using numpy wrong.

Instead of:

for p in ps: norm(p.center - point)

You should do:

centers = np.array([p.center for p in ps]) norm(centers - point, axis=1)

You’ll get your same speed up in 2 lines without introducing a new dependency

Re: Making Python faster with Rust

#84

Post by mgraupner »

alex_smart wrote: post #27
biomcgary wrote: post #9

Earlier quoted context omitted.

The slowness comes from the interaction of numpy and a Python object "Polygon", which in not numpy. I suspect that a sufficiently clever coder could have optimized the result without resorting to Rust, but at the cost of a substantial increase in complexity of the codebase. The proposed approach keep the Python code simple (and moves the complexity into having another language to deal with).

diff --git a/poly_match_v1.py b/poly_match_v1.py index 675c88a..4293a46 100644 --- a/poly_match_v1.py +++ b/poly_match_v1.py @@ -1,4 +1,5 @@ from functools import cached_property +from itertools import compress from typing import List, Tuple import numpy as np from dataclasses import dataclass @@ -56,11 +57,8 @@ def generate_example() -> Tuple[List[Polygon], List[np.array]]: def find_close_polygons( polygon_subset: L…

I wish the author would take your suggestion (and other recommendations) here and try it out again. Would be a very interesting follow up to read: "How we speed up Python and simplifying our codebase by removing on more dependency".

Re: Making Python faster with Rust

#85

Post by thisgoodlife »

karussell wrote: post #79

I wonder why GraalVM is not more often used for these speed critical cases: https://www.graalvm.org/python/ (Same for ruby https://www.graalvm.org/ruby/ ) Is the problem the Oracle involvement? Or is it not that fast as advertised or problems with the ecosystem (C libraries)?

My concern is that it’s not ready for prod yet.

“At this point, the Python runtime is made available for experimentation and curious end-users. “

https://www.graalvm.org/latest/reference-manual/python/

Re: Making Python faster with Rust

#86

Post by visarga »

bakuninsbart wrote: post #76
b0b10101 wrote: post #17

This is a great article but there's still a core problem there - why should developers have to choose between accessibility and performance? So much scientific computing code suffers between core packages being split away from their core language - at what point do we stop and abandon python for languages which actually make sense? Obviously julia is the big example here, but its interest, development and ecosystem d…

Because professional software developers with a background in CS are a minority of people who program today. The learning curve of pointers, memory-allocation, binary operations, programming paradigms, O-Notation and other things you need to understand to efficiently code in something like C is a lot to ask of someone who is for example primarily a sociologist or biologist. The use case btw. is often also very differ…

It also matters if you write code to run once or to serve in production, if it is experimental or stable.

If my script takes 3s to run and 5m to write in Python, vs 0.1s to run and 3h to write in C, I finish first with Python. I can try more ideas with Python.

Re: Making Python faster with Rust

#87

Post by brahbrah »

b0b10101 wrote: post #17

This is a great article but there's still a core problem there - why should developers have to choose between accessibility and performance? So much scientific computing code suffers between core packages being split away from their core language - at what point do we stop and abandon python for languages which actually make sense? Obviously julia is the big example here, but its interest, development and ecosystem d…

They would have gotten the same performance in python with numpy if they did it like this instead of calling norm for every polygon

centers = np.array([p.center for p in ps]) norm(centers - point, axis=1)

They were just using numpy wrong. You can be slow in any language if you use the tools wrong

Re: Making Python faster with Rust

#88

Post by masklinn »

okaleniuk wrote: post #10

Good for you! You did everything right: measure always, fix the bottleneck if possible, rewrite if necessary. A little tip, you don't have to compare actual distances, you can compare squared distances just as well. Then in `norm I once rewrote a GDI+ point transformation routine in pure C# and got 200x speedup just because the routine was riddled with needless virtual constructors, copying type conversions, and some…

The author did talk about it on reddit, but explained that for the purpose of the blog post they wanted to focus on the big stuff and profiling-guided optimisation of the process: https://reddit.com/r/rust/comments/125pbq0/blog_post_making_...

Re: Making Python faster with Rust

#89

Post by brahbrah »

jcolella wrote: post #8

This is possibly one of the best written articles end-to-end I have read. Excellent job telling the story

Agreed it was well written, but kinda pointless though since they could have “solved” the problem using the existing tools in a couple lines of code without any new deps. All that content annd profiling and they missed the fact that they were using numpy wrong.

Re: Making Python faster with Rust

#90

Post by masklinn »

brahbrah wrote: post #87
b0b10101 wrote: post #17

This is a great article but there's still a core problem there - why should developers have to choose between accessibility and performance? So much scientific computing code suffers between core packages being split away from their core language - at what point do we stop and abandon python for languages which actually make sense? Obviously julia is the big example here, but its interest, development and ecosystem d…

They would have gotten the same performance in python with numpy if they did it like this instead of calling norm for every polygon centers = np.array([p.center for p in ps]) norm(centers - point, axis=1) They were just using numpy wrong. You can be slow in any language if you use the tools wrong

You made this assertion multiple times, but so far it’s been entirely unsupported in fact, despite TFA having made the entire code set available for you to test your hypothesis on.
Post reply on HN