Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Python faster with Rust

ohadravid.github.io

Post reply on HN
Print view View original

121–130 of 223 posts

Re: Making Python faster with Rust

#121

Post by [deleted] »

masklinn wrote: post #90
brahbrah wrote: post #87

Earlier quoted context omitted.

They would have gotten the same performance in python with numpy if they did it like this instead of calling norm for every polygon centers = np.array([p.center for p in ps]) norm(centers - point, axis=1) They were just using numpy wrong. You can be slow in any language if you use the tools wrong

You made this assertion multiple times, but so far it’s been entirely unsupported in fact, despite TFA having made the entire code set available for you to test your hypothesis on.

[deleted]

Re: Making Python faster with Rust

#122

Post by thanatropism »

I feel a lot of the "Python perf" thing is an inferiority complex. cPython is getting faster all the time, and (obviously using libraries like numpy and others that hook into compiled code) I don't think I've ever seen it become a business bottleneck. If it's ever a scaling problem, then you hire lower-level language devs, it's a good problem to have.

Python is much, much easier to learn, and Rust is notoriously difficult. This obviously feeds into the inferiority complex. But -- while I do know there's a number of applications where perf is crucial -- I think it's well worth doing an ego check before moving from what's a lower friction path and gives you access to myriad developers, including ones trained in very hard disciplines.

Also: does it ever make sense to write something like a CRUD+ backend in Rust? Maybe 100X Rustaceans can do it with one hand tied behind their backs; but imagine what these ubermenschen could be achieving in Python?

Re: Making Python faster with Rust

#123

Post by brahbrah »

tecleandor wrote: post #106
brahbrah wrote: post #89

Earlier quoted context omitted.

Agreed it was well written, but kinda pointless though since they could have “solved” the problem using the existing tools in a couple lines of code without any new deps. All that content annd profiling and they missed the fact that they were using numpy wrong.

Totally out of curiosity, could you be a bit more concrete in your posted example? I can't get it to work (I'm inexperienced with numpy and I'm messing something when translating your quick example to python) Thanks!

Yea sorry that was just pseudo code. You want it in this form:

    centers = np.array([
        [3, 3],
        [4, 4]
    ])
    point = np.array([3.5, 3.5])
    vals = centers - point
    np.linalg.norm(vals, axis=1)

Re: Making Python faster with Rust

#124

Post by dunefox »

majoe wrote: post #39

I had a similar problem, when I was working as a PhD student a few years ago, where I needed to match the voxel representation of a 3D printer with the tetrahedral mesh of our rendering application. My first attempt in Python was both prohibitively slow and more complicated than necessary, because I tried to use vectorized numpy, where possible. Since this was only a small standalone script, I rewrote it in Julia in…

Long startup time is relative. I believe it's much lower now than a couple of versions ago. 0.15s or so? Interop between python and rust will also take time.

Re: Making Python faster with Rust

#125

Post by za3faran »

INTPenis wrote: post #11
ssivark wrote: post #2

Making Python (near infinitely) faster by using it as a glue language, and running all the computation outside Python :-P

Yeah what's wrong with that? I think this sounds amazing. It gives you all the fast prototyping and simplicity of Python, but once you hit that bottleneck all you have to do is bring in a ringer to replace key components with a faster language. No need to use Golang or Rust from the start, no need for those resources until you absolutely need the speed improvement. Sounds like a dream to a lot of people who find it m…

Then you give up the benefits of using a managed language, and you now have to maintain two stacks.

IMO/IME much better to go with a language where you don't have this dichotomy in the first place - e.g. Java or C#.

Re: Making Python faster with Rust

#126

Post by brahbrah »

masklinn wrote: post #90
brahbrah wrote: post #87

Earlier quoted context omitted.

They would have gotten the same performance in python with numpy if they did it like this instead of calling norm for every polygon centers = np.array([p.center for p in ps]) norm(centers - point, axis=1) They were just using numpy wrong. You can be slow in any language if you use the tools wrong

You made this assertion multiple times, but so far it’s been entirely unsupported in fact, despite TFA having made the entire code set available for you to test your hypothesis on.

On Google colab

    import numpy as np
    import time


    vals = np.random.randn(1000000, 2)
    point = np.array([.2, .3])
    s = time.time()
    for x in vals:
        np.linalg.norm(x - point)
~296x faster, significantly faster than the solution in the article. And my assertion was supported by nearly 20 years of numpy being a leading tool in various quantitative fields. It’s not hard to imagine that a ubiquitous tool that’s been used and optimized for almost 20 years is actually pretty good if used properly.

Re: Making Python faster with Rust

#127

Post by za3faran »

cornholio wrote: post #64

Language design request: Take from Rust Algebraic types, ahead of time compilation and strong types, functional features, a borrow / escape checker that automatically turns shared data into Rc or Arc, as necessary, instead of tormenting me to rewrite performance irrelevant code; Take from Python the simple syntax, default pass by reference of all non-numeric types, simplified string handling, unified slice and array…

Modern Java and modern C# should be closer to what you described.

On a side note, python is strictly pass by value. For non-primitives, their references are passed by value.

Re: Making Python faster with Rust

#128

Post by wcrossbow »

Rust is great but isn’t the core problem here using the wrong algorithm? It looks like this is ideally suited for a quad tree instead of a naive for loop. I would expect that to pulverise any current benchmark.

Re: Making Python faster with Rust

#129

Post by el_oni »

fbdab103 wrote: post #32
b0b10101 wrote: post #17

This is a great article but there's still a core problem there - why should developers have to choose between accessibility and performance? So much scientific computing code suffers between core packages being split away from their core language - at what point do we stop and abandon python for languages which actually make sense? Obviously julia is the big example here, but its interest, development and ecosystem d…

Today, there is a Python package for everything . The ecosystem is possibly best in class for having a library available that will do X. You cannot separate the language from the ecosystem. Being better, faster, and stronger means little if I have to write all of my own supporting libraries. Also, few scientific programmers have any notion of what C or Fortran is under the hood. Most are happy to stand on the shoulde…

This is how I got into software development.

During my PhD I was running some simulations using poorly written python code. initially it would take several hours. In that time i could go to the lab, run some wetlab experiments and the results of my simulations would be there when i got back to the office. It was only taking python "home" and building some of my own projects that i learned how to 1. write more pythonic code and 2. write more performant code. Now i work for a software company.

If i'd have stayed in in academia I would probably still be writing quick and dirty code and not worrying about the runtime because as a researcher there is always something else you can be doing.

Re: Making Python faster with Rust

#130

Post by pjmlp »

thanatropism wrote: post #122

I feel a lot of the "Python perf" thing is an inferiority complex. cPython is getting faster all the time, and (obviously using libraries like numpy and others that hook into compiled code) I don't think I've ever seen it become a business bottleneck. If it's ever a scaling problem, then you hire lower-level language devs, it's a good problem to have. Python is much, much easier to learn, and Rust is notoriously diff…

If writing Tcl extensions in C during the .com wave 23 years ago taught me something, was that glue languages are great, and that I don't want to use any that doesn't come with either AOT or JIT compiler in the box, other than for OS scripting tasks.
Post reply on HN