Using the unsafe keyword... somehow makes me wonder where else unsafe is used...
In this case, it actually didn't bring this much speedup. The program would have been plenty fast even without.
Recent adventures in performance optimization with Rust
31–40 of 75 posts
Re: Recent adventures in performance optimization with Rust
#32Comparing highly optimized code (including total algorithm rewrite and relying on unsafe and SIMD operations) without doing the same on the other side is a pointless exercise. It's like showing how much faster you can get your handcrafted assembly code to run vs a bash script.
To add on to the defense in another comment,
I don't think the point of the article is a rust-v-python comparison. Sure, unoptimized-python2rust2optimized rust was a part of the clickbait 180,000x title but... - the author explicitly says the python2rust bit is only a factor of 8. - the Python bit is just one section of a pretty long article - the author says they typically start implementing something in Python so... why wouldn't they include their initial implementation?
I don't think this article should be dismissed just for including a simple python implementation- that isn't the point.
Re: Recent adventures in performance optimization with Rust
#33Moore's law is dead, that means developers who will not waste CPU cycles will be in demand for the next generation of compute intensive development. Big Data needs big compute so obviously these kinds of articles are not only pertinent but is the way ahead.
I would not be surprised if we do rewrites of all the past 20 year code in to Rust or Go or any other performant language.
Lets agree on this, you are smart and the author is right.
I am sorry people who are complaining are not seeing where the ball is going.
Re: Recent adventures in performance optimization with Rust
#34> the point of this post isn’t to compare highly-optimized Python to highly-optimized Rust. The point is to compare “standard-Jupyter-notebook” Python to highly-optimized Rust. I guess the title gets clicks, but I'm curious how good python gets. I'm under the impression pandas is pretty fast despite it being python
Yeah, this seems like really unoptimized Python. It would be far more useful to see tricks for torturing Python/Pandas to make things faster and how far you can take it.
The author wrote in a strongly typed, ahead of time compiled language, ran a profiler to help wipe out hotspots for optimization, and the program is now dramatically faster than unoptimized Python.
Yay? I guess?
Re: Recent adventures in performance optimization with Rust
#35I love demo apps like this. You can see some idiomatic code and then watch it descend into chaos but you can see the pathway. Otherwise you sometimes see the end result and you don't know why it is what it is, but the pathway makes it comprehensible. Thanks for writing it.
Re: Recent adventures in performance optimization with Rust
#36Basically "Analyzing Data 180,000x Faster with AOT strongly typed compiled language" versus using a dynamically typed interpreted one.
Roughly half the article could have been done without leaving python though. I don't know if python has a good bitset implementation, but at the very least everything before that could have been done in python. So it is more "comparing manually optimized code against unoptimized crap"
Re: Recent adventures in performance optimization with Rust
#37Re: Recent adventures in performance optimization with Rust
#38Anyone else think this code is ugly as hell?
Re: Recent adventures in performance optimization with Rust
#39Comparing highly optimized code (including total algorithm rewrite and relying on unsafe and SIMD operations) without doing the same on the other side is a pointless exercise. It's like showing how much faster you can get your handcrafted assembly code to run vs a bash script.
articles like this aren't pointless, i don't think it is meant to be framed as a fair comparison of "rust-the-language is 5 orders of magnitude faster than python-the-language". this article does offer examples of how to use a profiler and some ideas of what kind of things can be bottlenecks and how to eliminate them, and also helps people who only ever work with very slow programming tech stacks to understand what k…
Intentional or not, it does come off as kind of “disruptive” in the negative sense of the word, ie “you’re all doing it wrong in field X. Let me show you how to work faster”.
But in a real scenario there are often so many other constraints that “optimized code” is far from the top. Salaries are also expensive, so if you factor in the lifetime cost of writing and maintaining a bit of exotic Rust for an exotic problem, it might actually be cheaper to buy a bigger machine and brute force the problem.
Sure, we could all benefit from learning a bit of Rust then, but if you’re in the Python data space you know that it’s almost as crowded as frontend frameworks, and learning Rust comes across as a yet-another-framework problem, that few of us really feel like we have time for.
Personally I’m stuck maintaining a wizard’s (who has now left the company) solo project which initially seemed like a great solution, but has caused more grief in maintenance and bugs than any existing not-as-custom-tailored community solution out there. So if I were to tell my manager “I’ve spent a week to rewrite this job that spends 40 minutes at 2 AM to spend 1 millisecond at 2 AM, from a language that everyone uses to a language that only I use” I’m not sure he’d appreciate it.
Re: Recent adventures in performance optimization with Rust
#40Basically "Analyzing Data 180,000x Faster with AOT strongly typed compiled language" versus using a dynamically typed interpreted one.
Roughly half the article could have been done without leaving python though. I don't know if python has a good bitset implementation, but at the very least everything before that could have been done in python. So it is more "comparing manually optimized code against unoptimized crap"
Yeah, then there is the implementation thing, if cython or PyPy have been used instead.