Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust std fs slower than Python? No, it's hardware

xuanwo.io

Post reply on HN
Print view View original

21–30 of 255 posts

Re: Rust std fs slower than Python? No, it's hardware

#21

Post by drtgh »

>Rust std fs slower than Python!? No, it's hardware!

>...

>Python features three memory domains, each representing different allocation strategies and optimized for various purposes.

>...

>Rust is slower than Python only on my machine.

if one library performs wildly better than the other in the same test, on the same hardware, how can that not be a software-related problem? sounds like a contradiction.

Maybe should be considered a coding issue and/or feature absent? IMHO it would be expected Rust's std library perform well without making all the users to circumvent the issue manually.

The article is well investigated so I assume the author just want to show the problem existence without creating controversy because other way I can not understand.

Re: Rust std fs slower than Python? No, it's hardware

#22

Post by xuanwo »

quietbritishjim wrote: post #18

I'm a bit confused about the premise. This is not comparing pure Python code against some native (C or Rust) code. It's comparing one Python wrapper around native code (Python's file read method) against another Python wrapper around some native code (OpenDAL). OK it's still interesting that there's a difference in performance, but it's very odd to describe it as "slower than Python". Did they expect that the Python…

Thanks for the comments. I have fixed the headers :)

Re: Rust std fs slower than Python? No, it's hardware

#23

Post by Pop_- »

drtgh wrote: post #21

>Rust std fs slower than Python!? No, it's hardware! >... >Python features three memory domains, each representing different allocation strategies and optimized for various purposes. >... >Rust is slower than Python only on my machine. if one library performs wildly better than the other in the same test, on the same hardware, how can that not be a software-related problem? sounds like a contradiction. Maybe should b…

The root cause is AMD's bad support for rep movsb (which is a hardware problem). However, python by default has a small offset when reading memories while lower level language (rust and c) does not, which is why python seems to perform better than c/rust. It "accidentally" avoided the hardware problem.

Re: Rust std fs slower than Python? No, it's hardware

#26

Post by thechao »

xuanwo wrote: post #6
sgift wrote: post #3

Either the author changed the headline to something less clickbaity in the meantime or you edited it for clickbait Pop_- (in that case: shame on you) - current headline: "Rust std fs slower than Python!? No, it's hardware!"

Sorry for the clickbaity title, I have changed it based on others advice.

I disagree that it's clickbait-y. Diving down from Python bindings to ucode is ... not how things usually go. Doubly so, since Python is a very mature runtime, and I'd be inclined to believe they've dug up file-reading Kung Fu not available to the Average Joe.

Re: Rust std fs slower than Python? No, it's hardware

#27

Post by CoastalCoder »

Pop_- wrote: post #23
drtgh wrote: post #21

>Rust std fs slower than Python!? No, it's hardware! >... >Python features three memory domains, each representing different allocation strategies and optimized for various purposes. >... >Rust is slower than Python only on my machine. if one library performs wildly better than the other in the same test, on the same hardware, how can that not be a software-related problem? sounds like a contradiction. Maybe should b…

The root cause is AMD's bad support for rep movsb (which is a hardware problem). However, python by default has a small offset when reading memories while lower level language (rust and c) does not, which is why python seems to perform better than c/rust. It "accidentally" avoided the hardware problem.

I'm not sure it makes sense to pin this only on AMD.

Whenever you're writing performance-critical software, you need to consider the relevant combinations of hardware + software + workload + configuration.

Sometimes a problem can be created or fixed by adjusting any one / some subset of those details.

Re: Rust std fs slower than Python? No, it's hardware

#28

Post by mwcampbell »

drtgh wrote: post #21

>Rust std fs slower than Python!? No, it's hardware! >... >Python features three memory domains, each representing different allocation strategies and optimized for various purposes. >... >Rust is slower than Python only on my machine. if one library performs wildly better than the other in the same test, on the same hardware, how can that not be a software-related problem? sounds like a contradiction. Maybe should b…

Years ago, Rust's standard library used jemalloc. That decision substantially increased the minimum executable size, though. I didn't publicly complain about it back then (as far as I can recall), but perhaps others did. So the Rust library team switched to using the OS's allocator by default.

Maybe using an alternative allocator only solves the problem by accident and there's another way to solve it intentionally; I don't yet fully understand the problem. My point is that using a different allocator by default was already tried.

Re: Rust std fs slower than Python? No, it's hardware

#29

Post by hobofan »

CoastalCoder wrote: post #27
Pop_- wrote: post #23

Earlier quoted context omitted.

The root cause is AMD's bad support for rep movsb (which is a hardware problem). However, python by default has a small offset when reading memories while lower level language (rust and c) does not, which is why python seems to perform better than c/rust. It "accidentally" avoided the hardware problem.

I'm not sure it makes sense to pin this only on AMD. Whenever you're writing performance-critical software, you need to consider the relevant combinations of hardware + software + workload + configuration. Sometimes a problem can be created or fixed by adjusting any one / some subset of those details.

If that's a bug that only happens with AMD CPUs, I think that's totally fair.

If we start adding in exceptions at the top of the software stack for individuals failures of specific CPUs/vendors, that seems like a strong regression from where we are today in terms of ergonomics of writing performance-critical software. We can't be writing individual code for each N x M x O x P combination of hardware + software + workload + configuration (even if you can narrow down the "relevant" ones).

Re: Rust std fs slower than Python? No, it's hardware

#30

Post by Pop_- »

CoastalCoder wrote: post #27
Pop_- wrote: post #23

Earlier quoted context omitted.

The root cause is AMD's bad support for rep movsb (which is a hardware problem). However, python by default has a small offset when reading memories while lower level language (rust and c) does not, which is why python seems to perform better than c/rust. It "accidentally" avoided the hardware problem.

I'm not sure it makes sense to pin this only on AMD. Whenever you're writing performance-critical software, you need to consider the relevant combinations of hardware + software + workload + configuration. Sometimes a problem can be created or fixed by adjusting any one / some subset of those details.

It's a known issue for AMD and has been tested by multiple people, and by the data provided by the author. It's fair to pin this problem to AMD.
Post reply on HN