Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust std fs slower than Python? No, it's hardware

xuanwo.io

Post reply on HN
Print view View original

41–50 of 255 posts

Re: Rust std fs slower than Python? No, it's hardware

#41

Post by fsniper »

The article itself is a great read and it has fascinating info related to this issue.

However I am more interested/concerned about another part. How the issue is reported/recorded and how the communications are handled.

Reporting is done over discord, which is a proprietary environment which is not indexed, or searchable. Will not be archived.

Communications and deliberations are done over discord and telegram, which is probably worse than discord in this context.

This blog post and the github repository is the lingering remains of them. If Xuanwo did not blog this. It would be lost in timeline.

Isn't this fascinating?

Re: Rust std fs slower than Python? No, it's hardware

#43

Post by jpc0 »

hobofan wrote: post #29
CoastalCoder wrote: post #27

Earlier quoted context omitted.

I'm not sure it makes sense to pin this only on AMD. Whenever you're writing performance-critical software, you need to consider the relevant combinations of hardware + software + workload + configuration. Sometimes a problem can be created or fixed by adjusting any one / some subset of those details.

If that's a bug that only happens with AMD CPUs, I think that's totally fair. If we start adding in exceptions at the top of the software stack for individuals failures of specific CPUs/vendors, that seems like a strong regression from where we are today in terms of ergonomics of writing performance-critical software. We can't be writing individual code for each N x M x O x P combination of hardware + software + work…

> We can't be writing individual code for each N x M x O x P combination of hardware + software + workload + configuration

That is kind of exactly what you would do when optimising for popular platforms.

If this error occurs on an AMD Cpu used by half your users is your response to your user going to be "just buy a different CPU" or are you going to fix it in code and ship a "performance improvement on XYZ platform" update

Re: Rust std fs slower than Python? No, it's hardware

#44

Post by xuanwo »

j16sdiz wrote: post #39
lynndotpy wrote: post #37

Earlier quoted context omitted.

One of the very first things in the article is a TLDR section that points you to the conclusion. > In conclusion, the issue isn't software-related. Python outperforms C/Rust due to an AMD CPU bug.

It is software-related. Just the CPU perform badly on some software instruction.

FSRM is a CPU feature embedded in the microcode (in this instance, amd-ucode) that software such as glibc cannot interact with. I refer to it as hardware because I consider microcode a part of the hardware.

Re: Rust std fs slower than Python? No, it's hardware

#48

Post by Waterluvian »

pmontra wrote: post #47

> However, mmap has other uses too. It's commonly used to allocate large regions of memory for applications. Slack is allocating 1132 GB of virtual memory on my laptop right now. I don't know if they are using mmap but that's 1100 GB more than the physical memory.

I’m not sure allocations mean anything practical anymore. I recall OSX allocating ridiculous amounts of virtual memory to stuff but never found OSX or the software to ever feel slow and pagey.

Re: Rust std fs slower than Python? No, it's hardware

#49

Post by richardwhiuk »

hobofan wrote: post #29
CoastalCoder wrote: post #27

Earlier quoted context omitted.

I'm not sure it makes sense to pin this only on AMD. Whenever you're writing performance-critical software, you need to consider the relevant combinations of hardware + software + workload + configuration. Sometimes a problem can be created or fixed by adjusting any one / some subset of those details.

If that's a bug that only happens with AMD CPUs, I think that's totally fair. If we start adding in exceptions at the top of the software stack for individuals failures of specific CPUs/vendors, that seems like a strong regression from where we are today in terms of ergonomics of writing performance-critical software. We can't be writing individual code for each N x M x O x P combination of hardware + software + work…

You are going to be disappointed when you find out there's lots of architecture and CPU specific code in software libraries and the kernel.

Re: Rust std fs slower than Python? No, it's hardware

#50

Post by Pop_- »

pmontra wrote: post #47

> However, mmap has other uses too. It's commonly used to allocate large regions of memory for applications. Slack is allocating 1132 GB of virtual memory on my laptop right now. I don't know if they are using mmap but that's 1100 GB more than the physical memory.

I don't know why but this really makes me laugh
Post reply on HN