Rust std fs slower than Python? No, it's hardware • Hacker News Board

Re: Rust std fs slower than Python? No, it's hardware

#251

Post by throwaway2037 » Fri, Dec 01, 2023, 5:24 AM UTC

JonChesterfield wrote: post #233

crabbone wrote: post #232

Earlier quoted context omitted.

If I may hazard a guess (I don't know why the original code used it), Python dictionaries are also ordered (and there's no option in Python's library to have them not ordered). Maybe they wanted to match Python's behavior.

Python dicts are ordered in a slightly weird way though, and only since 3.6 (before that it leaked the hashtable implementation scheme). Std::map orders things by some boolean comparator (and duly goes wrong if you give it a partial order comparison), binary tree style. Python currently orders by insertion order, very like storing a list of the keys as well as a hash table, so it can iterate over it in insertion orde…

   ordered in a slightly weird way

Do you mean "insertion ordered"? That means the order of iteration is guaranteed to match insertion order. C++'s std::map is ordered by key (less than comparison) to create a binary search tree. So iteration order will always be ordered by key value. C++'s std::unordered_map has no ordering guarantees (that I know). I don't think the standard C++ template library has the equivalent of a modern Python dict, nor Java LinkedHashMap. Does anyone know if that is incorrect?

Re: Rust std fs slower than Python? No, it's hardware

#252

Post by gpderetta » Fri, Dec 01, 2023, 9:17 AM UTC

renonce wrote: post #237

gpderetta wrote: post #207

Earlier quoted context omitted.

I'm completely making stuff up here, but I wonder if this is the effect of some last minute (or even post-release, via ucode update) bug fix, where page aligned fast rep movs had issues or were subject to some attack and got disabled.

Then fast rep movs should have been disabled in cpuid altogether

They might have benchmarked a set of loads they care about and seen that on average leaving that set was better. But they are all conjectures.

Re: Rust std fs slower than Python? No, it's hardware

#253

Post by the8472 » Fri, Dec 01, 2023, 4:26 PM UTC

collinmanderson wrote: post #246

the8472 wrote: post #244

Earlier quoted context omitted.

As you say, the reported size is not necessarily correct so it should only be treated as a hint. And if os.read directly translates to a read syscall then you're also not handling short reads.

Ahh ok so to be correct you have to keep reading until you get an empty read? Maybe I don’t need to query the file size at all?

querying the file size can be useful to choose the allocation size for a buffer. but yes, you have to keep reading until you get a zero-length read.

https://man7.org/linux/man-pages/man2/read.2.html

> On success, the number of bytes read is returned (zero indicates end of file), [...] It is not an error if this number is smaller than the number of bytes requested

Re: Rust std fs slower than Python? No, it's hardware

#254

Post by int_19h » Sun, Dec 03, 2023, 12:29 AM UTC

crabbone wrote: post #245

int_19h wrote: post #240

Earlier quoted context omitted.

Cython is pretty much Python without bytecode interpreter, translated to C instead, but retaining the object model. That's why it's so slow. And the reason why the object model is the way it is, is because it's an entrenched part of the Python ABI. Sure, if you break that, you can do things a lot faster - this isn't news, people have been doing this with projects like Jython and IronPython that can work a lot faster.…

Well, again, you are confused... and, most likely the CPython developers didn't bother. No. You don't need the Python object model when implementing Python dictionary. You have evidence right in front of you: std::map bindings are successfully used in its place. Why even keep arguing about this? In fact, you can implement your own dictionary, and if you expose all the same mapping protocol, it will work the same as t…

You don't need a Python object model when implementing a highly specialized dictionary that can only store very specific data types. But Python dict is a generic collection type that is designed to store any Python object (or rather, reference to such, since Python has reference semantics for anything).

And this part:

> if anyone would seriously consider improving CPython's performance they wouldn't touch dictionaries, at least not at first.

is just straight up nonsense, given how many times over Python's history dicts have been substantially rewritten. As it happens, I work on Python dev tooling, and the CPython team changing internal data structures for perf reasons has been a recurring headache for me, so I know full well what I'm talking about here.

Re: Rust std fs slower than Python? No, it's hardware

#255

Post by zare_st » Sun, Dec 03, 2023, 2:52 AM UTC

benrutter wrote: post #236

zare_st wrote: post #192

Earlier quoted context omitted.

There is a version of BASIC, a QuickBasic clone called Qb64 that is lightning fast because it transpiles to C++. By your admission a programmer should think that BASIC is fast because he only does BASIC and does not care about the environment details? It's actually the opposite, a Python programmer should know how to offload most, or use the libraries that do so, out of Python into C. He should not be oblivious to th…

I think maybe it's just semantics as long as everyone agrees where the speedup is happening (at the low level language calls). I noticed that you're pretty hard in the "basic isn't fast, the thing it transpiles to is fast" camp, but still accidentally said "there is a version of BASIC [...] that is lightning fast" which I'm not sure you think? Highlights just how tricky it is to talk about where speed lives

I agree with that.

There is clear distinction between original language design (an interpreter) and a project aiming to recreate a sub-standard of that language and support its legacy codebase via a transpiler.