Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust std fs slower than Python? No, it's hardware

xuanwo.io

Post reply on HN
Print view View original

241–250 of 255 posts

Re: Rust std fs slower than Python? No, it's hardware

#241

Post by upsuper »

fsniper wrote: post #239
upsuper wrote: post #224

Earlier quoted context omitted.

Yes, they are proprietary, which is not great. But I don't buy the allegation that they are not indexed or searchable. There are very few IMs that provide builtin publicly accessable log indexed or searchable by default. Does every IRC server come with public log? What about Matrix groups? How do discussion there not get lost in timeline? You can provide public log of them not because they are not proprietary, but th…

This is not a way to have bug discussions, or record them. Do you really think I could find this information on a search for a similar issue? Only thing that makes this bug and the process of the debug visible is this blog post. Another point is I don't think IRC or any instant messaging app is the correct place for this kinds of discussions. Unless important points are logged to some bug reporting tool, or perhaps a…

So there is nothing about being proprietary, but just about using IM?

I don't fully agree. IMs are a great place to discuss issues in a semi-synchronous way. Telecon or face-to-face meetings are sometimes better in velocity, but IMs have some edge on bringing random people happen to be online into the discussion. And it can also bring a different audience into the issue than bug reporting tools or mailing list.

When this issue was brought into the group, it just took several hours for curious people there to collaboratively find the conclusion. This is something unlikely to happen in any other form of discussions based on my experience.

But I agree that group chat is not a great way to record it, and that's why the findings are recorded on the GitHub issue, and group members also encouraged the author to write this up. Then it got posted on HN and on /r/rust by two different group members as well. (The author's initial posting on HN was mysteriously taken down, so the op here helped posting it again.)

Re: Rust std fs slower than Python? No, it's hardware

#242

Post by collinmanderson »

BTW, I've always thought Python uses way too many syscalls when working with files. Simple code like this uses something like 9 syscalls (shown in the article):

    with open('myfile') as f:
        data = f.read()
I'm not much of a C programmer myself. but I at least reported part of the issue to Python: https://bugs.python.org/issue45944

This is the fastest way to read a file on python that I've found, using only 3-4 syscalls (though os.fstat() doesn't work for some special files kernel files like those in /proc/ and /dev/):

    def read_file(path: str, size=-1) -> bytes:
        fd = os.open(path, os.O_RDONLY)
        try:
            if size == -1:
                size = os.fstat(fd).st_size
            return os.read(fd, size)
        finally:
            os.close(fd)

Re: Rust std fs slower than Python? No, it's hardware

#243

Post by fsniper »

upsuper wrote: post #241
fsniper wrote: post #239

Earlier quoted context omitted.

This is not a way to have bug discussions, or record them. Do you really think I could find this information on a search for a similar issue? Only thing that makes this bug and the process of the debug visible is this blog post. Another point is I don't think IRC or any instant messaging app is the correct place for this kinds of discussions. Unless important points are logged to some bug reporting tool, or perhaps a…

So there is nothing about being proprietary, but just about using IM? I don't fully agree. IMs are a great place to discuss issues in a semi-synchronous way. Telecon or face-to-face meetings are sometimes better in velocity, but IMs have some edge on bringing random people happen to be online into the discussion. And it can also bring a different audience into the issue than bug reporting tools or mailing list. When…

Oh I missed the github issue. My bad. In fact I searched github and probably my github search foo was not good. Sorry to disregard that fact.

If these are already in place I don't have any reservations against using IMs or Discord. In fact this is particularly great sample how this can be done.

- Bug report is in place - Blog/Article for historic events , documentation and pointers for related info - Fast communication for debug sessions

I hope you understand my original message was about pointing out a situation where only left overs are history of these chat tools.

There are lots of communities right now just using discord or IM for support, bug reporting or development purposes.

Re: Rust std fs slower than Python? No, it's hardware

#244

Post by the8472 »

collinmanderson wrote: post #242

BTW, I've always thought Python uses way too many syscalls when working with files. Simple code like this uses something like 9 syscalls (shown in the article): with open('myfile') as f: data = f.read() I'm not much of a C programmer myself. but I at least reported part of the issue to Python: https://bugs.python.org/issue45944 This is the fastest way to read a file on python that I've found, using only 3-4 syscalls…

As you say, the reported size is not necessarily correct so it should only be treated as a hint. And if os.read directly translates to a read syscall then you're also not handling short reads.

Re: Rust std fs slower than Python? No, it's hardware

#245

Post by crabbone »

int_19h wrote: post #240
crabbone wrote: post #230

Earlier quoted context omitted.

You are very confused between how something works right now and how it can work in principle. In this you very much resemble CPython developers: they never attempt optimizations that go beyond what Python C API can offer. This is very limiting (and, this is why all sorts of Python JIT compilers can in many circumstances beat CPython by a lot). The evidence to how absurd your claim is is right in front of you: Google'…

Cython is pretty much Python without bytecode interpreter, translated to C instead, but retaining the object model. That's why it's so slow. And the reason why the object model is the way it is, is because it's an entrenched part of the Python ABI. Sure, if you break that, you can do things a lot faster - this isn't news, people have been doing this with projects like Jython and IronPython that can work a lot faster.…

Well, again, you are confused... and, most likely the CPython developers didn't bother.

No. You don't need the Python object model when implementing Python dictionary. You have evidence right in front of you: std::map bindings are successfully used in its place.

Why even keep arguing about this?

In fact, you can implement your own dictionary, and if you expose all the same mapping protocol, it will work the same as the built-in one. Do you have to use Python objects for this? -- absolutely no. You can convert at the interface boundary. Experience shows that this works noticeably better than using Python objects all the way. Why did the original CPython developers not do it? -- I don't know, can only guess. I already wrote what my guess is. And, in all sincerity, CPython has a lot more and a lot worse problems. Compared to the rest of the codebase, the dictionary object is fine. So, if anyone would seriously consider improving CPython's performance they wouldn't touch dictionaries, at least not at first.

Re: Rust std fs slower than Python? No, it's hardware

#246

Post by collinmanderson »

the8472 wrote: post #244
collinmanderson wrote: post #242

BTW, I've always thought Python uses way too many syscalls when working with files. Simple code like this uses something like 9 syscalls (shown in the article): with open('myfile') as f: data = f.read() I'm not much of a C programmer myself. but I at least reported part of the issue to Python: https://bugs.python.org/issue45944 This is the fastest way to read a file on python that I've found, using only 3-4 syscalls…

As you say, the reported size is not necessarily correct so it should only be treated as a hint. And if os.read directly translates to a read syscall then you're also not handling short reads.

Ahh ok so to be correct you have to keep reading until you get an empty read?

Maybe I don’t need to query the file size at all?

Re: Rust std fs slower than Python? No, it's hardware

#247

Post by saagarjha »

andrekandre wrote: post #223
saagarjha wrote: post #220

Earlier quoted context omitted.

On certain platforms, it would break code signatures if they are tied to the pages the code is on.

> On certain platforms, it would break code signatures macos?

Yeah, or iOS. Or other platforms that adopt a similar model

Re: Rust std fs slower than Python? No, it's hardware

#248

Post by saagarjha »

munch117 wrote: post #234
saagarjha wrote: post #222

Earlier quoted context omitted.

You can look at the history of PyObject yourself: https://github.com/python/cpython/commits/main/Include/objec... . None of these changes were done because of weird CPU errata that meant that making the header bigger was a performance win. That isn't to say that the developers wouldn't be interested in such effects, or be able to detect them, but the fact that the object header happens to be large enough to avoid the…

What you don't see in the logs are the experiments and branches that weren't pursued further because they didn't perform well enough. Also: If you're going to prove that changes informed by performance measurements are absent from the commit logs, then you'll need to look in the logs for all the relevant places, which means also looking at I/O and bytes and allocator code.

Given that the performance is only affected by the size of that object header, the file I linked is all you'd need to see changes in. Look, the Python project is not picking their object sizes because it performs well on a quirk of Zen 3. End of story. I did performance work professionally in the past and now recreationally and this specific instance is 100% luck. This is not because I don't think the runtime people aren't smart or anything but this would be an insane thing to do on purpose.

Re: Rust std fs slower than Python? No, it's hardware

#249

Post by lmm »

JonChesterfield wrote: post #212
lmm wrote: post #195

Earlier quoted context omitted.

> Because for any nontrivial case you would expect python+compiled library and associated marshaling of data to be slower than that library in its native implementation without any inyerop/marshaling required. > When you see an interpreted language faster than a compiled one, it's worth looking at why, because most the time it's because there's some hidden issue causing the other to be slow (which could just be a dif…

> On the contrary, the compiled languages tend to only be faster in trivial benchmarks. In real-world systems the Python-based systems tends to be faster because they haven't had to spend so long twiddling which integers they're using and debugging crashes and memory leaks, and got to spend more time on the problem. This is an interesting premise. Python in particular gets an absolute kicking for being slow. Hence al…

> Can you cite any single example where this is the case?

Plenty of line-of-business systems I've seen, but systems big enough to matter tend not to be public. Bitbucket's cloud and on-prem version are the only case I can think of where you can directly compare something substantial between an implementation known to be written in Python and an implementation that's known to be written in C/C++ (and even then I'm not 100% that that's what they use).

Re: Rust std fs slower than Python? No, it's hardware

#250

Post by jwueller »

munch117 wrote: post #202
jwueller wrote: post #194

Earlier quoted context omitted.

How is it pure Python if it delegates all of the actual work to the Kernel?

All I/O delegates to the kernel, eventually. It's pure Python in that there's no cffi, no ctypes, no Cython, no C extensions of any kind.

Yes, which is why I would argue that IO is a particularly bad benchmark here, since everything is just a thin layer on top of the actual syscall, and those layers don't do any real work worth comparing.

The only thing that makes sense to compare when talking about pythons performance is how many instructions it needs to compute something, versus the instructions needed to compute the same thing in C. Those are probably a few orders of magnitude apart.

Post reply on HN