Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust std fs slower than Python? No, it's hardware

xuanwo.io

Post reply on HN
Print view View original

191–200 of 255 posts

Re: Rust std fs slower than Python? No, it's hardware

#191

Post by rowanG077 »

codr7 wrote: post #118
crabbone wrote: post #81

Earlier quoted context omitted.

> individually, highly optimised. Now why would you expect that ? What happened to OP is a pure chance. CPython's C code doesn't even care about const-consistency. It's flush with dynamic memory allocations, bunch of helper / convenience calls... Even stuff like arithmetic does dynamic memory allocation... Normally, you don't expect CPython to perform well, not if you have any experience working with it. Whenever you…

Have you ever attempted to write a scripting language that performs better? I have, several, and it's far from trivial. The basics are seriously optimized for typical use cases, take a look at the source code for the dict type.

> Have you ever attempted to write a scripting language that performs better?

Way to miss the mark. The point is precisely that Python is slow and one of the causes is that it is a scripting language. Stomping your foot and essentially: "You couldn't do any better" helps no one and is counterproductive.

Re: Rust std fs slower than Python? No, it's hardware

#192

Post by zare_st »

benrutter wrote: post #119
qd011 wrote: post #104

Earlier quoted context omitted.

I don't understand why Python gets shit for being a slow language when it's slow but no credit for being fast when it's fast just because "it's not really Python". If I write Python and my code is fast, to me that sounds like Python is fast, I couldn't care less whether it's because the implementation is in another language or for some other reason.

I wonder if its because we're sometimes talking cross purposes. For me, coding is almost exclusively using python libraries like numpy to call out to other languages like c or FORTRAN. It feels silly to say I'm not coding in Python to me. On the other hand, if you're writing those libraries, coding to you is mostly writing FORTRAN and c optimizations. It probably feels silly to say you're coding in Python just becaus…

There is a version of BASIC, a QuickBasic clone called Qb64 that is lightning fast because it transpiles to C++. By your admission a programmer should think that BASIC is fast because he only does BASIC and does not care about the environment details?

It's actually the opposite, a Python programmer should know how to offload most, or use the libraries that do so, out of Python into C. He should not be oblivious to the fact that any decent Python performance is due to shrinking down the ratio of actual Python instructions vs native instructions.

Re: Rust std fs slower than Python? No, it's hardware

#193

Post by zare_st »

analog31 wrote: post #171
qd011 wrote: post #104

Earlier quoted context omitted.

I don't understand why Python gets shit for being a slow language when it's slow but no credit for being fast when it's fast just because "it's not really Python". If I write Python and my code is fast, to me that sounds like Python is fast, I couldn't care less whether it's because the implementation is in another language or for some other reason.

I think the confusion comes from people not having a good understanding of what an interpreted programming language does, and what actual portion of time is spent in high versus low level code. I've always assumed that most of my programs amount to a bit of glue thrown in between system calls. Also, when we talk about "faster" and "slower," it's not clear the order of magnitude. Maybe an analysis of actual code execu…

Agreed. The speed of a language is reverse proportional to number of CPU instructions emitted to do something meaningful, e.g. solve a problem. Not whether it can target system calls without overhead and move memory around freely. That's a given.

Re: Rust std fs slower than Python? No, it's hardware

#194

Post by jwueller »

munch117 wrote: post #162
IshKebab wrote: post #152

Earlier quoted context omitted.

Because when people talk about Python performance they're talking about the performance of Python code itself, not C/Rust code that it's wrapping. Pretty much any language can wrap C/Rust code. Why does it matter? 1. Having to split your code across 2 languages via FFI is a huge pain. 2. You are still writing some Python. There's plenty of code that is pure Python. That code is slow.

Of course in this case there's no FFI involved - the open function is built-in. It's as pure-Python as it can get.

How is it pure Python if it delegates all of the actual work to the Kernel?

Re: Rust std fs slower than Python? No, it's hardware

#195

Post by lmm »

kbenson wrote: post #128
qd011 wrote: post #104

Earlier quoted context omitted.

I don't understand why Python gets shit for being a slow language when it's slow but no credit for being fast when it's fast just because "it's not really Python". If I write Python and my code is fast, to me that sounds like Python is fast, I couldn't care less whether it's because the implementation is in another language or for some other reason.

Because for any nontrivial case you would expect python+compiled library and associated marshaling of data to be slower than that library in its native implementation without any inyerop/marshaling required. When you see an interpreted language faster than a compiled one, it's worth looking at why, because most the time it's because there's some hidden issue causing the other to be slow (which could just be a differe…

> Because for any nontrivial case you would expect python+compiled library and associated marshaling of data to be slower than that library in its native implementation without any inyerop/marshaling required.

> When you see an interpreted language faster than a compiled one, it's worth looking at why, because most the time it's because there's some hidden issue causing the other to be slow (which could just be a different and much worse implementation).

On the contrary, the compiled languages tend to only be faster in trivial benchmarks. In real-world systems the Python-based systems tends to be faster because they haven't had to spend so long twiddling which integers they're using and debugging crashes and memory leaks, and got to spend more time on the problem.

Re: Rust std fs slower than Python? No, it's hardware

#196

Post by kbenson »

lmm wrote: post #195
kbenson wrote: post #128

Earlier quoted context omitted.

Because for any nontrivial case you would expect python+compiled library and associated marshaling of data to be slower than that library in its native implementation without any inyerop/marshaling required. When you see an interpreted language faster than a compiled one, it's worth looking at why, because most the time it's because there's some hidden issue causing the other to be slow (which could just be a differe…

> Because for any nontrivial case you would expect python+compiled library and associated marshaling of data to be slower than that library in its native implementation without any inyerop/marshaling required. > When you see an interpreted language faster than a compiled one, it's worth looking at why, because most the time it's because there's some hidden issue causing the other to be slow (which could just be a dif…

I don't doubt that can happen, but I'm also highly doubtful that it's the norm for large, established, mature projects with lots of attention, such as popular libraries and the standard library of popular languages. As time spent on the project increases, I suspect that any gain an interpreted language has over an (efficient) compiled one not only gets smaller, but eventually reverses in most cases.

So, like in most things, the details can sometimes matter quite a bit.

Re: Rust std fs slower than Python? No, it's hardware

#197

Post by lmm »

kbenson wrote: post #196
lmm wrote: post #195

Earlier quoted context omitted.

> Because for any nontrivial case you would expect python+compiled library and associated marshaling of data to be slower than that library in its native implementation without any inyerop/marshaling required. > When you see an interpreted language faster than a compiled one, it's worth looking at why, because most the time it's because there's some hidden issue causing the other to be slow (which could just be a dif…

I don't doubt that can happen, but I'm also highly doubtful that it's the norm for large, established, mature projects with lots of attention, such as popular libraries and the standard library of popular languages. As time spent on the project increases, I suspect that any gain an interpreted language has over an (efficient) compiled one not only gets smaller, but eventually reverses in most cases. So, like in most…

> I don't doubt that can happen, but I'm also highly doubtful that it's the norm for large, established, mature projects with lots of attention, such as popular libraries and the standard library of popular languages.

Code that has lots of attention is different, certainly, but it's also the exception rather than the rule; the last figure I saw was that 90% of code is internal business applications that are never even made publicly available in any form, much less subject to outside code review or contributions.

> As time spent on the project increases, I suspect that any gain an interpreted language has over an (efficient) compiled one not only gets smaller, but eventually reverses in most cases.

In terms of the limit of an efficient implementation (which certainly something like Python is nowhere near), I've seen it argued both ways; with something like K the argument is that a tiny interpreter that sits in L1 and takes its instructions in a very compact form ends up saving you more memory bandwidth (compared to what you'd have to compile those tiny interpreter instructions into if you wanted them to execute "directly") than it costs.

Re: Rust std fs slower than Python? No, it's hardware

#198

Post by p5a0u9l »

qd011 wrote: post #104
quietbritishjim wrote: post #18

I'm a bit confused about the premise. This is not comparing pure Python code against some native (C or Rust) code. It's comparing one Python wrapper around native code (Python's file read method) against another Python wrapper around some native code (OpenDAL). OK it's still interesting that there's a difference in performance, but it's very odd to describe it as "slower than Python". Did they expect that the Python…

I don't understand why Python gets shit for being a slow language when it's slow but no credit for being fast when it's fast just because "it's not really Python". If I write Python and my code is fast, to me that sounds like Python is fast, I couldn't care less whether it's because the implementation is in another language or for some other reason.

I constantly get low key shade for choosing to build everything in Python. It’s really interesting to me. People can’t break out of thinking, “oh, you wrote a script for that?”. Actually, no, it’s software, not a script.

99% of my use cases are easily, maintainably solved with good, modern Python. The Python execution is almost never the bottleneck in my workflows. It’s disk or network I/O.

I’m not against building better languages and ecosystems, and compiled languages are clearly appropriate/required in many workflows, but the language parochialism gets old. I just want to build shit that works and get stuff done.

Re: Rust std fs slower than Python? No, it's hardware

#199

Post by p5a0u9l »

fl0ki wrote: post #163
quietbritishjim wrote: post #18

I'm a bit confused about the premise. This is not comparing pure Python code against some native (C or Rust) code. It's comparing one Python wrapper around native code (Python's file read method) against another Python wrapper around some native code (OpenDAL). OK it's still interesting that there's a difference in performance, but it's very odd to describe it as "slower than Python". Did they expect that the Python…

The premise is that any time you say "Python [...] faster than Rust [...]" you get page views even if it's not true. People have noticed after the last few dozen times something like this was posted.

This is the answer. The thread is chasing various smart-people opinions about languages, interpreters, system calls. We got tricked into click bait title and are using the opportunity to rehash our favorite topics and biases.

On the other hand… so what? It’s kind of fun.

Re: Rust std fs slower than Python? No, it's hardware

#200

Post by josephg »

masklinn wrote: post #164
josephg wrote: post #149

Earlier quoted context omitted.

> jemalloc is still in use for rustc (again - not sure why they haven’t switched to mimalloc) Performance of rustc matters a lot! If the rust compiler runs faster when using mimalloc, please benchmark & submit a patch to the compiler.

I literally linked two attempts to use mimalloc in rustc just a few comments upthread.

Ah - my mistake; I somehow misread your comment. Pity about the RSS regression.

Personally I have plenty of RAM and I'd happily use more in exchange for a faster compile. Its much cheaper to buy more ram than a faster CPU, but I certainly understand the choice.

With compilers I sometimes wonder if it wouldn't be better to just switch to an arena allocator for the whole compilation job. But it wouldn't surprise me if LLVM allocates way more memory than you'd expect.

Post reply on HN