Live data from Hacker News

Rust is now overall faster than C in benchmarks

benchmarksgame-team.pages.debian.net

121–130 of 445 posts

Re: Rust is now overall faster than C in benchmarks

#121

Earlier quoted context omitted.

I doubt there's any performance to be gained that way, but if so, the C implementation can just use `restrict` to the same effect.

Have you ever used restrict in anger? I've done it when we really needed that performance for an inner loop(particle system). It can be a real bastard to keep the non-alias constraint held constant in a large, multi-person codebase and the error cases are really gnarly to chase down. Compare that to Rust which has this knowledge built in since it naturally falls out of the ownership model.

When people use the unsafe keyword are they always taking into account aliasing?

At least you can audit only those places though.

Re: Rust is now overall faster than C in benchmarks

#122
post #98

Earlier quoted context omitted.

You raise a good point. I've always been fascinating by PyPy's performance, personally -- anecdotally, I've achieved ~10x speedups from just running a script with `pypy` instead of `python`. I always attributed that to better performance of the JIT, but I could be wrong. I have nothing against Rust personally, but it's ultimately not an apples-to-apples comparison if they're not implementing the same algorithm, or ev…

comparing apples to apples is pointless, they are both apples. and you can always compare “equivalent” algorithms as some languages may not be able to efficiently express the same algorithm as another. i know what you are after, but trying to have some benchmark that is “fair” according to some spec that is important to your needs will just be seen as pointless to others. benchmark game at least lets us see what the…

> Nobody really thinks Rust is faster than C now.

The submission title would beg to differ.

(I know, it explicitly calls out "benchmarks" as the context.)

I think languages like Rust or Swift have significant advantages around safety over C/C++, while not sacrificing much in terms of performance. But if one language's benchmark contributors are willing to put in more effort than another's to eke out additional performance, then you're going to see skewed results in favor of whichever has the more fervent evangelists or whichever language has more to prove.

If the goal is to compare performance of two languages which can express the same optimization in exactly the same way, and only one uses it, then the benchmarks fail in that respect.

Re: Rust is now overall faster than C in benchmarks

#123
post #109

Earlier quoted context omitted.

Rust can be faster than C because in general C compilers have to assume that pointers to memory locations can overlap (unless you mark them __restrict). Rust forbids aliasing pointers. This opens up a whole world of optimizations in the Rust compiler. Broadly speaking this is why Rust can genuinely be faster than C. Same is true in FORTRAN, for what it's worth.

> C compilers have to assume that pointers to memory locations can overlap, unless you mark them __restrict... What I don't fully understand is: "GCC has the option -fstrict-aliasing which enables aliasing optimizations globally and expects you to ensure that nothing gets illegally aliased. This optimization is enabled for -O2 and -O3 I believe." (source: https://stackoverflow.com/a/7298596 ) Doesn't this mean that C…

Not for char. Compiler always assumes non-restrict for char pointers and arrays, which is important to remember if you're ever operating on a RGB or YCbCr matrix or something.

Re: Rust is now overall faster than C in benchmarks

#124

Earlier quoted context omitted.

n-body in C compiled by clang runs just as fast as Rust apparently: https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

It's not entirely surprising that a carefully-optimized C program using explicit SSE intrinsics, plus a fancy trick involving a low-precision square root instruction fixed up with two iterations of Newton's method, would be fast. :-) What impresses me is that the Rust version didn't do any of that stuff, just wrote very boring, straightforward code -- and got the same speed anyway. Some impressive compilation there!

Good point! It would be interesting to find out where the Rust version gets most of its speed from.

Re: Rust is now overall faster than C in benchmarks

#125

Once LLVM fixes some bugs with `noalias`, at which point Rust will begin using it again in more circumstances [1], I'd expect to see Rust get even faster in these benchmarks, given that the Rust compiler knows much more about which pointers do/do-not alias than most other programming languages [2] and the myriad optimizations this knowledge allows. [1] https://github.com/rust-lang/rust/issues/54878#issuecomment-... […

How often does benchmark code have a function that takes two pointers that could potentially alias each other? If it's as rare as I think it is, it might not have that much of an impact on Rust's position in the benchmarks game. Still, real world performance will probably benefit from this fix so it's a positive change regardless.

I've been fairly convinced for a while that once Rust matures (which is probably fairly close to "now", but I've held this opinion for years) that it's going to have a performance advantage in real code that's going to be hard to capture in benchmarks, because it's easy in a small benchmark to be very careful and ensure that you don't have aliasing, avoid extra copies, etc.

Where I expect Rust to really shine performance-wise is at the larger scale of real code, where it affords code that copies less often because the programmer isn't sure in this particular function whether or not they own this so they just have to take a copy, or the compiler can't work out aliasing, etc. Ensuring at scale that you don't take extra copies, or have an aliasing problem, or that you don't have to take copies of things just to "be sure" in multithreading situations, is hard, and drains a lot of performance.

Re: Rust is now overall faster than C in benchmarks

#126

Looking at the reverse-complement code, it appears that the Rust and C implementations are using different algorithms: https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/... On a quick inspection: - The Rust code is about twice as long. - The Rust code has CPU feature detection and SSE intrinsics, while the C code is more idiomatic. - The lookup…

What does "idiomatic" C even mean? It's a high level assembler and as such should not limit the creativity of programmers using it.

C code that pretends it does not need to care about it's platform is not idiomatic, it's just suboptimal.

Re: Rust is now overall faster than C in benchmarks

#127
post #109

Earlier quoted context omitted.

Rust can be faster than C because in general C compilers have to assume that pointers to memory locations can overlap (unless you mark them __restrict). Rust forbids aliasing pointers. This opens up a whole world of optimizations in the Rust compiler. Broadly speaking this is why Rust can genuinely be faster than C. Same is true in FORTRAN, for what it's worth.

> C compilers have to assume that pointers to memory locations can overlap, unless you mark them __restrict... What I don't fully understand is: "GCC has the option -fstrict-aliasing which enables aliasing optimizations globally and expects you to ensure that nothing gets illegally aliased. This optimization is enabled for -O2 and -O3 I believe." (source: https://stackoverflow.com/a/7298596 ) Doesn't this mean that C…

restrict and strict aliasing have to do with the same general concept, but aren't the same. They both have to do with allowing the compiler to optimize around assuming that writes to one pointer won't be visible while reading from another. As a concrete example, can the following branches be merged?

  void foo(/*restrict*/ bool* x, int* y) {
    if (*x) {
      printf("foo\n");
      *y = 0;
    }
    if (*x) {
      printf("bar\n");
    }
  }
Enabling strict aliasing is effectively an assertion that pointers of incompatible types will never point to the same data, so a write to y will never touch *x. restrict is an assertion to the compiler on that specific pointer that no other pointer aliases to it.

Re: Rust is now overall faster than C in benchmarks

#128
post #121

Earlier quoted context omitted.

Have you ever used restrict in anger? I've done it when we really needed that performance for an inner loop(particle system). It can be a real bastard to keep the non-alias constraint held constant in a large, multi-person codebase and the error cases are really gnarly to chase down. Compare that to Rust which has this knowledge built in since it naturally falls out of the ownership model.

When people use the unsafe keyword are they always taking into account aliasing? At least you can audit only those places though.

Yeah, `unsafe` is the developer signing a contract to uphold all the guarantees that safe rustc provided (at least from the perspective of the public interface, the invariants can be broken in private code).

That's why Rust developers don't take kindly to unnecessary unsafe usage, because the audit surface area (and interaction complexity) increases.

Re: Rust is now overall faster than C in benchmarks

#129

Apart from those benchmark games a lot of real world C is a lot less performant than people think it might be. I spent a fair amount of time reviewing C code in the last 5 years - and things that pop up in nearly every review are costly string operations. Linear counts due to the use of null terminated strings and extra allocations for substrings to attach null terminators, or just deep copies because ownership can’t…

Also lack of generics can make it slow, e.g. qsort() requires a function call for each comparison. So C++'s std::sort() can be significantly faster on an array of integers.

I benchmarked it several times in the past and couldn't replicate std::sort being faster (GCC with high optimization settings). Anyway both are slow. If you need fast sort you need an implementation without any function calls (no recursive calls) and both the pivot choice and the chunk size at which insert sort kicks in optimized to your data and hardware. My experience is that you can beat built in sort by 2x to 3x.

Re: Rust is now overall faster than C in benchmarks

#130

When Rust is faster than C in a benchmark in which C++ is also faster than C, I know I can safely ignore such benchmark.

> I know I can safely ignore such benchmark

And yet, rather than ignoring it, you are commenting on it, with a pithy retort which dismisses the entire benchmark without actually providing any additional insight.

Programming languages, compilers, library ecosystems, the groups of people who decide to sit down and try to produce a better result for a given language, and the benchmark maintainers who decide what submissions count for a given language (does a C solution that just uses entirely inline ASM count?) are incredibly complex systems. Any single metric is never going to capture the full richness of the language, is never going to be representative of the experience you will have for every single program, etc.

But does that make metrics useless? No, it just means that you should be informed about their limitations. You shouldn't just look at a single number, but instead make sure you understand well enough what is being measured to know how well that number represents anything useful.

So rather than just dismissing this benchmark, it would be useful to ask "why are the C++ results better than the C results on this benchmark?"

Some benchmark challenges like this allow pretty much any program that accepts the right input and produces the right output; which means you get results in which no computation is actually done, the output is simply hard-coded and you are basically just measuring the startup time or request time of the language or library.

This particular set of benchmarks imposes some constraints to avoid that kind of behavior. Programs have to follow the same basic algorithm, so you can't figure out some clever algorithmic optimization which applies only for the particular input used in this benchmark. For things like the regex challenge, you are expected to use either the built-in regex in your languages standard library, or a common general-purpose regex implementation, not a specialized regex implementation optimized just for this particular benchmark.

The goal of this set of benchmarks is to provide a reasonable set of reasonably realistic small problems, implemented using the same algorithm, and using the normal language and library features. It uses small simple problems in order to make it easy to read the programs and learn about the performance characteristics of the language.

So rather than dismissing, why don't we take a look at the fastest C and C++ implementations of some of the problems?

Here's the fastest implementation of the k-nucleotide problem in C and C++:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

I haven't sat down to do detailed profiling and performance comparison of each; but just off the top of my head, here are a few things I see which could be relevant.

The C++ implementation takes advantage of a number of C++ features; it uses a hash table from GNU pb_ds, a C++ container library which allows for a good amount of customization based on template parameters. The C implementation uses khash, a very fast hash table implementation which uses macros for similar customization.

The C++ implementation makes note of using move semantics in a number of places, which potentially allows for certain optimizations that wouldn't be possible if the compiler had to copy data.

The C++ implementation uses insertion into a templated ordered map to sort the results, while the C implementation uses the standard library qsort. This allows the comparison function to be inlined into the C++ sort, while it's called through a function pointer in the C implementation.

Without actually doing some experimentation and profiling, it's hard to say which of these makes a difference, or if it's something else. But this does show that C++ provides facilities for generic container and algorithm types that C does not. In the C implementation, macros are used to work around this for the hash table case, while function pointers are used for sorting.

Anyhow, rather than simply dismissing these results, why not dig into where the difference really lies, and provide a better implementation in C if you think that you can?

No one set of benchmark results should be taken as gospel. But I think this particular benchmark game is fairly useful for getting a rough sense of "if I write all of my code in this particular language, using either the standard library or commonly available off the shelf libraries, how much of a performance penalty am I likely to pay?"

I also find that the grouping of languages that he does, based on the minima of the kernel density estimation of their geometric mean scores, to be a bit more informative than absolute ranking within those groups. That gives a sense of the general class of languages. There's one group for C, C++, and Rust; languages which allow for performance without compromise, at the expense of lack of safety, higher complexity or learning curve, or both.

There's a next big group with a lot of languages; most of them have been around for a while, or been designed with an eye towards performance, but still have some amount of overhead due to GC or pointer chasing or greater thread synchronizaiton overhead or any number of other reasons; this group includes Fortran, Ada, C#, Java, Go, Haskell, etc.

Then there are a few groups of fairly high-level, dynamic languages, designed for scripting or rapid development, and which require you to trade off a fairly significant amount of performance for this. Dart, PHP, Python, Erlang, Ruby.

And finally, there's Matz's Ruby, all alone in a group at the end, slower than pretty much everything else. I'm not quite sure why it's separated out from Ruby, which seems to refer to yarv, but maybe it's so people who come here wondering what they can do about their slow Ruby programs can see that they can at least get a big boost by switching to yarv.

Anyhow, this benchmark and this grouping helps if you're considering what to do about some performance bottleneck you have in some code base; or if you're starting a project for something which will potentially be performance critical. Moving between languages in one group isn't all that likely to make a substantial difference; but moving to a language in another group would. For example, it lets you know that there's a pretty good chance that just rewriting a Python program that's a performance bottleneck in Go would improve that performance; but rewriting a Go program in Java, or vice versa, is less likely to be a performance win.

Post reply on HN