Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New Rust hash table leads Benchmarks Game

benchmarksgame.alioth.debian.org

Post reply on HN
Print view View original

101–110 of 156 posts

Re: New Rust hash table leads Benchmarks Game

#101

Post by rbehrends »

achamayou wrote: post #96
rbehrends wrote: post #92

Earlier quoted context omitted.

Just be careful to remember that good performance on microbenchmarks does not necessarily translate to good performance on large applications. Some factors to consider are: 1. Cache performance and locality may differ considerably for large applications. For example, a microbenchmark may perform well because it can monopolize the L1 cache in a way that is not possible for larger applications. 2. Aggressive inlining,…

It's still useful as a potential filter. If someone's implemented your microbenchmark in langB and it's within 2x of langA, then maybe they're worth comparing. If on the other hand langB is 20x away from langA in microbenchmarks, you can be pretty sure that it won't ever be a good replacement.

In my experience, if you're seeing a 20x difference, we're either already talking compilers vs. interpreters or fundamentally different implementations (e.g. one using SIMD intrinsics vs. one not using them).

Re: New Rust hash table leads Benchmarks Game

#102

Post by rbehrends »

nostrademons wrote: post #97
stcredzero wrote: post #79

Java is showing quite impressive numbers! 50% overhead over native C implementations was often cited as a good guess for the ultimate efficiency of JIT code generation back in the Self Hotspot days. People who were trying to castigate Go early on as having "Java-like speeds" were really just showing their ignorance of the state of the art of JIT compilation for managed languages and the JVM. Such outdated folk knowle…

HotSpot has had great speeds for numeric computation at least since 2005. I was doing financial software in Java in my first job out of college, our CTO was an ex-Sun architect who literally wrote the book on Java, and the speeds we got on numerical computations were basically equivalent to C. The part where Java really falls down is in memory use & management, which you can see on the binary-tree & mandelbrot benchm…

> The part where Java really falls down is in memory use & management, which you can see on the binary-tree & mandelbrot benchmarks, where it's roughly 4x slower than C.

binary-tree is not useful for comparing GCed and non-GCed languages. For non-GCed languages, you are allowed to use a memory pool of your choice (the C version uses the Apache Portable Runtime library), for GCed languages you are required to use the standard GC with the default settings (no adjustment of GC parameters permitted). This is apples and oranges.

For mandelbrot, the C version uses handcoded SIMD intrinsics. I.e. it's not even portable to non-x86 processors.

Re: New Rust hash table leads Benchmarks Game

#103

Post by burntsushi »

gpm wrote: post #100
igouy wrote: post #69

Previously std::collections::HashMap was used with the default hash function -- [46.03 secs] http://benchmarksgame.alioth.debian.org/u64q/program.php?tes... ) and then the hash function was changed to FnvHasher -- [17.10 secs] http://benchmarksgame.alioth.debian.org/u64q/program.php?tes... and then better use of quad core with futures_cpupool -- [9.44 secs] http://benchmarksgame.alioth.debian.org/u64q/program.php?tes…

> experimental hash table When discussing whether or not to use this at least someone mentioned that there company was using it in production. I don't think it really counts as experimental.

"experimental" is descriptive, not prescriptive:

> Experimental hash table implementation in just Rust (stable, no unsafe code).

https://github.com/bluss/ordermap

Re: New Rust hash table leads Benchmarks Game

#104

Post by pjmlp »

SomeCallMeTim wrote: post #98
pjmlp wrote: post #91

Earlier quoted context omitted.

There are lots of folk programming like the impact of bounds checking or that all game consoles except for XBox use OpenGL. Also many younger developers believe that C was always fast, and are unaware that early 8 and 16 bit compilers for home computers were like managed languages. The compilers generated way worse code than hobby Assembly developers.

> early 8 and 16 bit compilers for home computers were like managed languages Yeah, I wrote several published games entirely in assembly back when that was really the only reasonable option. Of course some of the things we had to deal with meant that even the compilers were good, they couldn't have been good enough . Extreme limited memory is the obvious one, but pages that need to be swapped out at runtime is the ot…

Nice tricks, it brings back memories from demoscene days!

I was just arguing about plain boring C code. :)

Re: New Rust hash table leads Benchmarks Game

#105

Post by achamayou »

nostrademons wrote: post #99
achamayou wrote: post #96

Earlier quoted context omitted.

It's still useful as a potential filter. If someone's implemented your microbenchmark in langB and it's within 2x of langA, then maybe they're worth comparing. If on the other hand langB is 20x away from langA in microbenchmarks, you can be pretty sure that it won't ever be a good replacement.

As long as you have people of roughly equivalent skill level within their language implementing the microbenchmarks. It's not that hard to get a 10x or even 100x speed improvement in the same language when an expert rewrites the naive code that a newbie wrote.

That's why microbenchmarks that have been on the internet for a while are quite nice: there's a good chance at least some experts from each language will have had a go and submitted a decent implementation.

Re: New Rust hash table leads Benchmarks Game

#106

Post by rbehrends »

achamayou wrote: post #105
nostrademons wrote: post #99

Earlier quoted context omitted.

As long as you have people of roughly equivalent skill level within their language implementing the microbenchmarks. It's not that hard to get a 10x or even 100x speed improvement in the same language when an expert rewrites the naive code that a newbie wrote.

That's why microbenchmarks that have been on the internet for a while are quite nice: there's a good chance at least some experts from each language will have had a go and submitted a decent implementation.

But that's a highly (and narrowly) optimized version that took a lot of effort to build. It's not something that the average programmer you have working for you will be able to replicate. Also, some languages will not have equivalent effort put in the corresponding implementations (simple example: some languages have parallelized solutions, some don't).

Re: New Rust hash table leads Benchmarks Game

#107

Post by nostrademons »

rbehrends wrote: post #106
achamayou wrote: post #105

Earlier quoted context omitted.

That's why microbenchmarks that have been on the internet for a while are quite nice: there's a good chance at least some experts from each language will have had a go and submitted a decent implementation.

But that's a highly (and narrowly) optimized version that took a lot of effort to build. It's not something that the average programmer you have working for you will be able to replicate. Also, some languages will not have equivalent effort put in the corresponding implementations (simple example: some languages have parallelized solutions, some don't).

I think the idea behind real-world software performance is that "if you need it, you really need to have it, but most of the time you don't need it." The benchmarks game isn't all that unrealistic for this. Most of the time, you won't care about performance at all. When you do care about performance, you'll be able to devote an experienced engineer to carefully optimize a specific hot spot, not unlike a microbenchmark. It matters how fast you can get the code to go when it's a hot spot, not how fast all your little support code runs.

My one complaint is that there's no benchmark that measures FFI performance. Realistically, if you build a system in Python or Ruby - you're going to be dropping down to C for your hot spots. And so scripting language performance on all these compute-intensive tasks is somewhat irrelevant, you really want to know how much overhead you'll incur crossing the scripting/C boundary (which, in my experience can sometimes be large enough that it wipes out all the gains of coding in C in the first place).

Re: New Rust hash table leads Benchmarks Game

#108

Post by rbehrends »

nostrademons wrote: post #107
rbehrends wrote: post #106

Earlier quoted context omitted.

But that's a highly (and narrowly) optimized version that took a lot of effort to build. It's not something that the average programmer you have working for you will be able to replicate. Also, some languages will not have equivalent effort put in the corresponding implementations (simple example: some languages have parallelized solutions, some don't).

I think the idea behind real-world software performance is that "if you need it, you really need to have it, but most of the time you don't need it." The benchmarks game isn't all that unrealistic for this. Most of the time, you won't care about performance at all. When you do care about performance, you'll be able to devote an experienced engineer to carefully optimize a specific hot spot, not unlike a microbenchmar…

I have some experience with codegolfing for speed and in my experience, these benchmarks do not reflect the reality of such problems, either. If you really want to tweak code for speed, you'll use a mix of algorithmic improvements (that's where the biggest gains are) tailored to your language and compiler/hardware, possibly using FFI/assembly if you absolutely need it. But the benchmark game explicitly disallows algorithmic improvements, for example. Except through the backdoor, where language implementors can sometimes tweak libraries to circumvent restrictions.

And to make things more complicated, the performance increase is generally a function of the time spent on optimizing the problem, which is dependent not only on any innate speed, but also the expressiveness of the language and the programmer's familiarity with the language. Given that you don't have infinite time, there are further tradeoffs here.

Re: New Rust hash table leads Benchmarks Game

#109

Post by igouy »

gpm wrote: post #100
igouy wrote: post #69

Previously std::collections::HashMap was used with the default hash function -- [46.03 secs] http://benchmarksgame.alioth.debian.org/u64q/program.php?tes... ) and then the hash function was changed to FnvHasher -- [17.10 secs] http://benchmarksgame.alioth.debian.org/u64q/program.php?tes... and then better use of quad core with futures_cpupool -- [9.44 secs] http://benchmarksgame.alioth.debian.org/u64q/program.php?tes…

> experimental hash table When discussing whether or not to use this at least someone mentioned that there company was using it in production. I don't think it really counts as experimental.

That is how the author of ordermap currently describes ordermap at the top of the GitHub README.rst

Re: New Rust hash table leads Benchmarks Game

#110

Post by igouy »

bluejekyll wrote: post #54
0xFFC wrote: post #51

Earlier quoted context omitted.

I am asking this genuinely, I always thought and heard LLVM optimizer is inferior to GCC's. Am I wrong ? is there any scientific benchmark for this ?

I think they make different choices in different situations. My understanding has been that they are generally the same, with one doing much better in different situations. But I have nothing to back this up. Wouldn't it be cool if the benchmark game provided a datapoint on this?

"If you're interested in something not shown on the benchmarks game website then please take the program source code and the measurement scripts and publish your own measurements."

http://benchmarksgame.alioth.debian.org/play.html#languagex

Post reply on HN