Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking 20 programming languages on N-queens and matrix multiplication

github.com

Post reply on HN
Print view View original

121–130 of 194 posts

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#121

Post by airstrike »

Very interesting. My two cents: because the stacked bars for php, ruby, perl and py:cpy, it's impossible to compare the other languages in that first chart. All the chart says is that the benchmark is much slower in those 4 languages relative to "all other languages we tested".

It would be nice to see those other languages in a chart that doesn't include the slower four. Alternatively, you could also show those slower four with "broken" columns like this https://peltiertech.com/broken-y-axis-in-excel-chart/

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#123

Post by nithril »

brabel wrote: post #62
Atotalnoob wrote: post #14

Aren’t JIT languages at a disadvantage since they are benchmarked through the CLI rather than using a benchmarking library to allow JIT to warmup?

For one-time problem runs, the JIT languages in practice will not be given time to warm up. All that matters for a user is how fast the application is in practice. It's not about "making it fair" for languages, it's about measuring how fast they go from nothing to results. It doesn't make sense to allow "warmup" time for them unless your expected application is a server which for most of the time will be running "war…

From nothing to result rarely happens in real live. I hardly see someone to start/stop a program per unit task (like piping commands).

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#124

Post by attractivechaos »

wk_end wrote: post #119

This is great, thanks for posting this. OP - are you interested in pull requests adding support for other languages?

Of course! Please implement at least nqueen and matmul as they have been implemented in every language in the benchmark.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#127

Post by montebicyclelo »

The reality is that it would be very hard to find Python code that does not use NumPy (or some tensor lib), for matmul.

Including time to JIT compile is questionable, why not also include time to compile the compiled languages?

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#128

Post by tmtvl »

airstrike wrote: post #121

Very interesting. My two cents: because the stacked bars for php, ruby, perl and py:cpy, it's impossible to compare the other languages in that first chart. All the chart says is that the benchmark is much slower in those 4 languages relative to "all other languages we tested". It would be nice to see those other languages in a chart that doesn't include the slower four. Alternatively, you could also show those slowe…

The second chart is basically the first chart with the slow 4 removed (well, not removed but at such a scale that they're irrelevant).

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#130

Post by gnufx »

brrrrrm wrote: post #40
daxfohl wrote: post #15

Odd that nqueens and sudoku have a high correlation but matmul seems to be largely doing its own thing. nqueen vs. sudoku: 0.531 matmul vs. sudoku: 0.362 matmul vs. nqueen: 0.127

matmul is old and useful -- there's a lot of hardware on a chip that makes it run much faster (prefetch, vectorization, instruction parallelism) and some of these languages have optimizations to expose those things automatically.

Those will not simply give you close to optimal performance (>~90% of peak), even, say, POWER10's 4x4 matmul instruction. You need a structure matching the micro-architecture -- the cache and register structure [1]. That's not a triply-nested loop. Any remotely decent compiler will unroll and vectorize the micro-kernel appropriately, but you may still have to resort to assembler-level prefetch fiddling for the last 10s of percent performance (specifically on avx512). Compilers may recognize the loop structure and replace it with an optimal-ish implementation. I've an idea that FORTRAN H extended did that, but Polly does it in clang.

1. https://www.cs.utexas.edu/users/flame/pubs/blis1_toms_rev3.p...

Post reply on HN