Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking 20 programming languages on N-queens and matrix multiplication

github.com

Post reply on HN
Print view View original

91–100 of 194 posts

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#91

Post by latenightcoding »

d0mine wrote: post #68
latenightcoding wrote: post #21

These are not the best benchmarks, but Python is indeed as slow as Perl, which I find insane considering that Python has 100-1000 more people working on the interpreter and performance has been a big emphasis the last few years.

The benchmark is not representative of how Python is actually used in practice. It ignores the existence of libraries (explicitly). For example, If numpy, pytorch were used for matmul, the results would be completely different.

Perl people would use PDL or Math::GSL. I look into these benchmarks because I'm interested in VMs, and yeah the Python VM can be quite slow if you are not calling C code.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#92

Post by WhereIsTheTruth »

keyle wrote: post #43

I love a good benchmark, thanks for putting this together! However, I have a bit of feedback. First, the graph is misleading, stacking times with languages that have half the implementation, they appear faster, until you dig in. I'd suggest producing an alternate graph that shows only the implemented puzzles in every language, or make a unique graph for every language:puzzle. Second, the examples are taken from roset…

There is one for data processing here: https://github.com/jinyus/related_post_gen

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#93

Post by cm2187 »

keyle wrote: post #43

I love a good benchmark, thanks for putting this together! However, I have a bit of feedback. First, the graph is misleading, stacking times with languages that have half the implementation, they appear faster, until you dig in. I'd suggest producing an alternate graph that shows only the implemented puzzles in every language, or make a unique graph for every language:puzzle. Second, the examples are taken from roset…

Also it may or may not be relevant to not exclude the warm up time for JIT languages. If it is for a calculation you will do continuously, including the warm up time may bias the results materially (I know for .net it can be significant).

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#94

Post by cm2187 »

yxhuvud wrote: post #72
lifthrasiir wrote: post #6

> It is obvious that c[i], b[k] and a[i][k] can be moved out of the inner loop to reduce the frequency of matrix access. [...] However, most other languages cannot optimize this nested loop. If we manually move a[i][k] to the loop above it, we can often improve their performance. This is only true when three matrices are independent of each other, and also why C has a `restrict` qualifier that enables this assumption…

In some cases there are other reasons for hoiting the dereference too. For example, Crystal will check if the array access is out of bounds and by hoisting variables that will be done a lot less seldom, which can have huge effects for code that does a lot of that, like matmul.

Doesn’t that get mostly optimised away by the cpu branch prediction?

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#95

Post by Aissen »

spense wrote: post #78
hyperman1 wrote: post #61

I wondered why rust is so far behind C/nim/zig, they should have similar behaviour. The difference is mostly in matmul. I see how C, for an n x n matrix, does 2 allocations, while rust does n+1. C's matrix rows are right next to each other, rusts are probably all over the place. Didn't look at nim or zig. Maybe a slice of slice of double would perform better than a vec of vec of double? Then again, an argument can be…

rust is equivalent to C when using static allocation i sent a pr to fix it

Doesn't your PR monomorphize the function every time N is changed ? I realize it's simpler since it keeps the structure, simplifies allocation of arrays, and elides bound check. But it explodes generated code size and matrix size can't be changed at runtime, which doesn't really match C.

Edit: I have tried making an iterator-based version to elide bound checks, but had to resort to unsafe, and it's barely 50% faster than the original rust version (not as fast as C): https://gist.github.com/anisse/6b580628206293ef242faa7db6219...

Edit 2: updated, and my rust iterator version now ~equivalent to C with no unsafe.

Edit 3: too late, the repo has been updated with an other iterator-based version that is just as fast.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#96

Post by spense »

Aissen wrote: post #95
spense wrote: post #78

Earlier quoted context omitted.

rust is equivalent to C when using static allocation i sent a pr to fix it

Doesn't your PR monomorphize the function every time N is changed ? I realize it's simpler since it keeps the structure, simplifies allocation of arrays, and elides bound check. But it explodes generated code size and matrix size can't be changed at runtime, which doesn't really match C. Edit: I have tried making an iterator-based version to elide bound checks, but had to resort to unsafe, and it's barely 50% faster…

C, Nim and Zig all benefit from static allocation and it's available in rust, so seems fair?

also, `const N` is used in rust's nqueen test, so figured it would be fine.

someone posted an `.iter().zip()` solution which benchmarks only a little slower (+10ms) than static allocation:

https://github.com/attractivechaos/plb2/pull/4#issuecomment-...

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#97

Post by Aissen »

spense wrote: post #96
Aissen wrote: post #95

Earlier quoted context omitted.

Doesn't your PR monomorphize the function every time N is changed ? I realize it's simpler since it keeps the structure, simplifies allocation of arrays, and elides bound check. But it explodes generated code size and matrix size can't be changed at runtime, which doesn't really match C. Edit: I have tried making an iterator-based version to elide bound checks, but had to resort to unsafe, and it's barely 50% faster…

C, Nim and Zig all benefit from static allocation and it's available in rust, so seems fair? also, `const N` is used in rust's nqueen test, so figured it would be fine. someone posted an `.iter().zip()` solution which benchmarks only a little slower (+10ms) than static allocation: https://github.com/attractivechaos/plb2/pull/4#issuecomment-...

I only looked at the C version, and it didn't use a generic over the allocation size, although yes, technically it was fixed at compile time.

I saw the update to the PR, but I still prefer my collect()-based version :-)

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#98

Post by attractivechaos »

jakobnissen wrote: post #22

Neat! The Julia matmul implementation has its rows and columns flipped though - unlike C, Julia uses row-major matrices. This has large implications for speed. Also, the code may be much faster if you enable SIMD in the function, which is disabled in the code because a) the code unnecessarily checks bounds at every index instead of at the top of the function, and b) float SIMD is opt-in since SIMD changes the roundin…

Just come here to say this has been fixed in PR #2. The figure has been updated accordingly.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#99

Post by attractivechaos »

hyperman1 wrote: post #61

I wondered why rust is so far behind C/nim/zig, they should have similar behaviour. The difference is mostly in matmul. I see how C, for an n x n matrix, does 2 allocations, while rust does n+1. C's matrix rows are right next to each other, rusts are probably all over the place. Didn't look at nim or zig. Maybe a slice of slice of double would perform better than a vec of vec of double? Then again, an argument can be…

FYI: rust has been updated to avoid unnecessary bound check in PR #4. It now matches the C version in performance.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#100

Post by attractivechaos »

keyle wrote: post #43

I love a good benchmark, thanks for putting this together! However, I have a bit of feedback. First, the graph is misleading, stacking times with languages that have half the implementation, they appear faster, until you dig in. I'd suggest producing an alternate graph that shows only the implemented puzzles in every language, or make a unique graph for every language:puzzle. Second, the examples are taken from roset…

Clarification: the examples are not taken from Rosetta code. Only the n-queens algorithm is inspired by (but still different from) an Rosetta code implementation; otherwise this benchmark has nothing to do with Rosetta code.
Post reply on HN