Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking 20 programming languages on N-queens and matrix multiplication

github.com

Post reply on HN
Print view View original

71–80 of 194 posts

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#71

Post by ggm »

p-e-w wrote: post #5

What is this supposed to demonstrate? There appears to be roughly the same code structure, ported to every language, while for some languages, arbitrary optimizations are introduced (such as using `array` instead of `list` in Python). But nobody working in Python uses matrix multiplication code written in Python. They use NumPy, which is a de facto standard library for people working in the relevant fields. It's as m…

So you'd be measuring the speed of loops over out calls to c or Fortran (nag library) or their vectorisation and only testing the cost of data representation changes

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#72

Post by yxhuvud »

lifthrasiir wrote: post #6

> It is obvious that c[i], b[k] and a[i][k] can be moved out of the inner loop to reduce the frequency of matrix access. [...] However, most other languages cannot optimize this nested loop. If we manually move a[i][k] to the loop above it, we can often improve their performance. This is only true when three matrices are independent of each other, and also why C has a `restrict` qualifier that enables this assumption…

In some cases there are other reasons for hoiting the dereference too. For example, Crystal will check if the array access is out of bounds and by hoisting variables that will be done a lot less seldom, which can have huge effects for code that does a lot of that, like matmul.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#73

Post by yxhuvud »

brrrrrm wrote: post #40
daxfohl wrote: post #15

Odd that nqueens and sudoku have a high correlation but matmul seems to be largely doing its own thing. nqueen vs. sudoku: 0.531 matmul vs. sudoku: 0.362 matmul vs. nqueen: 0.127

matmul is old and useful -- there's a lot of hardware on a chip that makes it run much faster (prefetch, vectorization, instruction parallelism) and some of these languages have optimizations to expose those things automatically.

And some languages that perform array bounds check on each array deindex operation, which basically stops some of the above even if the optimizer can do all of it. Crystal is an example of that, where it would be possible and quite straightforward to write a specialized matrix class instead of the very generic dynamic array implementation.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#74

Post by hgomersall »

nrabulinski wrote: post #63
hyperman1 wrote: post #61

I wondered why rust is so far behind C/nim/zig, they should have similar behaviour. The difference is mostly in matmul. I see how C, for an n x n matrix, does 2 allocations, while rust does n+1. C's matrix rows are right next to each other, rusts are probably all over the place. Didn't look at nim or zig. Maybe a slice of slice of double would perform better than a vec of vec of double? Then again, an argument can be…

The rust code is very unidiomatic, not only because of the Vec of Vecs which I’d say, even if it’s the obvious naive approach, no one experienced wouldn’t choose over a flat slice, the implementation itself is very naive and unidiomatic.

Also it's using checked indexing, which apart from being not idiomatic, is also going to slow things down. A fairer comparison would be to use the unchecked indexing variants.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#76

Post by rafark »

brrrrrm wrote: post #36
habibur wrote: post #9

Earlier quoted context omitted.

I put the pure math codes in C extension of PHP. Building C extension for PHP is easier than most of the other high level languages. And then things get blazingly fast.

doesn't look that easy... zend_parse_parameters? pre-baked configure + make scripts? check out how a modern language deals with this stuff https://bun.sh/docs/api/ffi#usage

Checkout what? PHP also has support for FFI. Or am I missing something?

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#77

Post by tsimionescu »

lifthrasiir wrote: post #42
tsimionescu wrote: post #35

Earlier quoted context omitted.

Generally, the point of language benchmarks is to show how the languages compare at solving the same problem, without external libraries. Including external libraries is pointless since any language can call any library, ultimately, so at best you'd be comparing the FFI overhead. So this shouldn't be taken as "how fast does a real-world Python program do at matrix multiplication", since of course no one writes real-w…

> Generally, the point of language benchmarks is to show how the languages compare at solving the same problem, without external libraries. If this were the only concern, it should be valid to create a blob of binary and call that function for the optimal performance. (Python's ctypes makes this very easy, for example.) So you want an idiomatic solution instead, and Numpy for matrix computation is considered idiomati…

No, creating a binary blob is not a realistic option, and it's still not a Python/Java/C++ whatever solution. It would be handwritten assembly that you use the FFI facilities to call into. So, it's not fair game in any honest benchmark.

And even if using a C library is idiomatic Python, it still has no place in a language benchmark. It's a C library, not a Python implementation.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#78

Post by spense »

hyperman1 wrote: post #61

I wondered why rust is so far behind C/nim/zig, they should have similar behaviour. The difference is mostly in matmul. I see how C, for an n x n matrix, does 2 allocations, while rust does n+1. C's matrix rows are right next to each other, rusts are probably all over the place. Didn't look at nim or zig. Maybe a slice of slice of double would perform better than a vec of vec of double? Then again, an argument can be…

rust is equivalent to C when using static allocation

i sent a pr to fix it

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#79

Post by morepedantic »

lifthrasiir wrote: post #24
morepedantic wrote: post #13

Earlier quoted context omitted.

IMO, uncompressed bytes is a better representation, because it can be used to compare relative expressive power for the particular problem. I'd bet Python cleans house here, but the write-only languages are a wild card.

"Expressive power" is a very subjective term, and uncompressed size is a bad proxy as it includes too many variables specific to coding conventions. Compressed size with a stupid enough algorithm (here gzip) is meant to reduce these variables. The true Kolmogorov complexity in comparison can't be computed, and too smart algorithms can start to infer enough about the language itself.

Kolmogorov complexity is absolutely the wrong metric. It doesn't account for big-O, timing, and many other production requirements.

You'll never have a perfect metric here, but human readable size of code base is well justified. Do you write minified javascript?

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#80

Post by morepedantic »

dawnofdusk wrote: post #17
morepedantic wrote: post #13

Earlier quoted context omitted.

IMO, uncompressed bytes is a better representation, because it can be used to compare relative expressive power for the particular problem. I'd bet Python cleans house here, but the write-only languages are a wild card.

Why would uncompressed bytes be better? Using a good compression algorithm better approximates the statistical entropy of the code which is at least correlated with e.g., Kolmogorov complexity.

Because humans read and write the uncompressed code. Gzip will hide problems like copy paste.
Post reply on HN