Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking 20 programming languages on N-queens and matrix multiplication

github.com

Post reply on HN
Print view View original

61–70 of 194 posts

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#61

Post by hyperman1 »

I wondered why rust is so far behind C/nim/zig, they should have similar behaviour.

The difference is mostly in matmul.

I see how C, for an n x n matrix, does 2 allocations, while rust does n+1. C's matrix rows are right next to each other, rusts are probably all over the place. Didn't look at nim or zig.

Maybe a slice of slice of double would perform better than a vec of vec of double? Then again, an argument can be made that rust pushes you to vec, so this impl is more honest for how a beginner would do it. Otoh, C has the optimization so why doesn't rust?

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#62

Post by brabel »

Atotalnoob wrote: post #14

Aren’t JIT languages at a disadvantage since they are benchmarked through the CLI rather than using a benchmarking library to allow JIT to warmup?

For one-time problem runs, the JIT languages in practice will not be given time to warm up. All that matters for a user is how fast the application is in practice. It's not about "making it fair" for languages, it's about measuring how fast they go from nothing to results.

It doesn't make sense to allow "warmup" time for them unless your expected application is a server which for most of the time will be running "warm" (and even then with scalable containers that assumption may not even be true in some cases). For servers, however, what matters is mostly how fast its HTTP library is and how good the async IO is... check Techempower benchmarks for that: https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s...

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#63

Post by nrabulinski »

hyperman1 wrote: post #61

I wondered why rust is so far behind C/nim/zig, they should have similar behaviour. The difference is mostly in matmul. I see how C, for an n x n matrix, does 2 allocations, while rust does n+1. C's matrix rows are right next to each other, rusts are probably all over the place. Didn't look at nim or zig. Maybe a slice of slice of double would perform better than a vec of vec of double? Then again, an argument can be…

The rust code is very unidiomatic, not only because of the Vec of Vecs which I’d say, even if it’s the obvious naive approach, no one experienced wouldn’t choose over a flat slice, the implementation itself is very naive and unidiomatic.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#64

Post by geek_at »

theteapot wrote: post #4

PHP results: I was stupid enough to write some scientific code in PHP once so know how slow it can be - mostly around array access and manipulation. But if your going to try, use the HHVM interpreter. It's much faster and is a drop in replacement for the PHP interpreter. Hack ( https://hacklang.org/ ) uses that under the hood by default.

Also since PHP was built for web requests it builds The cache on the First run. would be interesting to see All languages of the benchmark with a second run

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#65

Post by neonsunset »

borissk wrote: post #27
metaltyphoon wrote: post #20

Why not run C# as AOT? It’s a one line change.

Do people usually run C# code AOT compiled?

It’s gaining popularity for CLI and serverless, some people use it to replace C++ for writing native libraries and plugins as well.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#66

Post by xnorswap »

metaltyphoon wrote: post #20

Why not run C# as AOT? It’s a one line change.

Or use BenchmarkDotNet which, among other things to get an accurate benchmark, does JIT warmup outside of measurement.

( https://github.com/dotnet/BenchmarkDotNet ).

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#67

Post by tiehuis »

hyperman1 wrote: post #61

I wondered why rust is so far behind C/nim/zig, they should have similar behaviour. The difference is mostly in matmul. I see how C, for an n x n matrix, does 2 allocations, while rust does n+1. C's matrix rows are right next to each other, rusts are probably all over the place. Didn't look at nim or zig. Maybe a slice of slice of double would perform better than a vec of vec of double? Then again, an argument can be…

Zig had a similar issue which I made an MR to fix. I didn't actually notice Rust had the same issue but that's probably just my forgotten knowledge with how the vec! macros expand when nested.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#68

Post by d0mine »

latenightcoding wrote: post #21

These are not the best benchmarks, but Python is indeed as slow as Perl, which I find insane considering that Python has 100-1000 more people working on the interpreter and performance has been a big emphasis the last few years.

The benchmark is not representative of how Python is actually used in practice. It ignores the existence of libraries (explicitly). For example, If numpy, pytorch were used for matmul, the results would be completely different.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#69

Post by mratsim »

YetAnotherNick wrote: post #44
p-e-w wrote: post #41

Earlier quoted context omitted.

> But it can show the relative speed of pure Python at purely computational tasks. But that's irrelevant if nobody uses "pure Python" for computational tasks. It's like asking "how well do these languages run on a Lisp machine from 1979?". It simply has no relevance to real-world considerations today.

Everyone uses pure python for purely computational tasks. numpy or pytorch has far too few operation to even count as all the computational task. e.g. most of the operations of pandas is written in pure python, and at times I found using specialised libraries could give 10x improvement but with blow to developer experience compared to python.

Pandas/Numpy uses a lot of C and Scipy a lot of Fortran

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#70

Post by mratsim »

p-e-w wrote: post #23
morepedantic wrote: post #11

Earlier quoted context omitted.

It demonstrates that Python needs libraries like NumPy. Few problems are more heavily optimized than matrix multiplication in practice, so comparing matrix multiplication benchmarks across languages with NumPy is not representative of real-world performance for most programming use cases. It also means that adding performance to an existing Python program requires dropping into a different language, which is not only…

> It demonstrates that Python needs libraries like NumPy. People use matrix multiplication libraries (often written in Assembly) from every language if they really care about performance. That's because such libraries incorporate 100 PhD theses' worth of tricks that no individual can hope to reinvent in the course of solving another problem. There is absolutely nothing special about Python in this context. > It also…

> People use matrix multiplication libraries (often written in Assembly) from every language if they really care about performance. That's because such libraries incorporate 100 PhD theses' worth of tricks that no individual can hope to reinvent in the course of solving another problem. There is absolutely nothing special about Python in this context.

You don't have to use Assembly.

Case in point, this is as fast as OpenBLAS: https://github.com/mratsim/weave/tree/master/benchmarks/matm...

Post reply on HN