Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking 20 programming languages on N-queens and matrix multiplication

github.com

Post reply on HN
Print view View original

151–160 of 194 posts

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#151

Post by igouy »

montebicyclelo wrote: post #140
igouy wrote: post #138

Earlier quoted context omitted.

"Nonetheless, because most benchmarks run for several seconds, including the startup time does not greatly affect the results." https://github.com/attractivechaos/plb2#startup-time

Yes, I saw that and still consider the methodology questionable. A fairer approach might be: time from the cli, including compilation, for compiled languages. (Or warmup the jit compiled code.)

Those are unfounded conjectures. When you show the results for those programs have been greatly affected …

As-in "Wtf kind of benchmark counts the jvm startup time?"

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#152

Post by igouy »

vkazanov wrote: post #55
lifthrasiir wrote: post #37

Earlier quoted context omitted.

I strongly suspect that the author may have confused the JIT warmup (hard to measure, as you need to ensure that the performance figure have reached the stable point) from the startup overhead (easy to measure).

...and many jitted vms do not even reach a stable point at all, there was a big paper on this a couple of years ago.

This?

https://tratt.net/laurie/blog/2018/why_arent_more_users_more...

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#153

Post by bjourne »

Having checked the C implementations of the algorithms, I'm a little skeptical because the implementations aren't optimized. Tiling matrix multiplication exploiting SIMD could easily improve performance 100-fold or more. At those speeds the cost of memory transfer usually dominate so the languages that give you the most fine-grained control over how data is laid out in memory tend to win. And it may not be the same languages that are now winning in your benchmark.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#154

Post by nerdponx »

bjourne wrote: post #153

Having checked the C implementations of the algorithms, I'm a little skeptical because the implementations aren't optimized. Tiling matrix multiplication exploiting SIMD could easily improve performance 100-fold or more. At those speeds the cost of memory transfer usually dominate so the languages that give you the most fine-grained control over how data is laid out in memory tend to win. And it may not be the same l…

I think a benchmark of "naive" implementations is interesting too, because it shows you how fast your code is usually going to run, not how fast it theoretically could run at its best.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#155

Post by bjourne »

smitty1110 wrote: post #109
borissk wrote: post #29

N-queens and matrix benchmark without Fortran and Wolfram Mathematica?

I was just popping in to make a similar comment. Fortran is probably doing the heavy lifting for most of the entries here, you might as well show how much your language-specific overhead is by including it.

No, it isn't. OP is benchmarking language-native implementations of all algos.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#156

Post by igouy »

mtreis86 wrote: post #117

A benchmark I would like to see is a comparison of languages in terms of how fast they are to beginners vs experts. I've been thinking about how to design it to get that result. What I think would work is taking something like these simple puzzles and have maybe a hundred people write up different solutions, so we can compare them using the programmer's level of expertise as one of the factors.

How would you decide who were beginners and who were experts?

Here are naive line-by-line transliterations from an original C program:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Here exhaustively-optimised + multicore + vector-instruction programs are included:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#157

Post by bjourne »

dsharlet wrote: post #32
p-e-w wrote: post #23

Earlier quoted context omitted.

> It demonstrates that Python needs libraries like NumPy. People use matrix multiplication libraries (often written in Assembly) from every language if they really care about performance. That's because such libraries incorporate 100 PhD theses' worth of tricks that no individual can hope to reinvent in the course of solving another problem. There is absolutely nothing special about Python in this context. > It also…

This is really overstating how hard it is to compete with matrix multiply libraries. The main reason those libraries are so big and have had so much work invested in them is their generality: they're reasonably fast for almost any kind of inputs. If you have a specific problem with constraints you can exploit (e.g. known fixed dimensions, sparsity patterns, data layouts, type conversions, etc.), it's not hard at all…

> This is really overstating how hard it is to compete with matrix multiply libraries.

I'll file this under "talk is cheap". :) I tried it last year and got within 50% of BLAS. Getting above that is tons of work. Which you have to repeat for every processor model, NUMA, and every combination of matrix type (long thin, short wide, etc).

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#159

Post by bjourne »

mratsim wrote: post #70
p-e-w wrote: post #23

Earlier quoted context omitted.

> It demonstrates that Python needs libraries like NumPy. People use matrix multiplication libraries (often written in Assembly) from every language if they really care about performance. That's because such libraries incorporate 100 PhD theses' worth of tricks that no individual can hope to reinvent in the course of solving another problem. There is absolutely nothing special about Python in this context. > It also…

> People use matrix multiplication libraries (often written in Assembly) from every language if they really care about performance. That's because such libraries incorporate 100 PhD theses' worth of tricks that no individual can hope to reinvent in the course of solving another problem. There is absolutely nothing special about Python in this context. You don't have to use Assembly. Case in point, this is as fast as…

That's a bold claim. Do you have any benchmarks to back it up? Even if it was as fast as OpenBLAS on your machine when you benchmarked it that doesn't mean it will be as fast on others' machines.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#160

Post by attractivechaos »

neonsunset wrote: post #50

C# code should be using an official package for GEMM which is System.Numerics.Tensors, it is pure C# but runs at max hw efficiency (it is also idiomatic). Or at least use Vector instead of scalar operations. I’d expect this applies to most other popular languages here too.

System.Numerics.Tensors is disqualified because it uses a different algorithm. If you have a faster matmul implementation in Vector, a PR will be much appreciated. Thank you in advance.
Post reply on HN