Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking 20 programming languages on N-queens and matrix multiplication

github.com

Post reply on HN
Print view View original

41–50 of 194 posts

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#41

Post by p-e-w »

tsimionescu wrote: post #35
p-e-w wrote: post #5

What is this supposed to demonstrate? There appears to be roughly the same code structure, ported to every language, while for some languages, arbitrary optimizations are introduced (such as using `array` instead of `list` in Python). But nobody working in Python uses matrix multiplication code written in Python. They use NumPy, which is a de facto standard library for people working in the relevant fields. It's as m…

Generally, the point of language benchmarks is to show how the languages compare at solving the same problem, without external libraries. Including external libraries is pointless since any language can call any library, ultimately, so at best you'd be comparing the FFI overhead. So this shouldn't be taken as "how fast does a real-world Python program do at matrix multiplication", since of course no one writes real-w…

> But it can show the relative speed of pure Python at purely computational tasks.

But that's irrelevant if nobody uses "pure Python" for computational tasks.

It's like asking "how well do these languages run on a Lisp machine from 1979?". It simply has no relevance to real-world considerations today.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#42

Post by lifthrasiir »

tsimionescu wrote: post #35
p-e-w wrote: post #5

What is this supposed to demonstrate? There appears to be roughly the same code structure, ported to every language, while for some languages, arbitrary optimizations are introduced (such as using `array` instead of `list` in Python). But nobody working in Python uses matrix multiplication code written in Python. They use NumPy, which is a de facto standard library for people working in the relevant fields. It's as m…

Generally, the point of language benchmarks is to show how the languages compare at solving the same problem, without external libraries. Including external libraries is pointless since any language can call any library, ultimately, so at best you'd be comparing the FFI overhead. So this shouldn't be taken as "how fast does a real-world Python program do at matrix multiplication", since of course no one writes real-w…

> Generally, the point of language benchmarks is to show how the languages compare at solving the same problem, without external libraries.

If this were the only concern, it should be valid to create a blob of binary and call that function for the optimal performance. (Python's ctypes makes this very easy, for example.) So you want an idiomatic solution instead, and Numpy for matrix computation is considered idiomatic in Python, even more than the pure Python code.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#43

Post by keyle »

I love a good benchmark, thanks for putting this together! However, I have a bit of feedback.

First, the graph is misleading, stacking times with languages that have half the implementation, they appear faster, until you dig in. I'd suggest producing an alternate graph that shows only the implemented puzzles in every language, or make a unique graph for every language:puzzle.

Second, the examples are taken from rosetta code and are not necessarily what would be the best implementation, or even close to the best implementation, for benchmarking purposes.

Finally, those examples should be reproduced across various hardware platforms, I'm on arm64 Darwin myself, but you might find different results on Intel platforms due to the various compiler optimizations available based on the hardware.

More benchmarks would be interesting to see, such as actual real world operations, e.g. opening a file, reading it, parsing json, opening a socket server, etc.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#44

Post by YetAnotherNick »

p-e-w wrote: post #41
tsimionescu wrote: post #35

Earlier quoted context omitted.

Generally, the point of language benchmarks is to show how the languages compare at solving the same problem, without external libraries. Including external libraries is pointless since any language can call any library, ultimately, so at best you'd be comparing the FFI overhead. So this shouldn't be taken as "how fast does a real-world Python program do at matrix multiplication", since of course no one writes real-w…

> But it can show the relative speed of pure Python at purely computational tasks. But that's irrelevant if nobody uses "pure Python" for computational tasks. It's like asking "how well do these languages run on a Lisp machine from 1979?". It simply has no relevance to real-world considerations today.

Everyone uses pure python for purely computational tasks. numpy or pytorch has far too few operation to even count as all the computational task. e.g. most of the operations of pandas is written in pure python, and at times I found using specialised libraries could give 10x improvement but with blow to developer experience compared to python.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#45

Post by sroussey »

alberth wrote: post #10

> ” Timing on Apple M1 Macbook Pro” Given that its become increasingly more common for CPUs to have both Performance & Efficency cores … how do benchmarks ensure they are only being run on the P-cores?

It actually takes a bit of effort to run on the efficiency cores.

I believe Game Mode will push the processes to e cores to keep a consistent game play without thermal throttling.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#46

Post by sa46 »

daxfohl wrote: post #30

[flagged]

Reliably benchmarking optimizing VMs like Java and JavaScript is a research problem.

See “Virtual machine warmup blows hot and cold”. The authors run a variety of benchmarks (with a pristine methodology) and find 43% of them provide “bad inconsistent” results.

Paper summary: https://blog.acolyer.org/2017/11/07/virtual-machine-warmup-b...

Paper: https://dl.acm.org/doi/pdf/10.1145/3133876

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#47

Post by SideQuark »

p-e-w wrote: post #38
dsharlet wrote: post #32

Earlier quoted context omitted.

This is really overstating how hard it is to compete with matrix multiply libraries. The main reason those libraries are so big and have had so much work invested in them is their generality: they're reasonably fast for almost any kind of inputs. If you have a specific problem with constraints you can exploit (e.g. known fixed dimensions, sparsity patterns, data layouts, type conversions, etc.), it's not hard at all…

> If you are using python, you have no chance. Of course you do. Every special-case multiplication algorithm you might need already has an optimized implementation that you can just `pip install`, and move on with what you're actually working on. The whole scientific computing world runs on Python. Straightforward numerics code using NumPy tends to murder C/C++ code in regard to performance, unless that code is writt…

> The whole scientific computing world runs on Python

If you ignore the majority of scientific code running on supercomputers doing most of science in C++ and Fortran.

Even in areas where python is used, the majority of the compute runs on C/C++/Fortran, with a little python as glue.

If you think numpy (written in c/c++) murders c/c++ code, you should learn about HPC, where really high performance happens. They don't use numpy.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#49

Post by geysersam »

dsharlet wrote: post #32
p-e-w wrote: post #23

Earlier quoted context omitted.

> It demonstrates that Python needs libraries like NumPy. People use matrix multiplication libraries (often written in Assembly) from every language if they really care about performance. That's because such libraries incorporate 100 PhD theses' worth of tricks that no individual can hope to reinvent in the course of solving another problem. There is absolutely nothing special about Python in this context. > It also…

This is really overstating how hard it is to compete with matrix multiply libraries. The main reason those libraries are so big and have had so much work invested in them is their generality: they're reasonably fast for almost any kind of inputs. If you have a specific problem with constraints you can exploit (e.g. known fixed dimensions, sparsity patterns, data layouts, type conversions, etc.), it's not hard at all…

If you need a really fast compiled inner loop then you can often implement it in Python using Numba. Using that you can easily implement something like sparse matrix multiplication in Python.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#50

Post by neonsunset »

C# code should be using an official package for GEMM which is System.Numerics.Tensors, it is pure C# but runs at max hw efficiency (it is also idiomatic). Or at least use Vector instead of scalar operations. I’d expect this applies to most other popular languages here too.
Post reply on HN