Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking 20 programming languages on N-queens and matrix multiplication

github.com

Post reply on HN
Print view View original

31–40 of 194 posts

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#31

Post by lifthrasiir »

jakobnissen wrote: post #28
lifthrasiir wrote: post #16

Earlier quoted context omitted.

You don't need any C knowledge to use numpy. In fact, its conceptual similarity with Matlab is possibly the single most important reason for its popularity. Many other problems do need specialized treatments that would indeed require other languages, but numpy is not a good counterexample.

You're missing the point. The point is that, for any application, Python needs an underlying C library to be fast. So if you need to solve problems where no such library exists, Python is slow. In other words, Python IS slow, but it can call fast code written in other languages.

That's true but irrelevant here, especially given the original claim:

> adding performance to an existing Python program requires dropping into a different language

...is demonstrably false for a significant class of programs that can be rewritten into the array paradigm. The benchmark should have picked other numerical problem to avoid this issue. The Computer Language Benchmarks Game, for example, uses the `n-body` problem for this purpose.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#32

Post by dsharlet »

p-e-w wrote: post #23
morepedantic wrote: post #11

Earlier quoted context omitted.

It demonstrates that Python needs libraries like NumPy. Few problems are more heavily optimized than matrix multiplication in practice, so comparing matrix multiplication benchmarks across languages with NumPy is not representative of real-world performance for most programming use cases. It also means that adding performance to an existing Python program requires dropping into a different language, which is not only…

> It demonstrates that Python needs libraries like NumPy. People use matrix multiplication libraries (often written in Assembly) from every language if they really care about performance. That's because such libraries incorporate 100 PhD theses' worth of tricks that no individual can hope to reinvent in the course of solving another problem. There is absolutely nothing special about Python in this context. > It also…

This is really overstating how hard it is to compete with matrix multiply libraries. The main reason those libraries are so big and have had so much work invested in them is their generality: they're reasonably fast for almost any kind of inputs.

If you have a specific problem with constraints you can exploit (e.g. known fixed dimensions, sparsity patterns, data layouts, type conversions, etc.), it's not hard at all to beat MKL, etc... if you are using a language like C++. If you are using python, you have no chance.

It isn't even necessarily that different from a few nested loops. Clang is pretty damn good at autovectorizing, you just have to be a little careful about how you write the code.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#33

Post by jakobnissen »

Atotalnoob wrote: post #14

Aren’t JIT languages at a disadvantage since they are benchmarked through the CLI rather than using a benchmarking library to allow JIT to warmup?

Yes, but the author claims the longest JIT warmup is 0.3 seconds, so it's not an important issue in these benchmarks that take several seconds.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#34

Post by lifthrasiir »

dsharlet wrote: post #32
p-e-w wrote: post #23

Earlier quoted context omitted.

> It demonstrates that Python needs libraries like NumPy. People use matrix multiplication libraries (often written in Assembly) from every language if they really care about performance. That's because such libraries incorporate 100 PhD theses' worth of tricks that no individual can hope to reinvent in the course of solving another problem. There is absolutely nothing special about Python in this context. > It also…

This is really overstating how hard it is to compete with matrix multiply libraries. The main reason those libraries are so big and have had so much work invested in them is their generality: they're reasonably fast for almost any kind of inputs. If you have a specific problem with constraints you can exploit (e.g. known fixed dimensions, sparsity patterns, data layouts, type conversions, etc.), it's not hard at all…

You need to have enough experience to be able to be a little careful though. This is generally true for most languages, loop unrolling works even better in Python for example, but many Python programmers aren't even aware of this possibility.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#35

Post by tsimionescu »

p-e-w wrote: post #5

What is this supposed to demonstrate? There appears to be roughly the same code structure, ported to every language, while for some languages, arbitrary optimizations are introduced (such as using `array` instead of `list` in Python). But nobody working in Python uses matrix multiplication code written in Python. They use NumPy, which is a de facto standard library for people working in the relevant fields. It's as m…

Generally, the point of language benchmarks is to show how the languages compare at solving the same problem, without external libraries. Including external libraries is pointless since any language can call any library, ultimately, so at best you'd be comparing the FFI overhead.

So this shouldn't be taken as "how fast does a real-world Python program do at matrix multiplication", since of course no one writes real-world programs doing matrix multiplication in pure Python. But it can show the relative speed of pure Python at purely computational tasks.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#36

Post by brrrrrm »

habibur wrote: post #9
theteapot wrote: post #4

PHP results: I was stupid enough to write some scientific code in PHP once so know how slow it can be - mostly around array access and manipulation. But if your going to try, use the HHVM interpreter. It's much faster and is a drop in replacement for the PHP interpreter. Hack ( https://hacklang.org/ ) uses that under the hood by default.

I put the pure math codes in C extension of PHP. Building C extension for PHP is easier than most of the other high level languages. And then things get blazingly fast.

doesn't look that easy... zend_parse_parameters? pre-baked configure + make scripts?

check out how a modern language deals with this stuff https://bun.sh/docs/api/ffi#usage

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#37

Post by lifthrasiir »

jakobnissen wrote: post #33
Atotalnoob wrote: post #14

Aren’t JIT languages at a disadvantage since they are benchmarked through the CLI rather than using a benchmarking library to allow JIT to warmup?

Yes, but the author claims the longest JIT warmup is 0.3 seconds, so it's not an important issue in these benchmarks that take several seconds.

I strongly suspect that the author may have confused the JIT warmup (hard to measure, as you need to ensure that the performance figure have reached the stable point) from the startup overhead (easy to measure).

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#38

Post by p-e-w »

dsharlet wrote: post #32
p-e-w wrote: post #23

Earlier quoted context omitted.

> It demonstrates that Python needs libraries like NumPy. People use matrix multiplication libraries (often written in Assembly) from every language if they really care about performance. That's because such libraries incorporate 100 PhD theses' worth of tricks that no individual can hope to reinvent in the course of solving another problem. There is absolutely nothing special about Python in this context. > It also…

This is really overstating how hard it is to compete with matrix multiply libraries. The main reason those libraries are so big and have had so much work invested in them is their generality: they're reasonably fast for almost any kind of inputs. If you have a specific problem with constraints you can exploit (e.g. known fixed dimensions, sparsity patterns, data layouts, type conversions, etc.), it's not hard at all…

> If you are using python, you have no chance.

Of course you do. Every special-case multiplication algorithm you might need already has an optimized implementation that you can just `pip install`, and move on with what you're actually working on.

The whole scientific computing world runs on Python. Straightforward numerics code using NumPy tends to murder C/C++ code in regard to performance, unless that code is written by people who make a living hand-optimizing computational routines.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#40

Post by brrrrrm »

daxfohl wrote: post #15

Odd that nqueens and sudoku have a high correlation but matmul seems to be largely doing its own thing. nqueen vs. sudoku: 0.531 matmul vs. sudoku: 0.362 matmul vs. nqueen: 0.127

matmul is old and useful -- there's a lot of hardware on a chip that makes it run much faster (prefetch, vectorization, instruction parallelism) and some of these languages have optimizations to expose those things automatically.
Post reply on HN