Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking 20 programming languages on N-queens and matrix multiplication

github.com

Post reply on HN
Print view View original

131–140 of 194 posts

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#131

Post by airstrike »

tmtvl wrote: post #128
airstrike wrote: post #121

Very interesting. My two cents: because the stacked bars for php, ruby, perl and py:cpy, it's impossible to compare the other languages in that first chart. All the chart says is that the benchmark is much slower in those 4 languages relative to "all other languages we tested". It would be nice to see those other languages in a chart that doesn't include the slower four. Alternatively, you could also show those slowe…

The second chart is basically the first chart with the slow 4 removed (well, not removed but at such a scale that they're irrelevant).

Ah, duh, I see that now. That's what I get for commenting too quickly ;-)

The choice of stacked bar charts is also strange because the languages are not all comparable

> Every language has nqueen and matmul implementations. Some languages do not have sudoku or bedcov implementations. In addition, I implemented most algorithms in plb2 and adapted a few contributed matmul and sudoku implementations in plb. As I am mostly a C programmer, implementations in other languages may be suboptimal and there are no implementations in functional languages. Pull requests are welcomed!

So the point still stands that many charts doing one thing each would be better than fewer charts doing many things each

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#132

Post by YetAnotherNick »

mratsim wrote: post #69
YetAnotherNick wrote: post #44

Earlier quoted context omitted.

Everyone uses pure python for purely computational tasks. numpy or pytorch has far too few operation to even count as all the computational task. e.g. most of the operations of pandas is written in pure python, and at times I found using specialised libraries could give 10x improvement but with blow to developer experience compared to python.

Pandas/Numpy uses a lot of C and Scipy a lot of Fortran

No one said they don't. Just there still are and always will be parts which are computationally heavy but written in python. e.g. pandas map is 300x slower than numpy map[1] for the same operation. But not everything could be vectorized and even if some complex function could be vectorized it will involve multiple passes over memory for the same array.

[1]: https://www.exxactcorp.com/blog/Deep-Learning/how-to-speed-u...

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#133

Post by jltsiren »

nithril wrote: post #123
brabel wrote: post #62

Earlier quoted context omitted.

For one-time problem runs, the JIT languages in practice will not be given time to warm up. All that matters for a user is how fast the application is in practice. It's not about "making it fair" for languages, it's about measuring how fast they go from nothing to results. It doesn't make sense to allow "warmup" time for them unless your expected application is a server which for most of the time will be running "war…

From nothing to result rarely happens in real live. I hardly see someone to start/stop a program per unit task (like piping commands).

Code compilation is a common example. In many languages, when you do a clean build of a large project, you run a large number of short-lived processes.

With command line tools, the dominant design philosophy is that a program completes one task and exits. The tasks may be large or small, and the same tools are often expected to handle both.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#134

Post by igouy »

airstrike wrote: post #131
tmtvl wrote: post #128

Earlier quoted context omitted.

The second chart is basically the first chart with the slow 4 removed (well, not removed but at such a scale that they're irrelevant).

Ah, duh, I see that now. That's what I get for commenting too quickly ;-) The choice of stacked bar charts is also strange because the languages are not all comparable > Every language has nqueen and matmul implementations. Some languages do not have sudoku or bedcov implementations. In addition, I implemented most algorithms in plb2 and adapted a few contributed matmul and sudoku implementations in plb. As I am most…

> many charts doing one thing each

Would not be a summary.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#135

Post by EmberTwin »

theteapot wrote: post #4

PHP results: I was stupid enough to write some scientific code in PHP once so know how slow it can be - mostly around array access and manipulation. But if your going to try, use the HHVM interpreter. It's much faster and is a drop in replacement for the PHP interpreter. Hack ( https://hacklang.org/ ) uses that under the hood by default.

HHVM is not a drop-in replacement for a PHP interpreter. The semantics of Hack and PHP have diverged, typically in the direction of eliminating dynamic behavior from Hack that existed in PHP (examples: string -> function coercion, the PHP dual vector-hash-table array type, and non-throwing out-of-bounds array accesses are all gone from Hack). The semantics changes both simplify static analysis and make it easier to JIT fast code.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#136

Post by airstrike »

igouy wrote: post #134
airstrike wrote: post #131

Earlier quoted context omitted.

Ah, duh, I see that now. That's what I get for commenting too quickly ;-) The choice of stacked bar charts is also strange because the languages are not all comparable > Every language has nqueen and matmul implementations. Some languages do not have sudoku or bedcov implementations. In addition, I implemented most algorithms in plb2 and adapted a few contributed matmul and sudoku implementations in plb. As I am most…

> many charts doing one thing each Would not be a summary.

It would if you have some summary charts at the top that don't include all of the data, but just the key takeaways, with other charts providing more detail for discussion elsewhere

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#138

Post by igouy »

montebicyclelo wrote: post #127

The reality is that it would be very hard to find Python code that does not use NumPy (or some tensor lib), for matmul. Including time to JIT compile is questionable, why not also include time to compile the compiled languages?

"Nonetheless, because most benchmarks run for several seconds, including the startup time does not greatly affect the results."

https://github.com/attractivechaos/plb2#startup-time

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#139

Post by igouy »

wslh wrote: post #114

Nowadays I would add lines of code and syntax complexity to the benchmark. It is apples and oranges comparison but you could prefer clarity than performance depending on the circumstances. Also, it is relatively easy to do with today language tools.

For example:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

"How source code size is measured"

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#140

Post by montebicyclelo »

igouy wrote: post #138
montebicyclelo wrote: post #127

The reality is that it would be very hard to find Python code that does not use NumPy (or some tensor lib), for matmul. Including time to JIT compile is questionable, why not also include time to compile the compiled languages?

"Nonetheless, because most benchmarks run for several seconds, including the startup time does not greatly affect the results." https://github.com/attractivechaos/plb2#startup-time

Yes, I saw that and still consider the methodology questionable. A fairer approach might be: time from the cli, including compilation, for compiled languages. (Or warmup the jit compiled code.)
Post reply on HN