New Computer Language Benchmarks Game metric: time + source code size • Hacker News Board

Re: New Computer Language Benchmarks Game metric: time + source code size

#11

Post by gus_massa » Wed, Jun 01, 2022, 6:24 PM UTC

NeutralForest wrote: post #7

This presentation is pretty bad, there should be more context, some kind of color scheme or labels instead of text in the background, spacing between the languages represented, other benchmarks than the geometric mean, etc.

> other benchmarks than the geometric mean

The text is not clear enough, but "geometric mean" is not the benchmark. The 11 problems are listed in https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

The results of the 11 problems are combined using the "geometric mean" into a single number. Some people prefer the "geometric mean", other people prefer the "arithmetic mean" to combine the numbers, other people prefer the maximum, and there rare many other methods (like the average excluding both borders).

Re: New Computer Language Benchmarks Game metric: time + source code size

#12

Post by Thaxll » Wed, Jun 01, 2022, 6:31 PM UTC

The thing they should change is to forbid the nonsense like:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Actually if you look at all the top net core submissions the only one fast are the one using low level intrinsics etc ...

Re: New Computer Language Benchmarks Game metric: time + source code size

#13

Post by IshKebab » Wed, Jun 01, 2022, 6:34 PM UTC

With a totally arbitrary conversion of 1 second = 1 gzipped byte.

This is basically meaningless. I don't see why you'd even need to do this. You can easily show code size and performance on the same graph.

Re: New Computer Language Benchmarks Game metric: time + source code size

#14

Post by benstrumental » Wed, Jun 01, 2022, 6:39 PM UTC

kibwen wrote: post #4

For comparing multiple implementations of a single benchmark in a single language, this sort of data would be interesting as a 2D plot, to see how many lines it takes to improve performance by how much. But for cross-language benchmarking this seems somewhat confounding, as the richness of standard libraries varies between languages (and counting the lines of external dependencies sounds extremely annoying, not only…

Not exactly what you're looking for, but here are some 2D plots of code size vs. execution time with geometric means of fastest entries and smallest code size entries of each language:

https://twitter.com/ChapelLanguage/status/152442889069266944...

Re: New Computer Language Benchmarks Game metric: time + source code size

#15

Post by agentgt » Wed, Jun 01, 2022, 6:41 PM UTC

I really wish they aggregated the metric of build time (+ whatever).

That is a huge metric I care about.

You can figure out it somewhat by clicking on each language benchmark but it is not aggregated.

BTW as biased guy in the Java world I can tell you this is one area Java is actually mostly the winner even beating out many scripting languages apparently.

Re: New Computer Language Benchmarks Game metric: time + source code size

#16

Post by benstrumental » Wed, Jun 01, 2022, 6:47 PM UTC

simion314 wrote: post #6

kibwen wrote: post #4

For comparing multiple implementations of a single benchmark in a single language, this sort of data would be interesting as a 2D plot, to see how many lines it takes to improve performance by how much. But for cross-language benchmarking this seems somewhat confounding, as the richness of standard libraries varies between languages (and counting the lines of external dependencies sounds extremely annoying, not only…

And when you want to make the code readable you try to space things out, split things in small functions, use longer and clear variables name. I guess they are asking for running the code trough a minifier so their implementation gains some points.

I can't find the documentation for it, but you can see here that they measure the size of the source file after gzip compression, which reduces advantage of code-golf solutions:

https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...

Re: New Computer Language Benchmarks Game metric: time + source code size

#17

Post by spullara » Wed, Jun 01, 2022, 7:10 PM UTC

Thaxll wrote: post #12

The thing they should change is to forbid the nonsense like: https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Actually if you look at all the top net core submissions the only one fast are the one using low level intrinsics etc ...

All of the languages now have that trash in them. I'd like a "naive" benchmarks game where you write the code straight forwardly in a normal style for the language.

Re: New Computer Language Benchmarks Game metric: time + source code size

#18

Post by cpurdy » Wed, Jun 01, 2022, 7:25 PM UTC

Predictably, "The Computer Language Benchmarks Game" once again proves the worthlessness of "The Computer Language Benchmarks Game".

This thing has been a long running joke in the software industry, exceeded only by the level of their defensiveness.

SMH.

Re: New Computer Language Benchmarks Game metric: time + source code size

#19

Post by igouy » Wed, Jun 01, 2022, 7:40 PM UTC

sidkshatriya wrote: post #5

Geometric mean of (time + gzipped source code size in bytes) seems statistically wrong. What if you shifted time to nanoseconds ? Or source code size in terms of Megabytes. The rankings could change. The culprit is the '+' I would think Geometric mean of (time x gzipped source code size) is the correct way to compare languages together. It would not matter what the units of time or size are in that case. [Here the ge…

> The culprit is the '+'

That annotation does seem to have caused much frothing and gnashing.

Here's how the calculation is made — "How not to lie with statistics: The correct way to summarize benchmark results."

[pdf] http://www.cse.unsw.edu.au/~cs9242/11/papers/Fleming_Wallace...

Re: New Computer Language Benchmarks Game metric: time + source code size

#20

Post by igouy » Wed, Jun 01, 2022, 7:43 PM UTC

_b wrote: post #3

I'd be interested to see "C compiled with Clang" added as another language to the benchmark games. In part, digging into Clang vs gcc benchmarks is always interesting, and in part, as Rust & Clang share the same LLVM backend, it would shed light on how much of the C vs Rust difference is from frontend language stuff vs backend code gen stuff.

Already done:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...