Live data from Hacker News

New Computer Language Benchmarks Game metric: time + source code size

benchmarksgame-team.pages.debian.net

21–30 of 59 posts

Re: New Computer Language Benchmarks Game metric: time + source code size

#21
I dont buy these results at all. Julia at second place looks like plain lie and complete nonsense, to the point I'm gonna look into this and run it myself.

After trying hard to use julia for about a year and I came to conclusion it's one of the slowest things around. Maybe the stuff changed? Maybe, but julia code still remains incorrect.

I hope they fix both things, speed (including start up speed, it counts A LOT) and correctness.

Re: New Computer Language Benchmarks Game metric: time + source code size

#22

This presentation is pretty bad, there should be more context, some kind of color scheme or labels instead of text in the background, spacing between the languages represented, other benchmarks than the geometric mean, etc.

> other benchmarks than the geometric mean The text is not clear enough, but "geometric mean" is not the benchmark. The 11 problems are listed in https://benchmarksgame-team.pages.debian.net/benchmarksgame/... The results of the 11 problems are combined using the "geometric mean" into a single number. Some people prefer the "geometric mean", other people prefer the "arithmetic mean" to combine the numbers, other peop…

>The text is not clear enough, but "geometric mean" is not the benchmark.

Thanks that makes more sense, that's another issue for context then. I don't have anything against geometric means but there should be basic statistics like average, max, min,... available as well.

Re: New Computer Language Benchmarks Game metric: time + source code size

#23

Earlier quoted context omitted.

> other benchmarks than the geometric mean The text is not clear enough, but "geometric mean" is not the benchmark. The 11 problems are listed in https://benchmarksgame-team.pages.debian.net/benchmarksgame/... The results of the 11 problems are combined using the "geometric mean" into a single number. Some people prefer the "geometric mean", other people prefer the "arithmetic mean" to combine the numbers, other peop…

>The text is not clear enough, but "geometric mean" is not the benchmark. Thanks that makes more sense, that's another issue for context then. I don't have anything against geometric means but there should be basic statistics like average, max, min,... available as well.

> … basic statistics like…

median, quartiles

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: New Computer Language Benchmarks Game metric: time + source code size

#24

Earlier quoted context omitted.

And when you want to make the code readable you try to space things out, split things in small functions, use longer and clear variables name. I guess they are asking for running the code trough a minifier so their implementation gains some points.

I can't find the documentation for it, but you can see here that they measure the size of the source file after gzip compression, which reduces advantage of code-golf solutions: https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...

"How source code size is measured"

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: New Computer Language Benchmarks Game metric: time + source code size

#25
post #9

Earlier quoted context omitted.

And when you want to make the code readable you try to space things out, split things in small functions, use longer and clear variables name. I guess they are asking for running the code trough a minifier so their implementation gains some points.

On other benchmarks they measure the size of source code after it's been run through compression, as a way to normalize that. Not sure if that's been done here, but it should be.

Yes, they're the same measurements.

Re: New Computer Language Benchmarks Game metric: time + source code size

#26
post #12

The thing they should change is to forbid the nonsense like: https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Actually if you look at all the top net core submissions the only one fast are the one using low level intrinsics etc ...

All of the languages now have that trash in them. I'd like a "naive" benchmarks game where you write the code straight forwardly in a normal style for the language.

"simple" (2nd link on the homepage.)

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: New Computer Language Benchmarks Game metric: time + source code size

#27
post #23

Earlier quoted context omitted.

>The text is not clear enough, but "geometric mean" is not the benchmark. Thanks that makes more sense, that's another issue for context then. I don't have anything against geometric means but there should be basic statistics like average, max, min,... available as well.

> … basic statistics like… median, quartiles https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Could you guide me to the ones I mentioned, I'm not seeing them.

Re: New Computer Language Benchmarks Game metric: time + source code size

#28
post #12

The thing they should change is to forbid the nonsense like: https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Actually if you look at all the top net core submissions the only one fast are the one using low level intrinsics etc ...

> Actually if you look at all the top net core submissions the only one fast are the one using low level intrinsics etc ...

Do you mean "fast" like a C program using low level intrinsics?

Re: New Computer Language Benchmarks Game metric: time + source code size

#29
post #23

Earlier quoted context omitted.

> … basic statistics like… median, quartiles https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Could you guide me to the ones I mentioned, I'm not seeing them.

The bar in the middle of the box is an "average" - the median.

https://www.merriam-webster.com/dictionary/average

https://www.itl.nist.gov/div898/handbook/eda/section3/boxplo...

Re: New Computer Language Benchmarks Game metric: time + source code size

#30

Geometric mean of (time + gzipped source code size in bytes) seems statistically wrong. What if you shifted time to nanoseconds ? Or source code size in terms of Megabytes. The rankings could change. The culprit is the '+' I would think Geometric mean of (time x gzipped source code size) is the correct way to compare languages together. It would not matter what the units of time or size are in that case. [Here the ge…

It's not necessarily wrong to add disparate units like this. It's implicitly weighting one unit to the other. Changing to nanoseconds just gives more weight to the time metric in the unified benchmark. You could instead explicitly weight them without changing units, if you cared about the size more you could add a multiplier to it.
Post reply on HN