Intel Fortran now 20% faster than C, fastest language on Shootout • Hacker News Board

Re: Intel Fortran now 20% faster than C, fastest language on Shootout

#51

Post by igouy » Fri, Jan 13, 2012, 11:49 PM UTC

chalst wrote: post #50

igouy wrote: post #48

Earlier quoted context omitted.

You don't seem to have even looked at the web page. For each programming language implementation, that web page shows 7 descriptive values - not just the median! As I've already said - The outliers are not discarded. The outliers are shown both in the chart and in the table.

"7 descriptive values" - Indeed that is true. But the median is used for the ranking and that is why I criticise it. The worst value for Fortran, one of the included values, is more than five times worse than that for C. This suggests (not proves) that the Intel Fortran compiler has worse performance holes than gcc C that show up for some, but not all, of these benchmarks.

> that is why I criticise it

You criticise but don't even suggest an alternative, let alone evaluate how well or badly that alternative compares to the median.

That's empty criticism.

Re: Intel Fortran now 20% faster than C, fastest language on Shootout

#52

Post by chalst » Sat, Jan 14, 2012, 11:44 AM UTC

igouy wrote: post #51

chalst wrote: post #50

Earlier quoted context omitted.

"7 descriptive values" - Indeed that is true. But the median is used for the ranking and that is why I criticise it. The worst value for Fortran, one of the included values, is more than five times worse than that for C. This suggests (not proves) that the Intel Fortran compiler has worse performance holes than gcc C that show up for some, but not all, of these benchmarks.

> that is why I criticise it You criticise but don't even suggest an alternative, let alone evaluate how well or badly that alternative compares to the median. That's empty criticism.

What is bad about median is that a compiler that performs well on most benchmarks but then falls off a cliff for many other, but less than half, performs well altogether.

I think the distance-weighted estimator would serve the purpose of the ranking better.

http://en.wikipedia.org/wiki/Distance-weighted_estimator

This measure finds a measure of the centre of the distribution that gives less weight to items further away from that centre without discarding them. In this way, the measure is not dominated by outliers, but they still contribute to the final result.

I don't say that this is the best measure, only one that is better. A factor analysis might get at the extent to which the different benchmarks are doing the same thing. Some attempt to find measures of how these activities are represented in larger pieces of code might suggest weights for the benchmarks (there's a literature on analysing loads that might be relevant here). But this would make their analysis more complex and more prone to bias leaking into their analysis, so I don't say they ought to do this; my point is that median is simply a bad population measure for the purposes of their survey.

You have argued against me without trying to defend the choice of median as the measure used for the ranking. Is that denialism?

Re: Intel Fortran now 20% faster than C, fastest language on Shootout

#53

Post by igouy » Sat, Jan 14, 2012, 5:06 PM UTC

chalst wrote: post #52

igouy wrote: post #51

Earlier quoted context omitted.

> that is why I criticise it You criticise but don't even suggest an alternative, let alone evaluate how well or badly that alternative compares to the median. That's empty criticism.

What is bad about median is that a compiler that performs well on most benchmarks but then falls off a cliff for many other, but less than half, performs well altogether. I think the distance-weighted estimator would serve the purpose of the ranking better. http://en.wikipedia.org/wiki/Distance-weighted_estimator This measure finds a measure of the centre of the distribution that gives less weight to items further aw…

> Is that denialism?

It isn't any kind of ism.

> What is bad about median is that...

Again, that's what's bad about the median as the sole characterisation of the measurements - but the median is not presented as the sole characterisation of the measurements.

The outlier Fortran measurements are there for all to see.

Some variation on the distance-weighted estimator might work here -

http://shootout.alioth.debian.org/u64q/which-language-is-bes...