Calculating the mean of a list of numbers (2016)

An interesting article describing a useful framework.

In addition to the largest numbers that floating point can handle, a related issue can come up with numbers that are not near the edge.

A good student of floating point numbers will notice that there is a good chance you will get a different result summing a list of floating point numbers if they are sorted in a different order. This is due to the rounding error that can happen if you add a large number to a smaller one. Thus, if you add up all the small numbers first, you can get an intermediate sum that is large enough to not overflow when added to the next on the list. Thus, you should sort an array of floating point numbers and begin the summation with the smallest.

Some careful thought is needed if numbers are important to your computation. While the author of this article makes important points about one aspect of this, they seem to shrug off further study of the referenced 30 page paper at https://hal.archives-ouvertes.fr/file/index/docid/576641/fil....

The authors of that paper, however, seems to misstate one vital point. They note that the mathematical formulas are "unstable", but to the point of working with floating point numbers, it isn't the mathematics that is unstable, it is the failure to understand how deeply modern floating point numbers are an approximation. Methods that fail to take this into account will be unstable.

This article highlights and issue with floating point numbers, a substantial use case for data scientists (and as such, I value the input).

How do REPLs and databases handle this edge case?

If you're reaching these values then it's extremely likely that either:

(a) You're doing something wrong (usually, not standardizing your data, etc.) which means that you're not thinking about the fact that computers have finite numerical precision and adjusting your problem accordingly (e.g., have a wide dynamic range of numbers).

Or, (b) your problem is pretty ill-conditioned and there's probably no solving it in any useful way unless you crank up the precision.

Almost every problem I've encountered of this form (whenever I've worked on optimizers, or, more generally, solving problems) is of type (a). There aren't many more choices if your problem is of type (b), since you're almost always losing digits somewhere in any implementation of a numerical algorithm.

The mean of two numbers is a particularly easy example to analyze, but I don't think that the implementation is "wrong" since every (even slightly more complicated) numerical algorithm will run into the same problems. I think this is a good post in its content, I just don't agree with the author's final interpretation.

Because you're placing a higher weight on elements that come later in the list. Try it for the list [1, 2, 3, 4]

The mean is 2.5 if you sum them all and divide by 4.

With that method it is:

mean([1, 2, 3, 4]) = mean([1.5, 3, 4]) = mean([2.25, 4]) = 3.125

In all of these cases it could also be argued that the mean is not a useful metric for the data.