11–20 of 101 posts
As touched upon in the article, the objective not being differentiable is a big deal for modern machine learning methods.
However: if you look at the shape of the squareroot of sum squares, it's a circle, so you can rotate it. If you take the absolute, it's a square, so that cannot be rotated; the cuberoot of cubes and fourthroot of fourths, etc. look like rounded edge squares, and that cannot be rotated either, so if you have a change of vector basis, you're out of luck.
With the gaussian forms of other powers, none of them have the central limit property.
"inner products/gaussians" - the absolute value (and also cuberoot of absolute cubes, fourth root of fourth powers) also define inner products. Likewise, there are "gaussian-like formulas" which take these powers instead of squared. However: if you look at the shape of the squareroot of sum squares, it's a circle, so you can rotate it. If you take the absolute, it's a square, so that cannot be rotated; the cuberoot o…
We want a metric essentially because if we converge or have a good approximation in the metric then we are close in some important respects.
Squared error, then, gives one such metric.
But for some given data, usually there are several metrics we might use, e.g., absolute error (L^1), worst case error (L^infinity), L^p for positive integer p, etc.
From 50,000 feet up, the reason for using squared error is that get to have the Pythagorean theorem, and, more generally, get to work in a Hilbert space, a relatively nice place to be, e.g., we also get to work with angles from inner products, correlations, and covariances -- we get cosines and a version of the law of cosines. E.g., we get to do orthogonal projections which give us minimum squared error.
With Hilbert space, commonly we can write the total error as a sum of contributions from orthogonal components, that is, decompose the error into contributions from those components -- nice.
The Hilbert space we get from squared error gives us the nicest version of Fourier theory, that is, orthogonal representation and decomposition, best squared error approximation.
We also like Fourier theory with squared error because of how it gives us the Heisenberg uncertainty principle.
Under meager assumptions, for real valued random variables X and Y, E[Y|X], a function of X, is the best squared error approximation of Y by a function of X.
Squared error gives us variance, and in statistics sample mean and variance are sufficient statistics for the Gaussian; that is, for statistics, for Gaussian data, can take the sample mean and sample variance, throw away the rest of the data, and do just as well.
For more, convergence in squared error can imply convergence almost surely at least for a subsequence.
Then there is the Hilbert space result, every nonempty, closed, convex subset has a unique element of minimum norm (from squared error) -- nice.
Why do we assume gaussian errors? There is seldom a gaussian distribution in the real world usually because the probability for large error values doesn't not decay that fast. We use it because the math is easy and we can actually solve the problem assuming that.
An honest question - do we even need statistics when we have machine learning? Statistics to me appears as a hack/aggregation of data we couldn't process at once in the past; these days ML + Big Data can achieve that and instead of statistics we can do computational inference instead. To me this looks like looking back to "old ways" for a reference point instead of looking forward to the unknown but more exciting.
For minimizing the square of the errors I think the good reason is because, assuming your data has gaussian probability distribution, minimizing the square error corresponds to maximizing the likelihood of the measurement, as you and others have said. Why do we assume gaussian errors? There is seldom a gaussian distribution in the real world usually because the probability for large error values doesn't not decay tha…
You're talking about fat tails?
"inner products/gaussians" - the absolute value (and also cuberoot of absolute cubes, fourth root of fourth powers) also define inner products. Likewise, there are "gaussian-like formulas" which take these powers instead of squared. However: if you look at the shape of the squareroot of sum squares, it's a circle, so you can rotate it. If you take the absolute, it's a square, so that cannot be rotated; the cuberoot o…
What kind of inner products are defined by the absolute value, cuberoot of absolute cubes, fourth root of fourth powers? I never heard of that and would be glad to learn about it.